Running Models Locally

1Local LLMs¶

There are two reasons a statistician ends up running a language model on their own hardware, and only one of them is cost. The first is data governance: if the text you want to process is protected health information, student records, or unpublished data from a collaborator, sending it to a third-party API is often simply not permitted, and no amount of convenience changes that. The second is reproducibility, which this readership cares about more than the field does --- a local model with pinned weights and a fixed seed is a fixed object that will give the same answer next year, while a hosted endpoint is a moving target that may be silently updated or retired between your submission and the referee report. The cost of taking this route is that you must think about memory, numerical precision, and throughput, three subjects a statistician can normally ignore. This section is a practical guide to that arithmetic: how large a model fits in the hardware you have, what quantization does to the weights and therefore to the predictions, and how to decide honestly whether the smaller local model is good enough for your task.

1.1The Memory Arithmetic¶

The single calculation that determines whether a model runs on your machine is its weight footprint. A model with $N$ parameters stored at $b$ bits each requires

so a 7-billion-parameter model needs about 26 GiB in fp32, 13 GiB in fp16, and roughly 3.3 GiB at 4 bits. Inference needs more than the weights: activations, and above all the key--value cache, which for a transformer with $L$ layers, $H$ key/value heads of dimension $d_h$, context length $T$ and $b_c$ bits per cached value costs

linear in context length. At long context the cache can rival or exceed the weights, which is why a model that loads happily may still fail on a long document. (Serving runtimes attack this directly: vLLM’s paged allocator stores the cache in fixed-size blocks so that concurrent requests do not each reserve their worst-case contiguous slab Kwon et al., 2023.)

1.2Quantization as Deliberate Measurement Error¶

Quantizing a weight matrix replaces each entry with the nearest point on a coarse grid. For a symmetric uniform scheme with $b$ bits and scale $s$,

and the induced error $\varepsilon = \hat w - w$ behaves, to first order, like measurement error on the coefficients. The naive analysis --- treat $\varepsilon$ as uniform on $[-s/2, s/2]$, note the variance is $s^2/12$, conclude that error falls by a factor of two per bit --- is a useful first pass and is what the right panel of Figure the figure shows. It also explains why practice departs from it: $s$ is set by the largest weight in the group, so a handful of outlier weights inflates the step size for everything else. Every serious quantization method is a response to that fact, whether by grouping weights into small blocks with their own scales, keeping outlier channels in higher precision, or choosing the grid to be optimal for a normal prior, as in the NF4 format of QLoRA Dettmers et al., 2023. (The two post-training schemes you will meet by name when downloading weights, GPTQ Frantar et al., 2022 and AWQ Lin et al., 2023, are both of this kind: each uses a small calibration sample to decide which weights may be coarsened and which must be protected, which makes the calibration set a design choice worth recording.)

1.3Adapting a Local Model Without Retraining It¶

Full fine-tuning of even a 7B model requires optimizer state several times the size of the weights and is out of reach for most local hardware. Low-rank adaptation sidesteps this by freezing $W_0$ and learning a low-rank update Hu et al., 2021,

which trains a fraction of a percent of the parameters and yields an adapter file of a few tens of megabytes. Combining a 4-bit frozen base with a low-rank update in higher precision --- QLoRA --- brings fine-tuning of a mid-sized model onto a single consumer GPU Dettmers et al., 2023. The statistical reading is that $r$ is a rank constraint acting as a regularizer, and the effective number of parameters, not the nominal one, governs how much data you need.

1.4A Reproducible Local Workflow¶

The reason to prefer a local model in a paper is that it can be pinned. Record the model repository and revision hash, the quantization format, the runtime and its version, the decoding parameters, and the seed --- all five, because any one of them can change your outputs. Even then, exact bitwise reproducibility across different GPUs or batch sizes is not guaranteed, so the honest claim is distributional reproducibility, and the honest report includes several seeds.

name, rev = “microsoft/Phi-4-mini-instruct”, “main” # pin a commit hash here tok = AutoTokenizer.from_pretrained(name, revision=rev) model = AutoModelForCausalLM.from_pretrained( name, revision=rev, torch_dtype=torch.bfloat16, device_map=“auto”)

msgs = [{“role”: “user”, “content”: “State Bayes’ theorem in one sentence.”}] ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors=“pt”) torch.manual_seed(0) out = model.generate(ids.to(model.device), max_new_tokens=128, do_sample=True, temperature=0.7, top_p=0.95) print(tok.decode(out[0, ids.shape[-1]:], skip_special_tokens=True)) print(torch.cuda.max_memory_allocated() / 2**30, “GiB peak”)

1.5Tools in Practice¶

The categories below are what stands between a file of weights and a number in your paper. They are worth learning as categories, because the products inside each one turn over quickly while the division of labour between them has been stable: something loads weights, something quantizes them, something serves them at throughput, and something records what you ran.

• [Research-oriented model library] Load and inspect. Hugging Face transformers and its ecosystem: load weights by repository and revision, get the tokenizer and chat template that belong to them, and reach logits directly. Fits: anything that needs per-token log-probabilities --- the perplexity comparison of Exercise 5 cannot be done through a chat interface. Watch: defaults are convenient and unpinned. Pass an explicit revision, an explicit dtype and an explicit seed, or the object you loaded today is not the object you load next year.

• [Quantized-file runtime] CPU and consumer-hardware execution. llama.cpp and the GGUF file format it defines run quantized models on CPU and on Apple silicon; Ollama wraps the same machinery behind a model-pull command and a local HTTP endpoint. Fits: laptop-scale work, and any setting where no GPU is available. Watch: the file-format split is the trap. GGUF files do not load into transformers and safetensors checkpoints do not load into llama.cpp without conversion, and a converted file inherits the quantization choices of whoever converted it.

• [Throughput-oriented serving engine] Batched inference. vLLM and similar servers keep a GPU busy across concurrent requests, principally by managing the key--value cache of the equation in fixed-size pages rather than per-request slabs Kwon et al., 2023. Fits: scoring thousands of prompts, which is what an evaluation actually is. Watch: batching changes reduction order, so results are not bit-identical to the unbatched run. Fix the batch size along with the seed, and report distributional rather than exact reproducibility.

• [Post-training quantizers] Compression. GPTQ Frantar et al., 2022 and AWQ Lin et al., 2023 choose a coarse grid using a small calibration corpus, protecting the outlier weights that inflate the scale $s$ in the equation. Fits: the step between “the model does not fit” and “the model fits”. Watch: the calibration corpus is a design choice. A model calibrated on generic web text and deployed on clinical notes has been coarsened where it mattered; record which pre-quantized file you downloaded, not just the bit width.

• [Parameter-efficient fine-tuning stacks] Adaptation. Libraries implementing the equation and its quantized variant, so that a rank-$r$ adapter can be trained on one consumer GPU Hu et al., 2021Dettmers et al., 2023. Fits: teaching a local model a house format or vocabulary without touching the base weights. Watch: an adapter is meaningless without its base model and revision. Ship both, or the few tens of megabytes you saved are unusable.

• [Local embedding and reranking models] Retrieval components. The same open-weight route applies to the encoders of Section that section, which are small enough to run on CPU. Fits: keeping an entire retrieval pipeline inside your institution’s network, which is usually the reason the governance question arose in the first place. Watch: embeddings from different models are not comparable, so a change of encoder invalidates a stored index and every distance computed from it.

Before downloading anything, do the arithmetic. The calculation below combines the equation and the equation with a bandwidth-bound throughput ceiling; substitute your own device’s memory and bandwidth, which the runtime or the data sheet will tell you, and read off which configurations are feasible.

import numpy as np
GiB = 2**30
vram = 24.0                                  # your device, in GiB; read it from the runtime
bw = 400.0                                   # its memory bandwidth in GB/s, from the data sheet
N, L, Hkv, dh = 13e9, 40, 8, 128             # 13B-class, 8 key/value heads (grouped-query)
print(" w-bits  kv-bits  context   weights   KV cache   total+20%   fits   tok/s ceiling")
for bw_, kvb, T in [(16, 16, 8192), (8, 16, 8192), (4, 16, 8192),
                    (4, 16, 65536), (4, 8, 65536), (4, 8, 262144)]:
    w = N * bw_ / 8 / GiB
    kv = 2 * L * Hkv * dh * T * kvb / 8 / GiB
    tot = 1.20 * (w + kv)
    print(f"{bw_:7d}  {kvb:7d}  {T:7d}   {w:7.2f}   {kv:8.2f}   {tot:9.2f}   "
          f"{str(tot <= vram):>5s}   {bw / (w + kv):13.1f}")
Tstar = N * 16 / 8 / (2 * L * Hkv * dh * 16 / 8)
print(f"context at which the cache equals the fp16 weights: {Tstar:.0f} tokens")
print(f"same for 4-bit weights and fp16 cache:              {Tstar / 4:.0f} tokens")

 w-bits  kv-bits  context   weights   KV cache   total+20%   fits   tok/s ceiling
     16       16     8192     24.21       1.25       30.56   False            15.7
      8       16     8192     12.11       1.25       16.03    True            29.9
      4       16     8192      6.05       1.25        8.76    True            54.8
      4       16    65536      6.05      10.00       19.26    True            24.9
      4        8    65536      6.05       5.00       13.26    True            36.2
      4        8   262144      6.05      20.00       31.26   False            15.4
context at which the cache equals the fp16 weights: 158691 tokens
same for 4-bit weights and fp16 cache:              39673 tokens

The first row does not fit and the second does, which is the whole practical content of quantization. Two further readings are worth extracting. The throughput ceiling in the last column is bandwidth divided by bytes touched per token, and it falls as the context grows even though the weights have not changed: generation is memory-bound, so a longer conversation is slower for a reason that has nothing to do with arithmetic. And the crossover context --- where cache equals weights --- moves down by exactly the compression factor, so the more aggressively you quantize the weights, the sooner the cache becomes the binding constraint. That is why cache quantization and paged allocation Kwon et al., 2023 appear immediately after weight quantization in every serving stack: they are the next term in the same budget.

Substitute your own numbers before trusting any of this. The point of the calculation is that it is a calculation, not a benchmark: it takes two device constants and four model constants, all of which are published, and returns a feasibility answer that does not depend on anyone’s marketing.

1.6Exercises¶

1. Using the equation and the equation, compute the total memory needed to serve a 13B-parameter model ($L=40$, $H=40$, $d_h=128$) at fp16 weights and fp16 cache with a 16{,}384-token context. At what context length does the cache exceed the weights?

2. Show that for weights uniform on $[-s/2, s/2]$ rounding error the mean squared error is $s^2/12$, and hence that relative RMSE falls by a factor of two per additional bit. Then explain, in one paragraph, why the right panel of Figure the figure shows the max-clipping rule beating the $3\sigma$ rule at 8 bits but losing to it at 2 bits.

3. Count the trainable parameters in the equation for a $4096 \times 4096$ weight matrix at $r=8$, and express the result as a fraction of the full matrix. What rank would be needed to reach $10\%$?

4. Computational. Simulate the weight distribution of one layer as a heavy-tailed draw (say $t_3$ scaled to sd 0.02), quantize it with the equation at $b = 8, 4, 3$, and plot the relative RMSE for per-tensor scaling versus per-block scaling with blocks of 64. Quantify how much of the improvement from blocking is attributable to the tail.

5. Computational. Run a small instruct model locally at fp16 and again at 4 bits, on the same 50 prompts with the same seed. Report the mean per-token log-likelihood under each and the fraction of prompts where the two answers differ materially. Comment on whether the perplexity change would have predicted the task-level change.