Reliable and Reproducible Outputs

import numpy as np
rng = np.random.default_rng(4)

# Stand-in for a decoder: softmax over 5 candidate answers at varying temperature.
logits = np.array([2.0, 1.6, 0.4, 0.1, -0.5])
answers = list("ABCDE")

def probs(T):
    z = logits / T
    z -= z.max()                      # log-sum-exp: T near 0 overflows otherwise
    p = np.exp(z)
    return p / p.sum()

print(f"{'T':>5} {'modal':>6} {'P(modal)':>9} {'distinct/4k':>12} {'entropy':>8}")
for T in [0.01, 0.3, 0.7, 1.0, 1.5]:
    s = rng.choice(len(logits), size=4000, p=probs(T))
    c = np.bincount(s, minlength=5) / 4000
    ent = abs(-(c[c > 0] * np.log2(c[c > 0])).sum())   # entropy >= 0; abs kills the -0.0
    print(f"{T:>5.2f} {answers[c.argmax()]:>6} {c.max():>9.3f} "
          f"{int((c > 0).sum()):>12d} {ent:>8.2f}")
print("\nT->0 is deterministic (entropy 0); T=1.5 spreads mass over all five.")

    T  modal  P(modal)  distinct/4k  entropy
 0.01      A     1.000            1     0.00
 0.30      A     0.787            4     0.78
 0.70      A     0.564            5     1.51
 1.00      A     0.484            5     1.78
 1.50      A     0.392            5     2.06

T->0 is deterministic (entropy 0); T=1.5 spreads mass over all five.

import numpy as np
rng = np.random.default_rng(5)

# A model that is right with probability p on each independent attempt.
def majority_correct(p, m, trials=20000):
    draws = rng.random((trials, m)) < p
    return (draws.sum(axis=1) > m / 2).mean()

print(f"{'p':>5} " + " ".join(f"m={m:<6}" for m in [1, 3, 5, 9]))
for p in [0.4, 0.5, 0.6, 0.7, 0.8]:
    row = " ".join(f"{majority_correct(p, m):<8.3f}" for m in [1, 3, 5, 9])
    print(f"{p:>5.1f} {row}")
print("\nBelow p=0.5 voting makes it worse; above, it amplifies. Condorcet, not magic.")

    p m=1      m=3      m=5      m=9     
  0.4 0.403    0.346    0.313    0.267   
  0.5 0.499    0.497    0.504    0.500   
  0.6 0.595    0.651    0.682    0.730   
  0.7 0.701    0.785    0.835    0.904   
  0.8 0.801    0.897    0.941    0.982   

Below p=0.5 voting makes it worse; above, it amplifies. Condorcet, not magic.

0.1Reliable and Reproducible Outputs¶

Run the same prompt twice and you will often get two different answers. For a statistician this should provoke curiosity rather than alarm: a language model with nonzero sampling temperature is a random number generator, and a single response is a draw, not an estimate. The reproducibility question is therefore two questions wearing one coat. The first is stochastic --- can I obtain the same output again, and if not, how much does it vary? The second is substantive --- does the conclusion I drew survive that variation? A $p$-value computed by a script whose input was one lucky draw from a chat window is not reproducible in any sense a journal should accept, and the fix is not to hope for determinism but to characterise the sampling distribution the way you would for any other Monte Carlo procedure. This section takes that stance seriously. We treat repeated model calls as replicates, define run-to-run variance as an estimable quantity, examine what aggregation across draws does and does not buy, explain why fixing the seed is neither sufficient nor always possible, and lay out the pinning and logging discipline that makes an LLM-containing analysis rerunnable a year later.

0.1.1Decompose the variance before you try to remove it¶

Let $Y_{ijk}$ be the outcome of interest --- an accuracy indicator, an extracted number, a graded score --- for item $i$, prompt variant $j$, and repeated call $k$. A first-pass decomposition treats items, prompts and calls as crossed and nested random effects,

which is an ordinary mixed model your readers can fit before lunch. The value of writing it down is that it names the enemy. If $\sigma^2_{\text{call}}$ dominates, average more draws. If $\sigma^2_{\text{prompt}}$ dominates, no amount of averaging within a prompt helps, and you must either pre-specify the prompt or report across a pre-registered family of them. If $\sigma^2_{\text{item}}$ dominates --- the usual case --- your evaluation is item-limited, and the fix is more items, which is the sample-size argument made in the evaluation section.

0.1.2What majority voting actually buys¶

Sampling $B$ answers and taking the plurality --- “self-consistency” --- is the standard reliability trick Wang et al., 2022. Under the fiction of independent draws each correct with probability $p > 1/2$, the vote is correct with probability $\Pr(\mathrm{Bin}(B,p) > B/2) \to 1$, and the gain looks free. Draws from one model are not independent: they share a prompt, a parameter vector and a systematic misunderstanding. A convenient way to see the consequence is to let the per-call success probability itself be random, $P \sim \mathrm{Beta}(\alpha,\beta)$ with mean $p$ and intra-item correlation $\rho = 1/(\alpha+\beta+1)$, so that the vote count is beta-binomial. Then as $B$ grows

a hard ceiling strictly below one that no amount of extra sampling can breach. The interpretation is the one every meta-analyst knows: correlated replicates carry less information than their count suggests, and here the shortfall is not gradual but asymptotic.

0.1.3Seeds, determinism, and why $T=0$ is not enough¶

Setting temperature to zero makes decoding a deterministic argmax, and a seed makes any residual sampling reproducible on your machine. Neither delivers reproducibility across time or across machines. Floating-point addition is not associative, so a change in batch size, tensor-parallel layout, GPU model, or kernel library changes the summation order and hence the logits in the last decimal places; where two candidate tokens are nearly tied, that noise flips the argmax and the two continuations diverge irreversibly. Mixed-precision arithmetic makes this worse by design Micikevicius et al., 2018, and distributed execution reorders reductions as a matter of course Goyal et al., 2017Rajbhandari et al., 2020. This is not an abstract hazard even for a locally served model: high-throughput inference servers such as vLLM group incoming requests into batches whose composition depends on what else arrived at that moment, so the same prompt run twice on the same weights and the same hardware can be reduced in a different order and, at a near-tie, decode to a different token. The lesson generalises well beyond language models: a single training run is a draw from a distribution too, which is why seed-to-seed variance belongs in every reported comparison. :::{note} Author note Write the two-paragraph version of the floating-point argument with a tiny worked example --- sum three numbers in two orders and show the last-bit difference --- because readers will not believe it otherwise. Then give the operational recommendation: report results as a mean over $\ge 5$ seeds with a standard error, never as a single run, and treat exact bitwise reproduction as a nice-to-have rather than the goal. :::

0.1.4A pinning and logging discipline¶

Reproducing an analysis that contains a model call requires recording the things that determine the output. Figure the figure lists them by layer. The rule of thumb is that anything the provider can change without telling you must be captured on your side: the exact model identifier including its dated snapshot, the decoding parameters, the full prompt text, any retrieved context, and the raw response including token log-probabilities where available. For work that must be reproducible over years rather than months, the only robust option is a locally hosted open-weights model whose checkpoint you archive, since a hosted endpoint can be retired at the provider’s convenience. (Mechanically this means pinning by content rather than by name --- the Hugging Face Hub exposes an immutable commit hash per repository that transformers and vllm will both accept in place of a tag, and runtimes such as llama.cpp and Ollama address a quantised checkpoint by its file digest --- so that a re-run either loads the same bytes or fails loudly rather than loading something newer under an unchanged label.)

0.1.5Reporting standards for an LLM-in-the-loop analysis¶

The book should state a standard plainly, because none is yet settled. Report the model and snapshot, the decoding settings, the number of repeated calls, and the aggregation rule. Report a measure of run-to-run variability, not just a point estimate. Pre-specify the prompt, or pre-specify the family of prompts over which results are averaged, and report both. Archive raw responses. This is exactly the checklist culture that Pineau et al. (2021) argued for in machine learning and that Gebru et al. (2021) and Mitchell et al. (2019) argued for data and models respectively; the novelty here is only that the analysis instrument is now stochastic. One class of output resists this discipline almost completely: a report produced by an agent that searched the live web depends on what was retrievable at that moment, so the retrieved material must be archived alongside the response or the result is unrepeatable in principle rather than merely in practice (Section that section). :::{note} Author note Close with the framing that lands with this audience: an LLM call is a measurement instrument with a drift problem. Chemists calibrate instruments and record lot numbers; we should do the same. Cross-reference the technical-debt argument of Sculley et al. (2015). :::

0.1.6Tools in practice¶

The tools that matter for reproducibility are not the ones that make a model answer better; they are the ones that put a boundary around what can change underneath you. Three durable distinctions organise the field. First, local weights versus a hosted endpoint: only the former can be archived, and archiving the checkpoint is the difference between an analysis that reruns in five years and one that cannot. Second, a provider-specific client versus a routing layer that presents one interface over many backends: the routing layer is what lets the same script be re-executed against a locally archived model when the hosted one is withdrawn, and it is also where a cache belongs. Third, ad hoc logging versus a tracing store: if the raw request and response are not written somewhere durable at call time, they do not exist, because nothing in the stack will reconstruct them for you. Capabilities and interfaces in this area move quickly enough that any specific claim here would be stale before the book is read; check current documentation, and evaluate a candidate tool by asking which of these three boundaries it draws.

• [vLLM] Local serving. Serves open-weight models through a standard chat-completions interface with batching and paged attention. Fits: any analysis that must be rerunnable after the vendor landscape changes, and any data that cannot leave the institution. Watch: batching makes throughput depend on load and makes numerics depend on batch composition, so record the serving configuration, not only the weights.

• [Ollama / llama.cpp] Local serving, single machine. Run quantised checkpoints on a laptop or workstation without a GPU cluster. Fits: pilot studies, method development, and teaching, where a self-contained artefact matters more than throughput. Watch: the quantisation level is part of the instrument; results from a heavily quantised checkpoint are not interchangeable with those from the full-precision weights, and the quantisation must be reported.

• [**Hugging Face Hub and transformers**] Weight distribution and pinning. Hosts checkpoints addressable by an immutable per-commit revision hash rather than only by a mutable tag. Fits: the pinning layer of Figure the figure; record the revision in the run manifest. Watch: a repository can be renamed, gated or withdrawn by its owner, so for work that must survive, archive the checkpoint itself rather than a pointer to it.

• [LiteLLM] Routing and caching. Presents one client interface across hosted and locally served backends, with request caching and per-call logging. Fits: the seam between your analysis code and whichever model serves it; the natural place to implement the content-addressed cache below. Watch: a routing layer normalises parameters across backends, which means the request your code issued is not necessarily the request the backend received --- log what the backend saw.

• [MLflow / Langfuse] Experiment tracking and tracing. Store every prompt, response, parameter set and latency as a queryable record tied to a run identifier. Fits: the archival requirement of the reporting standard above, and the raw material for fitting the equation. Watch: tracing captures what your code sent, so redact identifiable content before it is written, and confirm that the store retains full response bodies rather than truncated previews.

• [DSPy] Prompt programs. Expresses a pipeline as typed modules and optimises the prompt text against a metric on a training split rather than by hand. Fits: the prompt-variance component --- it makes prompt selection an explicit fitted step with its own training data. Watch: optimising prompts against your evaluation set is fitting on the test data; hold out a genuinely untouched split, and report that the prompt was learned rather than written.

The cache the reporting standard asks for is a few lines, and writing it out makes clear what "the same call" has to mean. The key is a hash of everything that determines the response --- model identity, snapshot, decoding parameters and the exact prompt bytes --- so that a cache hit is a guarantee rather than a hope.

def call_key(model, snapshot, prompt, **decode): “”“Content hash of everything that determines the response.”“” spec = {“model”: model, “snapshot”: snapshot, “prompt”: prompt, “decode”: decode} blob = json.dumps(spec, sort_keys=True, separators=(“,”, “:”)) return hashlib.sha256(blob.encode(“utf-8”)).hexdigest()

base = dict(model=“local-instruct-model”, snapshot=“”, prompt=“Extract the smoking status.”, temperature=0.0, top_p=1.0)

variants = {"baseline ": base, "trailing space ": {**base, “prompt”: base[“prompt”] + " “}, “temperature changed”: {**base, “temperature”: 0.7}, “identical call “: dict(base)} keys = {name: call_key(**spec) for name, spec in variants.items()} for name, k in keys.items(): print(f”{name} {k[:16]} same as baseline: " f”{k == keys['baseline ']}”)

cache, network = {}, 0 other = {**base, “prompt”: “Extract the biopsy status.”} workload = [base, other, base, base] for spec in workload: k = call_key(**spec) if k not in cache: network += 1 cache[k] = “” print(f"\ncalls requested {len(workload)} calls actually issued {network}" f" distinct keys cached {len(cache)}")

calls requested 4 calls actually issued 2 distinct keys cached 2

The trailing space is the point of the example. A one-character edit that no reader would notice produces an entirely different key, which is the correct behaviour: it is also a different request, and a cache that treated the two as interchangeable would silently serve one prompt’s answer for another. The same strictness is what makes the cache a reproducibility device rather than a cost optimisation. Store the raw response body under this key, commit the store alongside the analysis, and every downstream figure regenerates with the network disabled --- which is the concrete form of the archival recommendation above, and the same argument made for caching scraped web data.

Pinning tells you what you asked for; it does not tell you whether what you got has changed. A canary suite closes that gap: fix a small set of probe prompts, run them at temperature zero, hash the responses, and re-run the suite whenever the analysis is re-executed. The interesting part is not the comparison but the sample-size question underneath it, since a suite too small to detect a change is a suite that certifies stability it never tested for.

def digest(text): return hashlib.sha256(text.encode(“utf-8”)).hexdigest()[:12]

1A canary suite: fixed probes run at T = 0, responses hashed and stored.¶

probes = [f"probe-{i:02d}" for i in range(12)] baseline = {p: digest(f"stored-response-{p}") for p in probes}

today = dict(baseline) # replayed from a later run today[“probe-03”] = digest(“stored-response-probe-03 (reworded)”) today[“probe-09”] = digest(“stored-response-probe-09 (reworded)”)

changed = [p for p in probes if today[p] != baseline[p]] n, x = len(probes), len(changed) ci = binomtest(x, n).proportion_ci(confidence_level=0.95, method=“exact”) print(f"changed {x}/{n}: {changed}“) print(f"drift rate {x / n:.3f} 95% CI ({ci.low:.3f}, {ci.high:.3f})”)

for d in (0.20, 0.10, 0.05, 0.01): n_needed = math.ceil(math.log(0.10) / math.log(1 - d)) print(f"90% chance of catching drift that touches {d:5.0%} of prompts:" f" {n_needed:3d} probes")

Two readings. The drift estimate itself is nearly uninformative --- twelve probes give an interval running from two percent to nearly half --- which is the same lesson the evaluation section makes about underpowered comparisons, arriving here in a monitoring rather than a benchmarking guise. The second block is the design calculation: to have a $90\%$ chance of catching a change that affects a fraction $d$ of prompts you need about $\log(0.10)/\log(1-d)$ probes, which is 22 at $d=0.10$ and 230 at $d=0.01$. A suite of five probes, which is what people build, detects only changes so pervasive that you would have noticed them anyway. Note also what the canary cannot see: it certifies that these prompts return the same bytes, and generalising from that to your workload assumes the probes are exchangeable with it.

1.1Exercises¶

1. Write the equation as a mixed model and give the expression for the intraclass correlation of two calls on the same item with the same prompt. Explain which variance component more items reduces and which more calls reduces.

2. Derive the limit the equation for the beta-binomial voting model and show it is strictly less than one for any $\rho > 0$. Find the value of $\rho$ at which $B = 31$ votes achieve less than half the accuracy gain that independence would give.

3. Suppose each call costs $c$ and the analyst values accuracy linearly. Using the beta-binomial ceiling, derive the optimal number of votes $B^\ast$ and show it is finite whenever $\rho > 0$ but unbounded when $\rho = 0$.

4. Explain why greedy decoding does not guarantee identical output across two machines, using the non-associativity of floating-point addition. Then argue whether bitwise reproducibility is the right target for a scientific analysis, or whether a reported variance is.

5. (Computational) Choose a task with 100 items. For each item, run 10 calls at $T=0.7$ under each of 4 prompt paraphrases. Fit the equation, report all four variance components with intervals, and state which one your evaluation budget should attack.

6. (Computational) Estimate $\rho$ from the data collected above, then plot observed majority-vote accuracy against $B$ alongside the binomial and beta-binomial predictions, reproducing Figure the figure on real output.

7. (Computational) Build a caching wrapper that hashes the prompt and decoding settings, stores every raw response, and replays from cache on re-execution. Demonstrate that a downstream analysis script produces byte-identical results on a second run with the network disabled.