Prompting as Conditioning - AI Tools for Scientists

Prompt engineering is the practice of writing the text that conditions a language model, and it deserves more respect from statisticians than its faintly unserious name suggests. A trained language model is a fixed conditional distribution $p_\theta(y \mid x)$ over token sequences; you cannot change $\theta$ without fine-tuning, so the only lever left is $x$ . Writing a prompt is therefore not a matter of coaxing a machine into cooperation but of selecting the conditioning event in a conditional distribution, and every intuition you have about conditioning --- that it can be uninformative, that it can be inadvertently selective, that conditioning on the wrong variable answers the wrong question --- carries over intact. Seen this way the field’s stock techniques stop being folklore: a few worked examples in the prompt are a conditioning set, a decoding temperature is a variance knob, a system message is an offset, and asking the model to reason step by step enlarges the sample space so that the answer token is no longer required to be a deterministic function of the question. The purpose of this section is to make those correspondences explicit, so that you can reason about prompts with the tools you already own rather than by collecting tricks. The catch, stated once here and returned to throughout, is that a prompt is a covariate you chose after seeing the outcome: the same freedom that makes prompting powerful makes any performance number obtained by iterating on a prompt an optimistically biased one, for exactly the reasons that make an unregistered subgroup analysis untrustworthy. If a prompt is a conditioning event, then an ambiguous prompt is a conditioning event that fails to select a subpopulation. A three-line naive-Bayes classifier over a six-document corpus makes the point without any model call: the same word carries two senses, and the posterior over senses is what the prompt actually controls.

import numpy as np
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
docs = ["power analysis determines the sample size", "sample size for adequate power",
        "estimating the required number of participants by power",
        "gpu compute power for training runs", "raw compute power of the cluster",
        "training throughput and available compute power"]
y = np.array([0, 0, 0, 1, 1, 1])                       # 0 = study design, 1 = hardware
v = CountVectorizer(); X = v.fit_transform(docs)
clf = MultinomialNB().fit(X, y)
for q in ["what sample size gives adequate power",
          "how many subjects do I need",
          "how much power do I need"]:
    pr = clf.predict_proba(v.transform([q]))[0]
    print(f"P(study design | q) = {pr[0]:.3f}   q = {q!r}")

P(study design | q) = 0.942   q = 'what sample size gives adequate power'
P(study design | q) = 0.500   q = 'how many subjects do I need'
P(study design | q) = 0.494   q = 'how much power do I need'

The first query resolves the sense almost completely; the third leaves it at a coin flip. “Be specific” is not style advice but a statement about the posterior: adding sense-disambiguating words is choosing a conditioning event on which the answer distribution is concentrated, and a vague prompt is a mixture over the tasks you might have meant.

A dictionary for prompt engineering: a prompt is the conditioning event of a fixed conditional distribution, and most prompting technique is design, aggregation or elicitation under another name.

ML / AI term	Statistical analogue	What is the same	What is different / the catch
Prompt	The conditioning event in $p_\theta(y\mid x)$	Restricting attention to a subpopulation of outputs	Free text you composed: no design matrix, no coding, no contrast to write down
Prompt engineering	Covariate construction; specification search	Changing the answer by changing what is conditioned on	Done against observed output, so a reported score is a maximum over unrecorded attempts
Zero-shot prompting Radford et al., 2019	Applying a fit to a new population uncalibrated	Reusing a model on a new task	Task identity rides on wording, so human-equivalent paraphrases are different events
In-context learning Brown et al., 2020	Conditioning on a small labeled sample	A few examples fix the mapping applied next	No parameters update, nothing persists, and example order changes the answer
Choice of exemplars	Sampling design for a tiny conditioning set	Coverage and representativeness matter	Class balance biases the output directly; drawing them from the evaluation set leaks
Instruction following Ouyang et al., 2022	A fit where a stated request predicts the response	A systematic input--outcome relationship	Induced by one annotator pool, so “be concise” means what they rewarded
System prompt	A fixed offset applied to every case	Shifting every prediction one way	Its pull decays with conversation length: a nudge, not a constraint
Chain-of-thought prompting	Introducing an intermediate latent quantity	Replacing a hard map by easier steps	The stated reasoning is generated text, not a trace of the computation
Self-consistency by majority vote	Monte Carlo with a majority aggregator; bagging Breiman, 1996	Averaging out the variance of a stochastic procedure	Draws share one model’s bias, so a tight majority is precision, not accuracy
Temperature, top- $p$ Holtzman et al., 2020	A dispersion parameter on the predictive law	Trading variance against typicality	Temperature zero is the greedy continuation, not the mean or the modal sequence
Format instructions	Fixing the measurement scale before collection	Deciding how a response is recorded	Compliance is probabilistic; rejects are not missing at random
Verbalized confidence	An elicited subjective probability	A self-reported probability	Poorly calibrated and wording-sensitive; score it first Gneiting & Raftery, 2007Guo et al., 2017
Soft prompts Lester et al., 2021Li & Liang, 2021	Estimating a few parameters, rest fixed	A low-dimensional adjustment to a frozen fit	A vector, not readable text: it cannot be audited or transferred
Prompt evaluation	Model selection on a validation sample	Comparing candidates by held-out score	Candidates are dependent and adaptive, so the winner’s score is a selected maximum

In-context learning conditions on a handful of labelled cases, so its behaviour should look like fitting on a very small sample. We stand in for the model with a logistic fit on $k$ examples per class and re-draw which examples 200 times, which is the source of variability a single prompt evaluation never sees.

import numpy as np
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(0)
d = 5
mu = np.zeros(d); mu[0] = 1.6
Xte = np.vstack([rng.normal(mu, 1, (400, d)), rng.normal(-mu, 1, (400, d))])
yte = np.r_[np.ones(400), np.zeros(400)]
print(" exemplars   mean accuracy   SD across exemplar draws")
for k in [1, 2, 4, 8, 16, 32]:
    acc = []
    for _ in range(200):                                  # 200 draws of WHICH exemplars
        Xtr = np.vstack([rng.normal(mu, 1, (k, d)), rng.normal(-mu, 1, (k, d))])
        ytr = np.r_[np.ones(k), np.zeros(k)]
        acc.append(LogisticRegression().fit(Xtr, ytr).score(Xte, yte))
    print(f"{2*k:10d}   {np.mean(acc):13.4f}   {np.std(acc, ddof=1):24.4f}")

 exemplars   mean accuracy   SD across exemplar draws
         2          0.8106                     0.1020
         4          0.8690                     0.0605
         8          0.9018                     0.0304
        16          0.9141                     0.0199
        32          0.9238                     0.0143
        64          0.9297                     0.0073

Mean accuracy improves with the number of exemplars and the standard deviation across draws falls roughly as $1/\sqrt{k}$ --- the ordinary small-sample learning curve. The column that matters is the second: at two exemplars the choice of exemplars moves accuracy by ten points, so a prompt reported without re-sampling its examples is a single draw from a distribution whose width was never estimated.

Exemplars are a sampling design, and the most common design error is an unbalanced one. Here the test population is balanced 50/50; only the class proportions among the exemplars change.

import numpy as np
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(0)
d = 5
mu = np.zeros(d); mu[0] = 1.6
Xte = np.vstack([rng.normal(mu, 1, (500, d)), rng.normal(-mu, 1, (500, d))])
yte = np.r_[np.ones(500), np.zeros(500)]
print(" exemplars (pos:neg)   P(predict positive)   accuracy")
for npos, nneg in [(4, 4), (6, 2), (7, 1)]:
    rate, acc = [], []
    for _ in range(300):
        Xtr = np.vstack([rng.normal(mu, 1, (npos, d)), rng.normal(-mu, 1, (nneg, d))])
        ytr = np.r_[np.ones(npos), np.zeros(nneg)]
        f = LogisticRegression().fit(Xtr, ytr)
        rate.append(f.predict(Xte).mean()); acc.append(f.score(Xte, yte))
    print(f"{npos:12d}:{nneg:<8d}{np.mean(rate):12.3f}{np.mean(acc):19.3f}")
print("the test population is balanced, so 0.500 is the correct positive rate")

 exemplars (pos:neg)   P(predict positive)   accuracy
           4:4              0.497              0.901
           6:2              0.643              0.834
           7:1              0.828              0.669
the test population is balanced, so 0.500 is the correct positive rate

The predicted positive rate tracks the exemplar proportion rather than the population proportion, and accuracy degrades accordingly. This is prior shift, familiar from case--control sampling: the conditioning set carries a prevalence, the model inherits it, and no wording of the instruction repairs a design that misrepresents the base rate.

A system prompt is a term added to every case, and the reason its influence fades is arithmetic rather than psychological: a constant does not grow with the amount of conditioning text that follows it.

import numpy as np
offset = 2.0                                       # the system prompt, in log-odds
per_turn = -0.30                                   # each later turn pushes the other way
print(" turn t   linear predictor   P(comply)   offset's share of |eta|")
for t in [0, 2, 5, 8, 12, 20]:
    eta = offset + per_turn * t
    share = abs(offset) / (abs(offset) + abs(per_turn * t)) if t else 1.0
    print(f"{t:6d}   {eta:16.3f}   {1/(1+np.exp(-eta)):9.3f}   {share:22.3f}")
flip = int(np.ceil(offset / -per_turn))
print(f"the offset is out-voted after t = {flip} turns of accumulating context")
print("a constant term does not scale with n: this is why a system prompt is a nudge, not a constraint")

 turn t   linear predictor   P(comply)   offset's share of |eta|
     0              2.000       0.881                    1.000
     2              1.400       0.802                    0.769
     5              0.500       0.622                    0.571
     8             -0.400       0.401                    0.455
    12             -1.600       0.168                    0.357
    20             -4.000       0.018                    0.250
the offset is out-voted after t = 7 turns of accumulating context
a constant term does not scale with n: this is why a system prompt is a nudge, not a constraint

The offset dominates the first few turns and is out-voted by turn seven, because the accumulating context contributes a term proportional to $t$ while the system prompt contributes a constant. Treating a system prompt as a hard constraint is the error of treating an intercept shift as a bound; if a rule must hold, enforce it outside the model.

Chain-of-thought works by introducing an intermediate quantity, and the reason it works is visible in a model with no language in it at all. The answer here is the XOR of two latent facts, each a linear function of the inputs; a linear learner cannot represent the answer directly but can represent each fact.

import numpy as np
from sklearn.linear_model import LogisticRegression as LR
rng = np.random.default_rng(0)
n, d = 3000, 6
X = rng.normal(0, 1, (n, d))
t1 = (X[:, 0] + X[:, 2] > 0).astype(int)                 # two intermediate facts,
t2 = (X[:, 1] - X[:, 3] > 0).astype(int)                 # each a linear function of x
y = t1 ^ t2                                              # the answer is their XOR
tr, te = slice(0, 400), slice(400, n)
print(f"direct  x -> y                  accuracy = {LR().fit(X[tr], y[tr]).score(X[te], y[te]):.3f}")
m1, m2 = LR().fit(X[tr], t1[tr]), LR().fit(X[tr], t2[tr])
print(f"  step 1: x -> t1               accuracy = {m1.score(X[te], t1[te]):.3f}")
print(f"  step 2: x -> t2               accuracy = {m2.score(X[te], t2[te]):.3f}")
chain = m1.predict(X[te]) ^ m2.predict(X[te])
print(f"chain   x -> (t1,t2) -> y       accuracy = {np.mean(chain == y[te]):.3f}")
print(f"same learner, same data, same features: only the intermediate targets were added")

direct  x -> y                  accuracy = 0.483
  step 1: x -> t1               accuracy = 0.979
  step 2: x -> t2               accuracy = 0.979
chain   x -> (t1,t2) -> y       accuracy = 0.958
same learner, same data, same features: only the intermediate targets were added

The direct fit is at chance, the two intermediate fits are near perfect, and composing them recovers the answer. Same learner, same features, same 400 training cases: only the decomposition changed. Chain-of-thought is not the model thinking harder; it is the analyst supplying a factorization through a latent variable that the one-step map could not express.

Self-consistency samples several answers and takes the majority, which is bagging with a majority-vote aggregator. Bagging reduces variance, and variance is only part of the error --- the simulation below contrasts independent draws with draws that share a systematic error on 30% of items.

import numpy as np
rng = np.random.default_rng(0)
Q = 20000
hard = rng.random(Q) < 0.30                          # items this model always gets wrong
p_easy = (0.60 - 0.30 * 0.02) / 0.70                 # so both settings average 0.60 per draw
print("  m   independent errors   shared bias on 30% of items")
for m in [1, 3, 5, 11, 21]:
    ind = ((rng.random((Q, m)) < 0.60).sum(1) > m / 2).mean()
    cor = (np.where(hard[:, None], rng.random((Q, m)) < 0.02,
                    rng.random((Q, m)) < p_easy).sum(1) > m / 2).mean()
    print(f"{m:3d}   {ind:18.3f}   {cor:27.3f}")
print(f"ceiling with a shared bias = 1 - 0.30 = {1 - 0.30:.3f}")

  m   independent errors   shared bias on 30% of items
  1                0.599                         0.610
  3                0.647                         0.661
  5                0.685                         0.689
 11                0.751                         0.705
 21                0.822                         0.708
ceiling with a shared bias = 1 - 0.30 = 0.700

With independent errors the majority vote climbs steadily with $m$ ; with a shared bias it saturates almost immediately at the fraction of items the model does not systematically get wrong. Averaging removes variance and cannot touch bias, so a tight majority across draws is evidence of precision only. Reporting the agreement rate as a confidence measure conflates the two.

Temperature zero is often described as returning “the most likely answer”. It returns the token-by-token greedy path, which is not the same object. A three-state chain of length three is enough to separate them.

import numpy as np
from itertools import product
P = np.array([[0.34, 0.33, 0.33],                 # row = current token, col = next token
              [0.10, 0.10, 0.80],
              [0.80, 0.10, 0.10]])
start = np.array([0.40, 0.35, 0.25])
greedy = [int(start.argmax())]
for _ in range(2):
    greedy.append(int(P[greedy[-1]].argmax()))
p_of = lambda s: start[s[0]] * P[s[0], s[1]] * P[s[1], s[2]]
best = max(product(range(3), repeat=3), key=p_of)
print(f"greedy (temperature 0) sequence {tuple(greedy)}  p = {p_of(greedy):.4f}")
print(f"most likely sequence            {best}  p = {p_of(best):.4f}")
print(f"the greedy path is {p_of(best)/p_of(greedy):.2f}x less likely than the modal one")
print(f"P(next token | start) argmax = {int(start.argmax())}, and it is a trap")

greedy (temperature 0) sequence (0, 0, 0)  p = 0.0462
most likely sequence            (1, 2, 0)  p = 0.2240
the greedy path is 4.84x less likely than the modal one
P(next token | start) argmax = 0, and it is a trap

The greedy path is nearly five times less likely than the modal sequence, because a locally attractive first token leads into a flat continuation. Greedy decoding is coordinate-wise maximization of a joint distribution, and coordinate-wise maxima are not joint maxima --- the same reason the vector of marginal posterior modes need not be the joint MAP estimate.

Format instructions decide how a response is recorded, and non-compliance is missing data. The question is whether it is missing at random. Here the model reports a number, and long answers --- which are the large ones --- are the ones that fail to parse.

import numpy as np
rng = np.random.default_rng(0)
n = 2000
x = rng.normal(10, 3, n)                                   # the quantity the model reports
parsed = rng.random(n) < 1 / (1 + np.exp(0.6 * (x - 10)))  # large answers get verbose and fail to parse
print(f"mean over all responses            = {x.mean():.3f}")
print(f"parse (compliance) rate            = {parsed.mean():.3f}")
print(f"complete-case mean, parsed only    = {x[parsed].mean():.3f}")
print(f"mean of the discarded responses    = {x[~parsed].mean():.3f}")
print(f"bias from dropping the failures    = {x[parsed].mean() - x.mean():+.3f}")

mean over all responses            = 9.916
parse (compliance) rate            = 0.518
complete-case mean, parsed only    = 8.234
mean of the discarded responses    = 11.723
bias from dropping the failures    = -1.682

Half the responses parse, and the mean of those that do is 1.7 units below the mean of all responses, because the parse failure depends on the value being reported. Dropping unparseable generations is a complete-case analysis under MNAR missingness. Log the rejects and compare them to the retained cases; a parse rate is a response rate and deserves the same scrutiny. (Constrained-decoding libraries such as Outlines and Guidance, and the structured-output modes of the provider APIs, remove this particular missingness by restricting generation to a grammar, which converts a parse failure into a forced parse rather than into a recorded one.)

Verbalized confidence is an elicited subjective probability, so score it before you use it. Group the model’s stated probabilities and compare each group’s stated value with the fraction actually correct.

import numpy as np
from sklearn.isotonic import IsotonicRegression
rng = np.random.default_rng(0)
n = 2000
stated = rng.choice([0.5, 0.6, 0.7, 0.8, 0.9, 0.95, 0.99], n)      # the model's verbalized probability
true_p = np.clip(0.30 + 0.55 * (stated - 0.5) / 0.49, 0, 1)        # what it is actually worth
y = rng.random(n) < true_p
print(" stated   n    empirical accuracy")
for s in [0.5, 0.7, 0.9, 0.95, 0.99]:
    m = stated == s
    print(f"  {s:.2f}  {m.sum():4d}      {y[m].mean():.3f}")
brier = np.mean((stated - y)**2)
iso = IsotonicRegression(out_of_bounds="clip").fit(stated[:1000], y[:1000])
print(f"Brier score as stated      = {brier:.4f}")
print(f"Brier after isotonic recal = {np.mean((iso.predict(stated[1000:]) - y[1000:])**2):.4f}")
print(f"mean stated {stated.mean():.3f} vs mean correct {y.mean():.3f}: overconfident by {stated.mean()-y.mean():.3f}")

 stated   n    empirical accuracy
  0.50   258      0.314
  0.70   286      0.517
  0.90   299      0.766
  0.95   300      0.813
  0.99   295      0.864
Brier score as stated      = 0.2229
Brier after isotonic recal = 0.1973
mean stated 0.784 vs mean correct 0.629: overconfident by 0.155

Every bin is overconfident, and the gap widens at the top, where the reported probability is most likely to be believed. Isotonic recalibration on a held-out half improves the Brier score without any change to the model, which is the same conclusion the calibration literature reaches for classifier scores: an uncalibrated probability is a ranking, and it becomes a probability only after you fit the map.

Iterating on a prompt against a scored set is model selection, and the winner’s score is a maximum over dependent candidates. Twenty prompts are evaluated on 100 items; nineteen are equally good and one is genuinely slightly better.

import numpy as np
rng = np.random.default_rng(0)
n, K = 100, 20                                             # 20 candidate prompts, 100 items
truth = np.full(K, 0.70); truth[0] = 0.74                  # prompt 0 is genuinely the best
dev = rng.binomial(n, truth) / n
win = int(dev.argmax())
print(f"winning prompt on dev   = {win}, score {dev.max():.3f}")
print(f"its true accuracy       = {truth[win]:.3f}")
print(f"selection bias          = {dev.max() - truth[win]:+.3f}")
print(f"re-scored on fresh items= {rng.binomial(n, truth[win]) / n:.3f}")
print(f"the truly best prompt (0) ranked {int((-dev).argsort().tolist().index(0)) + 1} of {K} on dev")

winning prompt on dev   = 18, score 0.770
its true accuracy       = 0.700
selection bias          = +0.070
re-scored on fresh items= 0.710
the truly best prompt (0) ranked 7 of 20 on dev

The reported “best” score overstates its own accuracy by seven points, and the prompt that is truly best did not even win. This is the winner’s curse on a validation set of the size people actually use. The fix is the ordinary one: hold out a set that is touched exactly once, and report the winner’s score from that set, not from the set used to pick it. (Frameworks that search prompts automatically, such as DSPy Khattab et al., 2023, make this problem larger rather than smaller, because they raise the number of candidates evaluated by orders of magnitude while leaving the held-out set the same size.)

import numpy as np
rng = np.random.default_rng(8)

# Toy in-context learner: estimate the slope from k examples, then predict.
def experiment(k, trials=4000, noise=0.3, beta=1.7):
    err = np.empty(trials)
    for t in range(trials):
        x = rng.normal(size=k) if k else np.zeros(0)
        y = beta * x + noise * rng.normal(size=k) if k else np.zeros(0)
        bhat = (x @ y) / (x @ x) if k and (x @ x) > 1e-9 else 0.0
        xq = rng.normal()
        err[t] = (bhat * xq - beta * xq) ** 2
    return err

print(f"{'k':>3} {'mean sq. err':>13} {'median':>9} {'90th pct':>10}")
for k in [0, 1, 2, 4, 8, 16]:
    e = experiment(k)
    print(f"{k:>3} {e.mean():>13.4f} {np.median(e):>9.4f} {np.quantile(e, 0.9):>10.4f}")
print("\nThe median falls steadily with k. The MEAN at k=1 is enormous: with one example")
print("the slope estimate divides by x^2, which is occasionally near zero. One shot is")
print("not a small improvement on zero shots -- it is a high-variance gamble.")

  k  mean sq. err    median   90th pct
  0        3.0105    1.3827     8.1336
  1       35.2542    0.0277     2.2483
  2        0.2566    0.0090     0.2636
  4        0.0410    0.0037     0.0893
  8        0.0149    0.0016     0.0352
 16        0.0068    0.0008     0.0162

The median falls steadily with k. The MEAN at k=1 is enormous: with one example
the slope estimate divides by x^2, which is occasionally near zero. One shot is
not a small improvement on zero shots -- it is a high-variance gamble.

1Tools in Practice¶

Prompting stops being a craft and starts being a method at the point where the prompt, the decoding parameters, the item set and the score are all under version control. The tool categories below are the ones that make that transition possible. Which product occupies each category changes quickly; the categories, and the failure each one is meant to prevent, do not.

[Chat templating in the model runtime] Input construction. Turn a list of role-tagged messages into the exact token sequence the model was tuned on --- for example the apply_chat_template call of the Hugging Face transformers tokenizer. Fits: the step between your text and the model’s input, for local and open-weight models. Watch: the template is a per-model convention, not a standard. Hand-rolling the role markers degrades output silently, which looks like a bad prompt.
[Constrained decoding and structured output] Measurement scale. Restrict generation to a JSON schema, regular expression or grammar --- libraries such as Outlines and Guidance, and the structured-output modes exposed by provider APIs. Fits: anywhere a response must become a column in a data frame. Watch: it guarantees the form, never the content, and it converts the informative non-response of the format-instruction example above into a compliant answer whose value you have not checked.
[Prompt-programming frameworks] Automated specification search. Express a pipeline declaratively and let a compiler search over instructions and exemplars against a metric you supply Khattab et al., 2023. Fits: replacing hand-iteration once you have an item set worth optimizing against. Watch: the optimism of Example prompt_engineering-10 scales with the number of candidates searched; an automated search needs a genuinely untouched test set, not a validation set it has already seen.
[Evaluation harnesses] Repeatable scoring. Run a fixed prompt set against one or more models and emit per-item scores --- promptfoo for hand-written suites, EleutherAI’s lm-evaluation-harness for standard task batteries. Fits: the comparison step, and any claim of the form “prompt A beats prompt B”. Watch: report per-item results, not just the aggregate. Paired scoring on identical items is the cheapest variance reduction available and most harnesses give it to you for free if you keep the item identifiers.
[Tracing and experiment tracking] Provenance. Log every call with its prompt, parameters, model identifier, latency and response --- the LLM tracing features in MLflow and Weights & Biases, or an equivalent local log. Fits: everything you intend to write up. Watch: a trace is only as good as what it pins. If the model identifier and the seed are not in the record, the run is not reproducible no matter how much else was logged.

Prompting an agent that searches, reads and synthesizes over many steps is a different problem, because the thing being conditioned on is chosen by the agent rather than by you; Section that section treats it separately.

The reason to be careful with an automated prompt search is not that it works badly but that it works well enough to overfit. Fix the item set at 100 questions, let candidate prompts differ only slightly in true quality, and watch what the winner’s development score does as the search widens.

import numpy as np
rng = np.random.default_rng(0)
n_dev, n_test = 100, 100
sigma = 0.02                                       # spread of true prompt quality
print("candidates K   dev score of winner   its true accuracy   optimism   held-out test score")
for K in [5, 20, 100, 500]:
    opt, dev_w, tru_w, tst_w = [], [], [], []
    for _ in range(400):
        truth = np.clip(rng.normal(0.70, sigma, K), 0, 1)   # candidate prompts differ a little
        dev = rng.binomial(n_dev, truth) / n_dev            # scored on the dev set
        w = dev.argmax()
        test = rng.binomial(n_test, truth[w]) / n_test      # a set touched exactly once
        dev_w.append(dev[w]); tru_w.append(truth[w]); tst_w.append(test)
        opt.append(dev[w] - truth[w])
    print(f"{K:12d}   {np.mean(dev_w):18.4f}   {np.mean(tru_w):17.4f}   {np.mean(opt):+9.4f}"
          f"   {np.mean(tst_w):19.4f}")
best = np.clip(rng.normal(0.70, sigma, 500), 0, 1)
print(f"expected best true accuracy among 500 candidates = {best.max():.4f}")
print("optimism grows with the number of candidates searched; the test-set score does not")

candidates K   dev score of winner   its true accuracy   optimism   held-out test score
           5               0.7572              0.7093     +0.0479                0.7095
          20               0.7884              0.7151     +0.0734                0.7147
         100               0.8187              0.7205     +0.0982                0.7220
         500               0.8422              0.7246     +0.1176                0.7223
expected best true accuracy among 500 candidates = 0.7605
optimism grows with the number of candidates searched; the test-set score does not

Widening the search from 5 candidates to 500 raises the winner’s development score by almost 9 points while raising its true accuracy by 1.5. Almost all of the apparent improvement is optimism, and the untouched test set reports it correctly at every $K$ . The search is still doing something --- true accuracy does rise, because there really are better prompts to find --- but the development score is a maximum over $K$ dependent estimates and overstates the gain by a factor that grows with $K$ . This is the multiple-comparison arithmetic of any specification search, and the remedy is the same: report the winner on data the search never touched, and report $K$ so the reader can judge how much selection there was.

References¶

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language Models are Unsupervised Multitask Learners [Techreport]. OpenAI.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., … Amodei, D. (2020). Language Models are Few-Shot Learners.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., & Lowe, R. (2022). Training language models to follow instructions with human feedback.
Breiman, L. (1996). Bagging Predictors. Machine Learning, 24, 123–140. 10.1007/BF00058655
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2020). The Curious Case of Neural Text Degeneration.
Gneiting, T., & Raftery, A. E. (2007). Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association, 102(477), 359–378. 10.1198/016214506000001437
Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On Calibration of Modern Neural Networks. Proceedings of the 34th International Conference on Machine Learning (ICML), 1321–1330.
Lester, B., Al-Rfou, R., & Constant, N. (2021). The Power of Scale for Parameter-Efficient Prompt Tuning. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Li, X. L., & Liang, P. (2021). Prefix-Tuning: Optimizing Continuous Prompts for Generation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL).
Khattab, O., Singhvi, A., Maheshwari, P., Zhang, Z., Santhanam, K., Vardhamanan, S., Haq, S., Sharma, A., Joshi, T. T., Moazam, H., Miller, H., Zaharia, M., & Potts, C. (2023). DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines. arXiv Preprint arXiv:2310.03714.