Retrieval-Augmented Generation

import numpy as np
rng = np.random.default_rng(2)

# 200 chunks; exactly 3 contain the answer. Retrieval returns top-k by score.
n, n_rel, k = 200, 3, 5
relevant = set(rng.choice(n, n_rel, replace=False))

def trial(signal):
    """signal = how much higher relevant chunks score than distractors."""
    scores = rng.normal(size=n)
    for i in relevant:
        scores[i] += signal
    top = set(np.argsort(-scores)[:k])
    return len(top & relevant)

for signal in [0.0, 1.0, 2.0, 3.0]:
    hits = np.array([trial(signal) for _ in range(2000)])
    print(f"signal={signal:.1f}  recall@{k}={hits.mean()/n_rel:5.1%}  "
          f"P(no relevant chunk retrieved)={np.mean(hits == 0):5.1%}")

signal=0.0  recall@5= 2.4%  P(no relevant chunk retrieved)=92.8%
signal=1.0  recall@5=14.9%  P(no relevant chunk retrieved)=60.4%
signal=2.0  recall@5=47.4%  P(no relevant chunk retrieved)=11.7%
signal=3.0  recall@5=78.1%  P(no relevant chunk retrieved)= 0.7%

import numpy as np

doc_len, ans_start, ans_len = 4000, 2570, 220        # the answer spans 2570-2790
ans_end = ans_start + ans_len

print(f"{'chunk':>6} {'chunks hit':>11} {'best single':>12} {'context if top-1':>17}")
for chunk in [100, 200, 500, 1000, 2000]:
    starts = np.arange(0, doc_len, chunk)
    hits = [(s, min(s + chunk, doc_len)) for s in starts
            if s < ans_end and s + chunk > ans_start]
    # how much of the answer the SINGLE best chunk contains
    best = max(min(e, ans_end) - max(s, ans_start) for s, e in hits)
    print(f"{chunk:>6} {len(hits):>11} {best:>7}/{ans_len:<4} {chunk:>17}")
print("\nSmall chunks send less irrelevant text but split the answer across several;")
print("large chunks keep it whole and spend context on the surrounding page.")

 chunk  chunks hit  best single  context if top-1
   100           3     100/220                100
   200           2     190/220                200
   500           1     220/220                500
  1000           1     220/220               1000
  2000           1     220/220               2000

Small chunks send less irrelevant text but split the answer across several;
large chunks keep it whole and spend context on the surrounding page.

1Retrieval Augmented Generation¶

Retrieval-augmented generation is the practice of retrieving relevant documents and placing them in the model’s context before asking it to answer, so that the response is conditioned on evidence you supplied rather than on whatever the weights happen to encode. For a statistician this is a change of estimand: the model is no longer being asked for a marginal prediction from its training distribution but for a conditional one given a small, auditable set of documents. That reframing is what makes the technique interesting for scientific work. It gives you provenance --- each claim can be traced to a retrieved passage --- and it lets a general-purpose model operate on a corpus it never saw, whether that is your lab’s protocols, a regulatory codebook, or last month’s literature. It is also the most common thing a researcher actually builds with an LLM, and the most commonly built badly, because almost all of the performance is determined by the retrieval step and almost all of the attention goes to the prompt. This section develops RAG as a two-stage estimator, makes the error decomposition explicit, and treats evaluation with the seriousness the failure modes deserve.

1.1RAG as a Two-Stage Estimator¶

Write the answer distribution as a marginal over which documents were retrieved. With retriever $p_\eta(z \mid x)$ over passages $z$ in a corpus $\mathcal{Z}$ and generator $p_\theta(y \mid x, z)$,

where $\mathcal{Z}_k(x)$ is the retrieved top-$k$ and $\tilde p_\eta$ is the renormalized retriever score. This is a mixture model whose mixing distribution is estimated, and the standard practice of concatenating all $k$ passages into one prompt is a further approximation --- it replaces the mixture over documents with a single evaluation conditioned on their union. Naming the approximation is useful because it tells you exactly what is being ignored: the uncertainty in which documents are relevant.

Equation the equation is a mixture over which documents were retrieved, and concatenating the top-$k$ into one prompt replaces that mixture with a single evaluation. When the retrieved passages disagree, the two are not interchangeable.

import numpy as np
rng = np.random.default_rng(0)
val = np.array([0.20, 0.55, 0.90])                         # the number each passage implies
w = np.array([0.50, 0.30, 0.20])                           # renormalized retriever scores
draw = val[rng.choice(3, 20000, p=w)]                      # honest mixture over retrievals
print(f"mixture over z: mean {draw.mean():.4f}  SD {draw.std():.4f}"
      f"  95% range [{np.quantile(draw,.025):.2f}, {np.quantile(draw,.975):.2f}]")
print(f"single call on the concatenated top-3: one number, no interval")
print(f"share of retrievals disagreeing with the modal passage: {(draw != 0.20).mean():.4f}")
sens = [val[np.argsort(-w)[:k]] @ (w[np.argsort(-w)[:k]] / w[np.argsort(-w)[:k]].sum()) for k in [1, 2, 3]]
print("sensitivity to k (the cheapest available check):", np.round(sens, 4))
print(f"answer moves {max(sens)-min(sens):.4f} across k while the reported point estimate carries no SE")

mixture over z: mean 0.4466  SD 0.2734  95% range [0.20, 0.90]
single call on the concatenated top-3: one number, no interval
share of retrievals disagreeing with the modal passage: 0.5033
sensitivity to k (the cheapest available check): [0.2    0.3312 0.445 ]
answer moves 0.2450 across k while the reported point estimate carries no SE

The mixture puts half its mass on an answer that contradicts the modal passage; the concatenated call reports one number with no indication that the evidence was split, and the answer moves by 0.245 as $k$ goes from 1 to 3. Neither number is wrong --- they estimate different things --- but only the mixture exposes stage-one uncertainty, which is the quantity a two-stage estimator is supposed to propagate.

1.2Where the Error Comes From¶

A useful accounting splits end-to-end accuracy by whether the evidence was retrieved at all. Let $r$ be the probability that the top-$k$ contains a passage supporting the answer, $p_{\text{g}}$ the probability of a correct answer given that it did, and $p_{\text{u}}$ the probability of a correct answer when it did not. Then

which makes the division of labour explicit: the retriever sets the ceiling and the generator sets the slope. Most disappointing RAG systems have a retrieval problem, and most engineering effort goes into the prompt, which can only move $p_{\text{g}}$. The decomposition also explains a counterintuitive observation: a weaker model with a stronger prior can score better at low $r$, because $p_{\text{u}}$ is doing the work --- and it is doing it by answering from memory, which is precisely what you deployed RAG to avoid. A second effect complicates the picture further: accuracy depends on where in the context the supporting passage sits, with evidence placed in the middle of a long context used less reliably than evidence at either end Liu et al., 2023. That is a position effect of exactly the kind survey statisticians control for by randomizing item order, and it is a reason to keep contexts short and to check sensitivity to ordering. Equation the equation has a break-even point, and it is worth computing because it shows retrieval can hurt. Set grounded accuracy at 0.85, closed-book accuracy at 0.45, and let irrelevant retrieved context depress the model to 0.20.

import numpy as np
pg, pu = 0.85, 0.45                                        # grounded and closed-book accuracy
print("   r     RAG accuracy   closed-book   RAG better?")
for r in [0.0, 0.2, 0.4, 0.55, 0.6, 0.8, 1.0]:
    acc = r * pg + (1 - r) * 0.20                          # unsupported context distracts: 0.20
    print(f"{r:5.2f}      {acc:.3f}          {pu:.3f}         {str(acc > pu):>5s}")
print(f"break-even recall r* = (pu - pdistract)/(pg - pdistract) = {(pu-0.20)/(pg-0.20):.3f}")
print("below that recall, retrieval is worse than answering from memory")

   r     RAG accuracy   closed-book   RAG better?
 0.00      0.200          0.450         False
 0.20      0.330          0.450         False
 0.40      0.460          0.450          True
 0.55      0.557          0.450          True
 0.60      0.590          0.450          True
 0.80      0.720          0.450          True
 1.00      0.850          0.450          True
break-even recall r* = (pu - pdistract)/(pg - pdistract) = 0.385
below that recall, retrieval is worse than answering from memory

Below a recall of about 0.385 the retrieval-augmented system is worse than the same model answering from memory, because irrelevant context displaces what the model knew. The threshold is a ratio of accuracy gaps, so it rises whenever the model has a strong prior --- which is exactly the regime where RAG is most often deployed without measuring $r$.

Because recall rises with $k$ while a longer context is harder to use, end-to-end accuracy is a product of an increasing and a decreasing factor. Sweeping $k$ shows the shape that the $r p_{\text{g}}$ term of the equation predicts.

import numpy as np
rng = np.random.default_rng(0)
nq = 4000
print("  k   recall@k   P(correct | retrieved)   end-to-end accuracy")
for k in [1, 2, 3, 5, 10, 20, 40]:
    r = 1 - 0.72**k                                        # more passages, better recall
    pg = 0.92 * 0.97**(k - 1)                              # ...but a longer, noisier context
    got = rng.random(nq) < r
    acc = np.where(got, rng.random(nq) < pg, rng.random(nq) < 0.18).mean()
    print(f"{k:3d}     {r:.3f}            {pg:.3f}                 {acc:.3f}")
print("recall is monotone in k; accuracy is not -- it peaks and then decays as context is stuffed")

  k   recall@k   P(correct | retrieved)   end-to-end accuracy
  1     0.280            0.920                 0.384
  2     0.482            0.892                 0.511
  3     0.627            0.866                 0.615
  5     0.807            0.814                 0.683
 10     0.963            0.699                 0.676
 20     0.999            0.516                 0.525
 40     1.000            0.280                 0.307
recall is monotone in k; accuracy is not -- it peaks and then decays as context is stuffed

Recall is monotone in $k$ and accuracy is not: it peaks around $k = 5$ and then falls, even though the evidence is present more often. Reporting recall@$k$ alone therefore recommends the wrong $k$. This is a bias--variance trade-off in disguise, with context length as the complexity parameter and the optimum well short of “retrieve everything”.

Position effects are measurable without any new data: hold the retrieved passages fixed and permute their order. Evidence in the middle of the context is used less often than evidence at either end, so accuracy depends on an ordering nobody reported.

import numpy as np
rng = np.random.default_rng(0)
nq, k, B = 300, 5, 10                                      # 5 passages, 10 orderings each
use = np.array([0.90, 0.68, 0.62, 0.66, 0.86])             # P(use evidence) by slot: ends beat middle
u = rng.random(nq)                                         # per-question propensity, fixed across orderings
ans = np.array([u < use[rng.integers(0, k, nq)] for _ in range(B)]).T
per = ans.mean(0)
print("accuracy by ordering:", np.round(per, 3))
print(f"mean {per.mean():.3f}   SD across orderings {per.std(ddof=1):.4f}   range {per.max()-per.min():.3f}")
print(f"binomial SE from a single ordering at n=300: {np.sqrt(per.mean()*(1-per.mean())/nq):.4f}")
print(f"questions whose answer flips on reordering alone: {(ans.min(1) != ans.max(1)).mean():.3f}")
print(f"slot means over all runs: {np.round(use, 2)} -- the middle three are the penalty")

accuracy by ordering: [0.647 0.677 0.72  0.673 0.687 0.73  0.653 0.71  0.703 0.68 ]
mean 0.688   SD across orderings 0.0275   range 0.083
binomial SE from a single ordering at n=300: 0.0267
questions whose answer flips on reordering alone: 0.280
slot means over all runs: [0.9  0.68 0.62 0.66 0.86] -- the middle three are the penalty

Accuracy ranges over 8 points across orderings of the same evidence, and the spread across orderings is as large as the binomial standard error from a single run --- so a system evaluated once has an unreported source of variability comparable to its sampling error. Twenty-eight percent of questions flip on order alone. Randomize passage order across evaluation items and report the spread, exactly as a survey statistician rotates item order.

1.3Chunking Is a Design Decision¶

Splitting documents into retrievable units determines what can be retrieved, and it is made before any modelling. Chunks that are too small lose the context that makes a passage interpretable; chunks that are too large dilute the embedding, so that a single relevant sentence is averaged away by surrounding text. The practical compromise --- moderate chunks with overlap, plus a stored parent pointer so a retrieved chunk can be expanded to its section --- is worth stating explicitly because it is rarely written down. (Orchestration frameworks such as LangChain and LlamaIndex ship this pattern under names like “parent document retriever” and “recursive splitter”; adopting one does not relieve you of choosing the unit, it only supplies a default you did not choose.)

Chunking is the choice of the unit of analysis, and its effect on retrieval is mechanical: a chunk embedding is roughly an average of its sentences, so a long chunk averages the answer together with whatever surrounds it. One sentence answers the query; only the chunk size changes.

import numpy as np
rng = np.random.default_rng(0)
d, T, S = 16, 6, 10                                        # 6 topics of 10 sentences, in order
cent = rng.normal(0, 1, (T, d))
cent[0] = -cent[1]                                         # topic 0 opposes topic 1: averaging cancels
cent[4] = 0.90 * cent[1] + 0.44 * rng.normal(0, 1, d)      # topics 4 and 5 sit near topic 1
cent[5] = 0.90 * cent[1] + 0.44 * rng.normal(0, 1, d)
sent = np.repeat(cent, S, axis=0) * 2.2 + rng.normal(0, 1, (T * S, d))
key = 17                                                    # the sentence answering the query
q = sent[key] + 0.9 * rng.normal(0, 1, d)
unit = lambda A: A / np.linalg.norm(A, axis=-1, keepdims=True)
print(" chunk size   #chunks   rank of the chunk holding the answer   its cosine")
for cs in [1, 2, 5, 10, 20, 30]:
    ch = sent.reshape(-1, cs, d).mean(1)                    # a chunk embedding averages its sentences
    s = unit(ch) @ unit(q)
    hit = key // cs
    print(f"{cs:9d}   {len(ch):7d}   {int(np.where(np.argsort(-s) == hit)[0][0]) + 1:36d}   {s[hit]:10.3f}")
print("once a chunk straddles the topic boundary the answer is averaged away and a coherent neighbour wins")

 chunk size   #chunks   rank of the chunk holding the answer   its cosine
        1        60                                      1        0.888
        2        30                                      1        0.847
        5        12                                      1        0.808
       10         6                                      1        0.773
       20         3                                      2        0.533
       30         2                                      2        0.264
once a chunk straddles the topic boundary the answer is averaged away and a coherent neighbour wins

Cosine to the answer-bearing chunk falls monotonically, and past 20 sentences the correct chunk is no longer ranked first --- it is beaten by a chunk that is uniformly on-topic but contains nothing. Choosing the retrieval unit is the same decision as choosing between patient-, visit-, and measurement-level analysis, and like that decision it is made once, silently, and determines what can be found.

1.4Evaluating a RAG System¶

Evaluation must be layered, because the two stages fail differently. At the retrieval layer, use the ranking measures of the previous section on a set of questions with known supporting passages. At the generation layer, score faithfulness --- whether each assertion in the answer is entailed by the retrieved context --- separately from correctness, since the two come apart in both directions. A faithful summary of an irrelevant passage is wrong; a correct answer the model produced from memory while ignoring the context is right for the wrong reason and will not generalize to your next corpus. If a model is used as the judge of faithfulness, its agreement with human adjudication must itself be estimated on a subsample, with a $\kappa$ or a simple agreement rate. (Packaged RAG evaluation suites such as RAGAS Es et al., 2023 compute faithfulness, answer relevance and context precision for you, but they compute them with a model judge, so the agreement estimate is a prerequisite for reading their output, not an optional extra.)

Faithfulness and correctness are different measurements and they come apart in both directions. Simulating 500 answers with known support and known truth gives the joint distribution that a single accuracy number hides.

import numpy as np
rng = np.random.default_rng(0)
n = 500
supported = rng.random(n) < 0.70                           # the context entails the claim
true_fact = rng.random(n) < 0.65                           # the claim is actually true
faithful = np.where(supported, rng.random(n) < 0.90, rng.random(n) < 0.25)
correct = np.where(faithful, true_fact, rng.random(n) < 0.30)
print(f"faithfulness rate = {faithful.mean():.3f}")
print(f"correctness rate  = {correct.mean():.3f}")
print(f"faithful AND wrong (right answer to a wrong document) = {np.mean(faithful & ~correct):.3f}")
print(f"unfaithful AND right (answered from memory)           = {np.mean(~faithful & correct):.3f}")
tab = np.array([[np.sum(faithful & correct), np.sum(faithful & ~correct)],
                [np.sum(~faithful & correct), np.sum(~faithful & ~correct)]])
print("2x2 table [faithful/not] x [correct/not]:", tab.tolist())
print(f"correlation between the two labels = {np.corrcoef(faithful, correct)[0,1]:.3f}")

faithfulness rate = 0.688
correctness rate  = 0.522
faithful AND wrong (right answer to a wrong document) = 0.266
unfaithful AND right (answered from memory)           = 0.100
2x2 table [faithful/not] x [correct/not]: [[211, 133], [50, 106]]
correlation between the two labels = 0.272

A quarter of the answers are faithful and wrong --- accurate summaries of an unhelpful passage --- and a tenth are correct while ignoring the context, which is the model answering from memory and will not survive a change of corpus. The two labels correlate at only 0.27, so neither is a proxy for the other, and a system reported on correctness alone may be right for a reason that does not generalize.

If a language model adjudicates faithfulness, its agreement with human judgement is itself an estimate with a standard error. Score the same items both ways and compute $\kappa$ on subsamples of the size people actually adjudicate.

import numpy as np
rng = np.random.default_rng(0)
n = 300
truth = rng.random(n) < 0.70                               # the claim really is supported
judge = np.where(truth, rng.random(n) < 0.88, rng.random(n) < 0.22)   # LLM judge
human = np.where(truth, rng.random(n) < 0.93, rng.random(n) < 0.08)   # human adjudication
for m in [30, 100, 300]:
    a, b = judge[:m], human[:m]
    po = (a == b).mean()
    pe = a.mean() * b.mean() + (1 - a.mean()) * (1 - b.mean())
    kap, se = (po - pe) / (1 - pe), np.sqrt(po * (1 - po) / m) / (1 - pe)
    print(f"m={m:3d}  agreement {po:.3f}  kappa {kap:.3f}  approx 95% CI [{kap-1.96*se:.3f}, {kap+1.96*se:.3f}]")
print(f"judge-scored faithfulness {judge.mean():.3f} vs human-scored {human.mean():.3f}")
print("the judge's own error rate must be estimated before its score can be reported")

m= 30  agreement 0.833  kappa 0.634  approx 95% CI [0.341, 0.927]
m=100  agreement 0.820  kappa 0.610  approx 95% CI [0.446, 0.773]
m=300  agreement 0.830  kappa 0.635  approx 95% CI [0.543, 0.726]
judge-scored faithfulness 0.623 vs human-scored 0.640
the judge's own error rate must be estimated before its score can be reported

At $m = 30$ --- a typical hand-checked subsample --- the interval on $\kappa$ runs from 0.34 to 0.93, which is compatible with anything from fair to near-perfect agreement. The judge’s headline faithfulness rate differs from the human rate by only two points, but that comparison means nothing until the agreement is pinned down, and pinning it down takes a few hundred adjudicated items.

The arithmetic of a small evaluation set determines what a comparison can conclude, and scoring both systems on identical questions is the cheapest variance reduction available.

import numpy as np
rng = np.random.default_rng(0)
Q = 50
print("questions   accuracy A   accuracy B   unpaired 95% CI on diff   paired 95% CI on diff")
for Q in [50, 200]:
    hard = rng.random(Q) < 0.4                             # shared difficulty across systems
    a = np.where(hard, rng.random(Q) < 0.30, rng.random(Q) < 0.85)
    b = np.where(hard, rng.random(Q) < 0.40, rng.random(Q) < 0.92)
    d = b.astype(float) - a
    up = 1.96 * np.sqrt(a.mean()*(1-a.mean())/Q + b.mean()*(1-b.mean())/Q)
    pp = 1.96 * d.std(ddof=1) / np.sqrt(Q)
    print(f"{Q:9d}   {a.mean():10.3f}   {b.mean():10.3f}   [{d.mean()-up:+.3f}, {d.mean()+up:+.3f}]"
          f"          [{d.mean()-pp:+.3f}, {d.mean()+pp:+.3f}]")
print(f"n for a paired half-width of 0.05 at this SD: {int(np.ceil((1.96*d.std(ddof=1)/0.05)**2))}")

questions   accuracy A   accuracy B   unpaired 95% CI on diff   paired 95% CI on diff
       50        0.580        0.780   [+0.021, +0.379]          [+0.052, +0.348]
      200        0.660        0.725   [-0.025, +0.155]          [-0.009, +0.139]
n for a paired half-width of 0.05 at this SD: 434

With 50 questions the unpaired interval on the difference is 0.36 wide; the paired interval is narrower because shared question difficulty cancels. At 200 questions the difference between the systems is no longer distinguishable from zero under either analysis --- the 50-question result was a small sample flattering a small effect. Reaching a paired half-width of 5 points here needs about 434 questions, an order of magnitude more than a typical golden set.

1.5Tools in Practice¶

A RAG system has more moving parts than anything else in this chapter, and correspondingly more places to accept a default without noticing. The categories below follow the pipeline of Figure the figure from left to right, ending with the layer that most implementations skip.

• [Document extraction and layout parsing] Corpus preparation. Convert PDFs and office documents into text with structure preserved --- reading order, column boundaries, table cells, figure captions. Fits: the offline path along the bottom of Figure the figure, before chunking. Watch: this is where scientific corpora fail first, and it fails silently: a two-column paper read in raw line order interleaves the columns, and a table read without cell coordinates pairs labels with the wrong numbers. No retrieval metric in this section can detect either, as the example below shows.

• [Chunking and orchestration frameworks] Pipeline assembly. LangChain and LlamaIndex supply splitters, retrievers, parent-document expansion and the glue between index and generator. Fits: getting a working pipeline quickly, and standardizing one across a group. Watch: every default is a decision you did not make --- chunk size, overlap, $k$, the fusion rule. Read them out of the configuration and record them next to your results; they move accuracy more than the choice of generator does.

• [Hybrid retrieval and fusion] Stage-one recall. Run a dense and a lexical ranker Robertson & Zaragoza, 2009 and fuse by reciprocal rank, as in Section that section. Fits: corpora containing identifiers, which in a statistical or biomedical setting is all of them. Watch: fusion needs no score calibration, which is its main virtue, but it does need both rankers to be evaluated separately. A fused system whose lexical half is broken looks merely mediocre rather than broken.

• [Citation-bearing generation] Attribution. Prompting or decoding patterns that require each sentence to name the passage supporting it, so the output can be audited against the retrieved context. Fits: anything you will quote in writing. Watch: a passage identifier attached to a sentence is a claim, not a verification. The mechanism to check it is the same one used for citations in a research report; see Section that section.

• [RAG evaluation suites] Layered scoring. Frameworks that score faithfulness, answer relevance and context precision on a question set Es et al., 2023. Fits: the layered evaluation this section argues for, once you have a question set with known supporting passages. Watch: the scorer is a model. Its agreement with human adjudication is an unknown parameter of your evaluation, and Example retrieval_augmented_generation-7 shows how many adjudicated items it takes to pin down.

• [Tracing and per-item logging] Provenance. Record, for every question, the retrieved passage identifiers, their scores, the assembled prompt and the answer. Fits: the debugging loop, and the paired comparisons of Example retrieval_augmented_generation-8. Watch: without per-item retrieval logs you cannot compute $r$ in the equation, and without $r$ you cannot tell whether a disappointing system has a retrieval problem or a generation problem --- which is the first question to answer and the one an aggregate score never answers.

An agent that runs its own multi-step search over an open corpus, rather than answering from a fixed retrieved context, changes the estimand again and introduces a verification problem of its own; Section that section treats it.

The extraction layer deserves a measurement of its own because its errors are invisible downstream. Consider a results table read by a parser that recovers the text but not the cell coordinates: it emits a stream of row labels and a stream of numbers, and pairs them by position. A single label that wraps onto two printed lines inserts one spurious label and shifts every subsequent pairing by one.

import numpy as np
rng = np.random.default_rng(0)
R = 24                                              # a results table with 24 rows
labels = [f"row{r:02d}" for r in range(R)]
values = np.round(rng.normal(0.5, 0.2, R), 3)
def extract(p_wrap):
    # A layout-unaware reader emits a label stream and a value stream, then pairs them
    # by position.  A label that wraps onto two printed lines adds a spurious label.
    lab = []
    for r in range(R):
        lab.append(labels[r])
        if rng.random() < p_wrap:
            lab.append("(cont.)")                   # the wrapped remainder of a long label
    return list(zip(lab, list(values) + [np.nan] * (len(lab) - R)))
print(" P(label wraps)   pairs recovered   correctly paired   first row misread")
for p in [0.0, 0.05, 0.15, 0.40]:
    acc, first = [], []
    for _ in range(2000):
        pairs = extract(p)
        ok = np.array([l == labels[k] and v == values[k]
                       for k, (l, v) in enumerate(pairs[:R])])
        acc.append(ok.mean())
        first.append(R if ok.all() else int(np.argmin(ok)))
    print(f"{p:14.2f}   {R:15d}   {np.mean(acc):16.3f}   {np.mean(first):17.1f}")
print("a layout-aware extractor keeps cell coordinates, so pairing is exact at every wrap rate")

 P(label wraps)   pairs recovered   correctly paired   first row misread
          0.00                24              1.000                24.0
          0.05                24              0.592                14.2
          0.15                24              0.275                 6.6
          0.40                24              0.106                 2.5
a layout-aware extractor keeps cell coordinates, so pairing is exact at every wrap rate

A wrap probability of 0.05 --- one label in twenty running long, which is unremarkable in a real results table --- leaves only 59% of label--value pairs correct, and the first error arrives around row 14. At 0.15 the majority of the table is wrong. The number of extracted pairs is right in every row of the output, so a pipeline that counts records rather than checking them reports full success. Nothing in the retrieval evaluation of this section detects this: the passage is retrieved, it is on topic, and the number inside it belongs to a different row.

Two lessons follow. The first is that extraction quality belongs in the error decomposition of the equation alongside $r$, because a passage that is retrieved but corrupted contributes to $1 - p_{\text{g}}$ for a reason no amount of prompt work can repair. The second is procedural: validate the extractor before evaluating the system, by hand-checking a random sample of extracted records against the source document and reporting that agreement rate. It is the same instrument-validation step you would insist on before analysing data from an automated assay, and for the same reason --- the downstream analysis cannot see the instrument’s error, so it will report it as a substantive finding.

1.6Exercises¶

1. Starting from the equation, show that concatenating the top-$k$ passages into a single prompt is not in general equal to the mixture, and describe a situation --- two retrieved passages that contradict each other --- where the two give qualitatively different answers.

2. Using the equation, derive the condition on $p_{\text{g}}$, $p_{\text{u}}$ and $r$ under which adding retrieval makes accuracy worse than a closed-book model. Interpret it in one sentence.

3. You measure recall@5 $=0.62$ on 40 evaluation questions. Give a 95% interval for the true recall. How many questions would you need for a half-width of 5 percentage points?

4. Computational. Build a small RAG system over 200 abstracts of your choosing. Sweep $k \in \{1,3,5,10,20\}$ and plot both recall@$k$ and end-to-end answer accuracy on a fixed question set. Identify the $k$ at which accuracy stops improving and explain the gap between the two curves using the equation.

5. Computational. Take a fixed retrieved context of five passages and permute their order ten times, holding everything else constant. Report the distribution of answers. Does position affect the result, and how would you report that sensitivity in a paper?