Literature Discovery and Synthesis

This section covers the first half of a literature workflow: finding the papers, seeing how they relate, and pulling structured information out of them. Every tool below does useful work, and every one of them introduces a measurement step that a statistician is unusually well equipped to audit. The organising idea is that a discovery pipeline is a screening instrument. It has a sampling frame, a sensitivity, a precision and a cost per unit of yield, and none of those are reported to you by default. The advice here is therefore not “use tool $X$” but “instrument your search the way you would instrument any other measurement”. Interfaces and capabilities change every few months, so check current documentation for what a tool can do today and treat the descriptions below as descriptions of a job. Report generators that write a synthesised narrative with inline citations are covered separately in Section that section, and the verification machinery developed there applies to everything in this section.

1Query-and-extract assistants¶

1.1Elicit¶

• [Elicit] Query-and-extract assistant. Turns a research question into a table whose rows are papers and whose columns are fields extracted from them. Fits: scoping a question, and first-pass extraction before full-text reading. Watch: a filled cell is a model’s reading of the paper, not the paper. Extraction error is field-dependent, so accuracy has to be estimated per column rather than per tool.

The productive way to think about a column of extracted values is as a surrogate for the true value in the paper. That is a measurement-error problem with a developed theory Carroll et al., 2006: carried into a downstream regression or meta-analysis, non-differential error attenuates the association and differential error can push it either way. The remedy is the standard one, a validation subsample. Extract $m$ papers automatically, hand-check $k$ of them, and report per-field accuracy alongside the synthesis. Section that section works the extraction case out in detail.

1.2Consensus¶

• [Consensus] Claim-level search engine. Answers an empirical yes/no question by retrieving papers and summarising each one’s stance on the claim. Fits: rapid orientation on a question outside your own field. Watch: the aggregate stance display is a vote count over a non-random sample of the literature -- no inverse-variance weighting, no risk-of-bias assessment, no publication-bias correction.

This is worth stating bluntly to a statistical audience, because the display invites the wrong reading. A bar showing that most retrieved papers support a claim is an unweighted tally over whatever an index surfaced. A meta-analysis weights by precision, models heterogeneity and interrogates small-study effects. The tally is a triage signal; it is not an effect estimate.

1.3Scite¶

• [Scite] Smart citation index. Extracts the sentence in which each citation occurs and classifies its intent as supporting, mentioning or contrasting Nicholson et al., 2021. Fits: checking whether a result you intend to build on has been contradicted, and reading the sentences that cite it. Watch: the labels come from a classifier. Contrasting citations are rare, so the class you most want to trust has the least training signal and the worst precision.

The citation-context view is the durable contribution: being handed the sentences that cite a paper is valuable even if you ignore every label. When the labels do matter, treat them as you would any automated coding scheme and validate on a subsample.

# Do a tool's citation-intent labels agree with hand labels?
# 60 citations were drawn and labelled by hand as a validation sample.
labs = ["supporting", "mentioning", "contrasting"]
n_tab = {  # (tool label, human label): count
    ("supporting",  "supporting"): 18,
    ("supporting",  "mentioning"):  7,
    ("supporting",  "contrasting"): 1,
    ("mentioning",  "supporting"):  4,
    ("mentioning",  "mentioning"): 25,
    ("mentioning",  "contrasting"): 2,
    ("contrasting", "supporting"):  0,
    ("contrasting", "mentioning"):  2,
    ("contrasting", "contrasting"): 1,
}
n = sum(n_tab.values())
po = sum(v for (a, b), v in n_tab.items() if a == b) / n
row = {l: sum(n_tab[(l, h)] for h in labs) for l in labs}
col = {h: sum(n_tab[(t, h)] for t in labs) for h in labs}
pe = sum(row[l] * col[l] for l in labs) / n**2
kappa = (po - pe) / (1 - pe)
print(f"n = {n}  agreement {po:.3f}  chance {pe:.3f}  kappa {kappa:.3f}")

for l in labs:
    tp = n_tab[(l, l)]
    print(f"{l:>12}: tool {row[l]:>2}, human {col[l]:>2}, "
          f"precision {tp/row[l]:.2f}, recall {tp/col[l]:.2f}")

n = 60  agreement 0.733  chance 0.455  kappa 0.511
  supporting: tool 26, human 22, precision 0.69, recall 0.82
  mentioning: tool 31, human 34, precision 0.81, recall 0.74
 contrasting: tool  3, human  4, precision 0.33, recall 0.25

Raw agreement of 0.733 looks respectable until chance agreement is removed; $\kappa = 0.511$ is moderate on the conventional descriptive scale Cohen, 1960Landis & Koch, 1977. Look at the per-class breakdown: the contrasting class, the reason a researcher opens the tool at all, is recovered in one of the four cases where the human said so. An aggregate accuracy figure hides that completely.

2Citation-graph exploration¶

2.1ResearchRabbit and Litmaps¶

• [ResearchRabbit] Citation-graph explorer. Grows a visual neighbourhood of papers outward from a seed set along citation and co-authorship edges. Fits: expanding a small set of known-relevant papers into a candidate pool before formal screening. Watch: it explores one connected component. A parallel literature with different vocabulary and different ancestors is never reached.

• [Litmaps] Citation-graph explorer with monitoring. The same graph substrate, organised around a persistent map of a topic that can alert you when new work attaches to it. Fits: maintaining coverage over the life of a project, which matters for a review that takes a year to publish. Watch: an alert stream is a running search, so its inclusion criteria must be recorded as carefully as the original search if the review is to be reproducible.

What distinguishes both from embedding search is that they traverse edges rather than compare text, so their recall is a property of the citation graph’s coverage, not of the phrasing of your query. Snowballing is chain-referral sampling and behaves like it: yield per generation rises, then falls as the neighbourhood saturates.

# Snowballing on the citation graph: yield per generation
nbr = {  # paper -> citation neighbourhood
    "S1": ["A1","A2","A3"], "S2": ["A2","A4","B1"],
    "A1": ["B1","B2"],      "A2": ["B2","B3","C1"],
    "A3": ["B3","C2"],      "A4": ["B4","C1"],
    "B1": ["C1","C3"],      "B2": ["C2","C4"],
    "B3": ["C3"],           "B4": ["C4","C5"],
    "C1": [], "C2": [], "C3": [], "C4": [], "C5": [],
}
rel = {"S1","S2","A1","A2","A4","B1","B2","B4","C1","C4"}

front, seen = ["S1","S2"], {"S1","S2"}
found = seen & rel
print(f"{'gen':>3} {'screened':>9} {'new rel':>8} {'recall':>7}")
print(f"{0:>3} {len(seen):>9} {len(found):>8} {len(found)/len(rel):>7.2f}")
for gen in (1, 2, 3):
    nxt = sorted({p for f in front for p in nbr[f] if p not in seen})
    seen |= set(nxt)
    found |= set(nxt) & rel
    print(f"{gen:>3} {len(nxt):>9} {len(set(nxt) & rel):>8} "
          f"{len(found)/len(rel):>7.2f}")
    front = nxt
print(f"screened {len(seen)} in all; crawl precision "
      f"{len(found)/len(seen):.2f}")

gen  screened  new rel  recall
  0         2        2    0.20
  1         5        4    0.60
  2         6        3    0.90
  3         2        1    1.00
screened 15 in all; crawl precision 0.67

Two generations recover ninety percent of the relevant set at a precision of about two thirds; the third generation buys the last paper at the cost of screening more records. That shape -- steep early yield, long tail -- is why a protocol should fix a snowballing stopping rule in advance rather than stopping when the researcher gets tired.

3Grounded synthesis over a corpus you supply¶

3.1NotebookLM¶

• [NotebookLM] Closed-corpus grounded assistant. Answers questions using only documents you upload, citing spans in those documents. Fits: interrogating a fixed reading pile -- the forty papers already judged in scope, a protocol, a set of your own drafts. Watch: a closed corpus removes fabricated references but not misattribution. A citation to a real span is not evidence that the span supports the sentence.

The retrieval-augmented pattern here Lewis et al., 2020 underlies most document assistants: embed the corpus Reimers & Gurevych, 2019, retrieve the passages nearest the query, condition generation on them. Because you control the corpus, the type of error changes rather than disappearing. The check that generalises is a groundedness test: does the quoted span occur in the cited source, and does it say what the summary says it says? Only the first half is mechanical.

# Groundedness: does the quoted span occur in the cited source?
src = {
  "doc1": "Screening was done independently by two reviewers, with "
          "disagreements resolved by a third. Agreement before "
          "adjudication was moderate.",
  "doc2": "We fitted a mixed effects model with a random intercept "
          "for site. Residual diagnostics showed no heteroscedasticity.",
}
claims = [
  ("doc1", "independently by two reviewers"),
  ("doc1", "Agreement before adjudication was substantial"),  # altered
  ("doc2", "a random intercept for site"),
  ("doc2", "we adjusted for baseline severity"),              # invented
]
ok = 0
for doc, span in claims:
    hit = span.lower() in src[doc].lower()
    ok += hit
    print(f"[{'PASS' if hit else 'FAIL'}] {doc}: {span!r}")
print(f"verbatim-grounded: {ok}/{len(claims)}")
print("PASS means the string is present, not that it")
print("supports the sentence built on it")

[PASS] doc1: 'independently by two reviewers'
[FAIL] doc1: 'Agreement before adjudication was substantial'
[PASS] doc2: 'a random intercept for site'
[FAIL] doc2: 'we adjusted for baseline severity'
verbatim-grounded: 2/4
PASS means the string is present, not that it
supports the sentence built on it

The two failures are different animals. The second claim is a near-miss paraphrase that flips a magnitude word; the fourth invents content. A verbatim test catches both. Neither test catches a real, correctly quoted span attached to a conclusion it does not license -- that failure is the subject of Section that section and it requires a human reader and a sampling plan.

4Programmable indexes¶

4.1Semantic Scholar and OpenAlex APIs¶

• [Semantic Scholar API] Programmable scholarly index. Serves records, abstracts, citation edges and paper embeddings over HTTP Kinney et al., 2023. Fits: building a reproducible search of your own, or auditing what a point-and-click tool returned. Watch: coverage and field completeness vary by discipline; missing abstracts and missing DOIs are common enough to change a count.

• [OpenAlex API] Fully open scholarly index. An openly licensed graph of works, authors, venues, institutions and topics, with bulk as well as query access Priem et al., 2022. Fits: anything needing the whole frame rather than a page of results -- coverage audits, field-level denominators, bibliometrics. Watch: openness is not completeness. Author disambiguation and topic assignment are themselves model outputs with error rates.

Both are increasingly reachable by coding agents through Model Context Protocol servers rather than through a browser; that connection pattern, and how it differs from a skill, is developed in Section that section. The statistical reason to prefer an API to a search box is reproducibility: a query string plus a snapshot date is a method, and a screenshot is not. A minimal reproducible query is one line, and belongs in your repository rather than in your browser history.

curl -s 'https://api.openalex.org/works?filter=title.search:systematic%20review,\
from_publication_date:2020-01-01&per-page=200&cursor=*' > works_page1.json

Anyone who queries two indexes discovers quickly that merging them is where the errors live. Identifiers are formatted inconsistently, titles differ by punctuation, and a naive union double-counts.

# Merging two bibliographic feeds: which key catches which duplicate?
import re, unicodedata

feed_a = [   # e.g. an OpenAlex-shaped export
  {"doi": "https://doi.org/10.1038/S41598-023-41032-5",
   "title": "Fabrication and errors in bibliographic citations"},
  {"doi": "10.1186/s13643-016-0384-4",
   "title": "Rayyan\u2014a web and mobile app for systematic reviews"},
  {"doi": None,
   "title": "Toward Systematic Review Automation"},
  {"doi": "10.1136/bmj.n71",
   "title": "The PRISMA 2020 statement"},
]
feed_b = [   # e.g. a Semantic Scholar-shaped export
  {"doi": "10.1038/s41598-023-41032-5",
   "title": "Fabrication and errors in bibliographic citations"},
  {"doi": " 10.1186/S13643-016-0384-4 ",
   "title": "Rayyan - a web and mobile app for systematic reviews"},
  {"doi": "10.1186/s13643-019-1074-9",
   "title": "Toward systematic review automation"},
  {"doi": "10.1162/qss_a_00146",
   "title": "scite: a smart citation index"},
]

def norm_doi(d):
    if not d:
        return None
    return re.sub(r"^https?://(dx\.)?doi\.org/", "", d.strip().lower())

def norm_title(t):
    t = unicodedata.normalize("NFKD", t).lower()
    return re.sub(r"[^a-z0-9]+", " ", t).strip()

for name, fn, fld in (("doi", norm_doi, "doi"),
                      ("title", norm_title, "title")):
    ka = {fn(r[fld]) for r in feed_a if fn(r[fld])}
    kb = {fn(r[fld]) for r in feed_b if fn(r[fld])}
    print(f"{name:>5} key: A={len(ka)} B={len(kb)} matched={len(ka & kb)}")

merged = {}
for r in feed_a + feed_b:
    merged.setdefault(norm_doi(r["doi"]) or norm_title(r["title"]), r)
print("records in, records out:", len(feed_a) + len(feed_b),
      "->", len(merged))
print("pair the DOI key alone would miss:",
      norm_title(feed_a[2]["title"]) == norm_title(feed_b[2]["title"]))

  doi key: A=3 B=4 matched=2
title key: A=4 B=4 matched=3
records in, records out: 8 -> 6
pair the DOI key alone would miss: True

Normalising the DOI recovers two of the shared records; falling back to a normalised title recovers the third, which had no DOI in one feed. This is record linkage, and the usual cautions apply: a permissive key merges distinct papers, a strict key inflates the count, and you should report which key you used and how many records matched under each.

5Measuring the stack rather than trusting it¶

A discovery pipeline is worth evaluating whole. Two mechanisms with different failure modes -- lexical matching Robertson & Zaragoza, 2009 and approximate nearest-neighbour search over embeddings Malkov & Yashunin, 2020 -- retrieve overlapping but non-identical sets, and the union usually beats either.

# recall@k for two retrieval mechanisms, and for their union
gold = {"W%d" % i for i in range(1, 11)}     # 10 known-relevant papers

semantic = ["W3","X1","W7","W1","X2","W9","X3","W4","X4","X5",
            "W2","X6","X7","W10","X8","X9","W5","X10","X11","X12"]
keyword  = ["W1","W2","X13","W5","X14","W4","X15","X16","X24","X17",
            "W8","X18","W3","X19","X20","W7","X21","X22","X23","W9"]

def recall(ranked, k):
    return len(set(ranked[:k]) & gold) / len(gold)

def union_recall(a, b, k):
    return len((set(a[:k]) | set(b[:k])) & gold) / len(gold)

print(f"{'k':>3} {'semantic':>9} {'keyword':>8} {'union':>6} {'gain':>6}")
for k in (5, 10, 20):
    s, w = recall(semantic, k), recall(keyword, k)
    u = union_recall(semantic, keyword, k)
    print(f"{k:>3} {s:>9.2f} {w:>8.2f} {u:>6.2f} {u - max(s, w):>6.2f}")

missed = gold - (set(semantic) | set(keyword))
print("relevant, found by neither at k=20:", sorted(missed))

  k  semantic  keyword  union   gain
  5      0.30     0.30   0.50   0.20
 10      0.50     0.40   0.70   0.20
 20      0.80     0.80   0.90   0.10
relevant, found by neither at k=20: ['W6']

The marginal column is the argument for running two mechanisms: at $k=10$ the union recovers seventy percent against fifty for the better of the two. It is also the argument for not stopping there, since one relevant record is reached by neither at $k=20$. Sensitivity is what a review is judged on, sensitivity is bought with reading time, and the exchange rate is the number needed to read.

# Number needed to read for four search strategies
n_gold = 20                       # papers known to be relevant
strategies = {
    "broad boolean":    (900, 19),   # (titles returned, relevant found)
    "narrow boolean":   (120, 13),
    "semantic search":  (200, 17),
    "narrow+snowball":  (260, 19),
}
print(f"{'strategy':>17} {'hits':>5} {'sens':>6} {'prec':>7} {'NNR':>6}")
for name, (hits, found) in strategies.items():
    sens, prec = found / n_gold, found / hits
    print(f"{name:>17} {hits:>5} {sens:>6.2f} {prec:>7.4f} {1/prec:>6.1f}")
print("NNR = titles read per relevant paper found")

         strategy  hits   sens    prec    NNR
    broad boolean   900   0.95  0.0211   47.4
   narrow boolean   120   0.65  0.1083    9.2
  semantic search   200   0.85  0.0850   11.8
  narrow+snowball   260   0.95  0.0731   13.7
NNR = titles read per relevant paper found

The broad Boolean strategy is the most sensitive and costs forty-seven titles per relevant paper; the narrow strategy is cheap and misses a third of the literature. A narrow query plus snowballing reaches the same sensitivity as the broad query for less than a third of the reading. Reading time is one budget. If you are pushing abstracts through a model, tokens are another, and it is worth costing before you start.

# Token budget for an abstract-synthesis pass. The two rates are
# placeholders you overwrite from your own provider's invoice;
# nothing here is a claim about what anyone charges.
n_abstracts     = 1200
tokens_per_abs  = 320      # measure this on your own corpus
prompt_overhead = 900      # instructions resent with every batch
batch_size      = 25
out_per_batch   = 500
r_in  = 1.0 / 1_000_000    # <-- your input rate per token
r_out = 5.0 / 1_000_000    # <-- your output rate per token

batches = -(-n_abstracts // batch_size)          # ceiling division
tok_in  = batches * (prompt_overhead + batch_size * tokens_per_abs)
tok_out = batches * out_per_batch
cost = tok_in * r_in + tok_out * r_out
print(f"batches: {batches}")
print(f"input tokens {tok_in:,}   output tokens {tok_out:,}")
print(f"overhead share of input: "
      f"{batches * prompt_overhead / tok_in:.1%}")
print(f"one pass, in your currency units: {cost:.3f}")
for reruns in (1, 3, 8):
    print(f"  after {reruns} full re-run(s): {cost * reruns:.3f}")

batches: 48
input tokens 427,200   output tokens 24,000
overhead share of input: 10.1%
one pass, in your currency units: 0.547
  after 1 full re-run(s): 0.547
  after 3 full re-run(s): 1.642
  after 8 full re-run(s): 4.378

Two things in that snippet are deliberate. The rates are variables you overwrite from your own invoice, because any price printed in a textbook about this market is wrong within a year. And the overhead line matters: at this batch size a tenth of the input is instructions resent with every batch, which is the first thing to optimise and the reason long single-document prompts and small batches behave so differently. Note too that stuffing more abstracts into one call is not free of statistical consequence -- retrieval quality degrades for material buried in the middle of a long context Liu et al., 2023, so batch size trades cost against attention.

6What to record in a paper¶

For the search to be reproducible, a methods section must state: the tool or API used, with its version or the date of access; the exact query string and filter set, and for an API the endpoint and any snapshot identifier; the seed set for any citation-graph expansion and the stopping rule that ended it; the count of records at each stage, so a reader can reconstruct the funnel; the model and prompt used for any extraction or summarisation step, quoted verbatim in an appendix or deposited with the code; and the size and result of every hand-validation sample, reported as a proportion with an interval rather than as “spot-checked”. If an agent ran the search, say which tools it was able to call. A search you cannot describe at this level of detail is not a method; it is an anecdote about a productive afternoon.

7Exercises¶

1. Modify the recall example so the two retrieval mechanisms are strongly correlated: have the keyword list return the semantic ranking with small perturbations. Recompute the marginal gain from the union at $k=5,10,20$ and explain, in terms of the correlation between two screening instruments, why the marginal column collapses.

2. Using the citation-intent example, treat the tool label as a diagnostic test for “contrasting”. Compute sensitivity, specificity and positive predictive value for that class alone, then recompute the predictive value under a base rate of 0.02 rather than the $4/60$ in the table. Write one sentence for a collaborator explaining why the tool’s headline accuracy is uninformative here.

3. Build a gold set of ten papers you know well in your own area. Run one Boolean query and one semantic query for the same question, record the top twenty of each, and compute recall@$k$, precision and the number needed to read for each and for their union. Report which relevant papers neither strategy found, and diagnose why.

4. Extend the record-linkage example with a third feed in which ten percent of titles carry an OCR error (swap two characters at random). Compare exact normalised-title matching against a similarity threshold on character trigrams. Report false-match and missed-match counts at three thresholds, and state which error you would rather make in a systematic review, and why.

5. Re-run the token-budget example with tokens_per_abs measured on a corpus you actually have, then add a second cost column for a two-pass design in which every abstract is processed by two independent prompts and disagreements are escalated to a human. At what disagreement rate does the two-pass design cost more in human time than it saves in errors? State the assumptions you had to add to answer that.