Deep Research Agents

A deep research tool takes a question, runs a multi-step loop of searching, reading and note-taking, and returns a long prose report with inline citations. It is the single AI capability most likely to be misused in scientific work, for one reason: the output has the shape of a literature review. It has sections, hedged claims, and a bibliography. A reader’s usual cues for “someone did the reading” all fire, and none of them are load-bearing any more.

This section is not a buyer’s guide. Specific capabilities, interfaces and prices move faster than any textbook can track, and you should check current documentation before committing to a tool. What does not move is the shape of the problem: a generated report is a measurement instrument with an uncharacterised error rate, and characterising error rates is the thing statisticians are for. By the end of the section you should be able to state what a deep research run is a sample of, separate the errors a script can find from the errors only a person can find, and report the second kind as a proportion with an interval rather than as a shrug.

1The loop, and where it can go wrong¶

Under the hood these systems are all variations on the same pattern: an underlying language model is given search as a tool, plans a set of queries, issues them, reads the returned passages, decides whether to search again, and finally writes a report conditioned on the passages it kept. The retrieve-then- generate skeleton is the retrieval-augmented generation architecture Lewis et al., 2020, and the plan--act--observe control loop is the reason-and-act pattern Yao et al., 2022, with search as one of the available tools Schick et al., 2023.

Three consequences follow for a researcher, and none of them are vendor-specific.

First, the retrieval index is a sampling frame. If a body of work is not in the index -- because it is paywalled, because it is a thesis, because it is not in English -- it cannot be cited, and nothing in the report will tell you it is missing. Non-coverage in a survey is invisible from inside the sample; the same is true here.

Second, position matters. Models attend unevenly to long contexts, degrading on material buried in the middle of a long retrieved passage set Liu et al., 2023, so a source can be retrieved and still be effectively unread.

Third, the citation is generated, not looked up. In most architectures the sentence and its attribution are produced by the same decoding pass. That is what makes the two failure modes below different failures rather than two symptoms of one.

The loop itself -- how the harness budgets context, when it spawns fresh-context children to read in parallel, how a scholarly index is exposed to it as a connected tool -- is a design object with its own vocabulary and its own literature. That material belongs to Section that section and is not repeated here; the distinction to carry into this section is only that a deep research product is a pre-packaged harness whose internals you do not set, which is exactly why its output must be audited from the outside.

2The tools¶

Each callout below names the job, the workflow step, and the failure mode to watch. The durable distinctions between these systems are: what corpus they search (open web versus a curated scholarly index), whether attribution is per-sentence or per-paragraph, whether the retrieved sources are exposed to you or hidden, and whether the pipeline is inspectable or a black box. Ask those four questions of any new entrant.

2.1Hosted general-web agents¶

• [OpenAI Deep Research] Hosted general-web research agent. Runs an extended browse-and-synthesise loop from a single prompt and returns a cited report. Fits: the orientation step, before you know the vocabulary of a literature well enough to write your own queries. Watch: the search trace is summarised rather than exported, so the set of queries actually issued is not recoverable for a methods section.

• [Gemini Deep Research] Hosted general-web research agent. Plans a browsing itinerary, executes it, and returns a report; the plan is shown to the user before execution in the current interface. Fits: the same orientation step; the shown-plan-then-execute mode is a consumer instance of the spec-first pattern of Section that section, and the plan is the closest thing to a search strategy these tools emit, so copy it into your notes. Watch: an approved plan feels like a pre-registration and is not one; the executed searches may diverge from it.

• [Perplexity Deep Research] Search-first answer engine with a research mode. Optimised for fast, source-linked answers over web content. Fits: fact-finding and terminology checks, where you want the source link more than the prose. Watch: general-web ranking mixes preprints, press releases and secondary summaries; a claim can be “well sourced” entirely to reporting about a paper rather than the paper.

2.2Scholarly-corpus systems¶

• [Ai2 ScholarQA / Asta] Open scholarly question answering over a curated paper corpus. Answers are grounded in indexed full texts rather than the open web; the underlying OpenScholar work is published and the pipeline is open Asai et al., 2026. Fits: the systematic step, where the frame must be a literature index rather than the web; scholarly indexes are increasingly exposed to general agents as connected tools over the Model Context Protocol (Section that section), so the same frame can be searched from your own harness. Watch: the frame is the corpus. Coverage gaps are now knowable, which is an improvement -- so go and look at what the corpus contains before you trust a null result.

• [STORM] Open-source outline-first report generator. Builds a topic outline by simulating multi-perspective question-asking, then writes a grounded article section by section Shao et al., 2024. Fits: drafting the structure of a background section when you already know the field and want coverage of angles you did not think of. Watch: it optimises breadth of perspective, not depth of evidence; treat the outline as the deliverable and the prose as disposable.

• [Elicit] Structured extraction over a scholarly index; the research-report mode returns prose. Screens papers against inclusion criteria and extracts named fields (population, design, outcome, effect) into a table. Fits: the extraction step of a screening pipeline, where the output should be a data frame you can audit row by row, not prose. Watch: extracted fields are model outputs carrying an error rate; validate against a hand-coded subsample before computing from a column.

The last of these points at the general principle. A tool that returns a table lets you validate a column against a gold standard, which is a statistical problem you know how to solve. A tool that returns prose makes you validate sentence by sentence, which is far more expensive. Where you have the choice, prefer the tool whose output has rows.

3Two hallucinations, not one¶

Citation hallucination: the cited reference does not exist. The identifier resolves to nothing, or it resolves to a different paper than the one named. This has been measured directly: a study of chatbot-generated bibliographies found substantial rates of fabricated and erroneous citations Walters & Wilder, 2023. It is detectable by machine, and therefore there is no excuse for shipping one.

Statement hallucination: the reference exists, is real, may even be the right paper, but does not support the sentence attached to it. This is what the generative-search-engine literature measures as citation precision and recall Liu et al., 2023Gao et al., 2023, and what report-level benchmarks try to automate Li et al., 2025. No identifier check touches it. Someone has to open the paper.

Published rates for both vary enormously across fields, systems and years. Do not carry a headline number around. The useful move -- the statistician’s move -- is to measure the rate in your own output, on your own topic, with the tool version you actually used.

3.1Auditing the bibliography: a census¶

The first pass is a census, because it is cheap. Resolve every identifier and compare the returned title with the cited title. Real code calls Crossref for DOIs and the arXiv API for preprint identifiers:

curl -s https://api.crossref.org/works/10.1038/s41598-023-41032-5 \
  | python -c 'import json,sys; print(json.load(sys.stdin)["message"]["title"][0])'
curl -s "http://export.arxiv.org/api/query?id_list=2402.14207" | grep -o "<title>.*"

The example below runs offline by substituting a dictionary for the network call, so the mechanism is visible without a live connection. One entry has a fabricated DOI and one cites a real DOI under the wrong title -- the two things the check exists to catch.

# One row per cited claim in the generated report: (claim id, identifier, title as cited).
cited = [
    ("C1", "10.1038/s41598-023-41032-5",
     "Fabrication and errors in the bibliographic citations generated by ChatGPT"),
    ("C2", "arXiv:2402.14207",
     "Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models"),
    ("C3", "10.1214/aos/1176344991",
     "Sequential model selection for high-dimensional regression"),
    ("C4", "10.1214/aos/1176344136", "Estimating the dimension of a model"),
    ("C5", "10.18653/v1/2023.emnlp-main.398",
     "Evaluating Verifiability in Generative Search Engines"),
]

# Stand-in for the network resolver: identifier -> title as the registry holds it.
# The real version calls Crossref or the arXiv API; see the shell block in the text.
REGISTRY = {
    "10.1038/s41598-023-41032-5":
        "Fabrication and errors in the bibliographic citations generated by ChatGPT",
    "arXiv:2402.14207":
        "Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models",
    "10.1214/aos/1176344136": "Estimating the Dimension of a Model",
    "10.18653/v1/2023.emnlp-main.398":
        "Enabling Large Language Models to Generate Text with Citations",
}

def normalise(s):
    return "".join(ch.lower() for ch in s if ch.isalnum())

def verify(identifier, claimed_title):
    found = REGISTRY.get(identifier)
    if found is None:
        return "UNRESOLVED", ""
    a, b = normalise(claimed_title), normalise(found)
    if a == b or a in b or b in a:
        return "OK", found
    return "MISMATCH", found

counts = {"OK": 0, "UNRESOLVED": 0, "MISMATCH": 0}
for cid, ident, title in cited:
    status, found = verify(ident, title)
    counts[status] += 1
    print(f"{cid}  {status:<10} {ident}")
    if status == "MISMATCH":
        print(f"      cited as: {title}")
        print(f"      actually: {found}")

m = len(cited)
print(f"\ncited references m = {m}")
for k in ("OK", "UNRESOLVED", "MISMATCH"):
    print(f"  {k:<10} {counts[k]}  ({counts[k]/m:.0%})")

C1  OK         10.1038/s41598-023-41032-5
C2  OK         arXiv:2402.14207
C3  UNRESOLVED 10.1214/aos/1176344991
C4  OK         10.1214/aos/1176344136
C5  MISMATCH   10.18653/v1/2023.emnlp-main.398
      cited as: Evaluating Verifiability in Generative Search Engines
      actually: Enabling Large Language Models to Generate Text with Citations

cited references m = 5
  OK         3  (60%)
  UNRESOLVED 1  (20%)
  MISMATCH   1  (20%)

Note what the mismatch row shows. The DOI is real, it resolves, and a checker that only asked “does this identifier exist?” would pass it. Comparing titles is what catches a reference that was attached to the wrong sentence.

3.2Auditing the claims: a sample¶

Statement hallucination cannot be enumerated, so it must be sampled. Draw $k$ of the $m$ cited claims at random -- not the $k$ that look suspicious, which destroys the estimator -- read the sources, and record whether each supports its sentence. Two readers are better than one, because the judgement is itself a measurement:

# Two readers independently judge whether the cited source supports the sentence.
# 1 = supports, 0 = does not support. These are statement-level judgements: every
# identifier here already resolved, so the mechanical check found nothing wrong.
A = [1,1,1,0,1,1,1,1,0,1,1,0,1,1,1,1,1,0,1,1]
B = [1,1,0,0,1,1,1,1,0,1,1,1,1,0,1,1,1,0,1,1]

n = len(A)
n11 = sum(a == 1 and b == 1 for a, b in zip(A, B))
n00 = sum(a == 0 and b == 0 for a, b in zip(A, B))
n10 = sum(a == 1 and b == 0 for a, b in zip(A, B))
n01 = sum(a == 0 and b == 1 for a, b in zip(A, B))

po = (n11 + n00) / n
pa1, pb1 = sum(A) / n, sum(B) / n
pe = pa1 * pb1 + (1 - pa1) * (1 - pb1)
kappa = (po - pe) / (1 - pe)

print(f"n pairs judged        {n}")
print(f"both support          {n11}")
print(f"both not supported    {n00}")
print(f"reader A only         {n10}")
print(f"reader B only         {n01}")
print(f"observed agreement    {po:.3f}")
print(f"chance agreement      {pe:.3f}")
print(f"Cohen's kappa         {kappa:.3f}")

# Unsupported rate, counting a pair as unsupported if EITHER reader flags it
# (the conservative adjudication rule: flag now, resolve by discussion later).
flagged = sum(a == 0 or b == 0 for a, b in zip(A, B))
print(f"\nflagged by either     {flagged}/{n} = {flagged/n:.2f}")
print(f"agreed unsupported    {n00}/{n} = {n00/n:.2f}")

n pairs judged        20
both support          14
both not supported    3
reader A only         2
reader B only         1
observed agreement    0.850
chance agreement      0.650
Cohen's kappa         0.571

flagged by either     6/20 = 0.30
agreed unsupported    3/20 = 0.15

A $\kappa$ in this range says the two readers disagree about “supports” often enough that a single-reader audit would have produced a noticeably different number. Report the adjudication rule you used along with the rate.

Now the interval. With $x$ flagged out of $k$ checked, use a Wilson score interval Wilson, 1927 rather than the Wald interval, which behaves badly at the small $k$ and small $p$ that this problem always has:

from math import sqrt

def wilson(x, n, z=1.959964):
    """Wilson score interval for a binomial proportion (Wilson, 1927)."""
    if n == 0:
        return (0.0, 1.0)
    p = x / n
    d = 1 + z * z / n
    c = (p + z * z / (2 * n)) / d
    h = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return (max(0.0, c - h), min(1.0, c + h))

print(f"{'checked k':>9} {'flagged x':>9} {'p-hat':>7}  {'95% Wilson interval':>22} {'half-width':>11}")
for x, n in [(0, 5), (1, 5), (0, 10), (1, 10), (2, 40), (4, 40), (10, 200)]:
    lo, hi = wilson(x, n)
    print(f"{n:>9} {x:>9} {x/n:>7.3f}  [{lo:>8.3f}, {hi:>8.3f}] {(hi-lo)/2:>11.3f}")

checked k flagged x   p-hat     95% Wilson interval  half-width
        5         0   0.000  [   0.000,    0.434]       0.217
        5         1   0.200  [   0.036,    0.624]       0.294
       10         0   0.000  [   0.000,    0.278]       0.139
       10         1   0.100  [   0.018,    0.404]       0.193
       40         2   0.050  [   0.014,    0.165]       0.076
       40         4   0.100  [   0.040,    0.231]       0.095
      200        10   0.050  [   0.027,    0.090]       0.031

Read the first row carefully, because it is the one that matters in practice. Checking five claims and finding nothing wrong is consistent with an unsupported-claim rate anywhere up to roughly 0.43. That is the arithmetic behind the claim that spot-checking is not a method.

3.3How many to check¶

Turn the interval around and solve for $k$. Because the cited claims are a finite population of size $m$, a finite-population correction applies and the required $k$ is a substantial fraction of $m$ once you want real precision:

from math import sqrt, ceil

def wilson(x, n, z=1.959964):
    p = x / n
    d = 1 + z * z / n
    c = (p + z * z / (2 * n)) / d
    h = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return (max(0.0, c - h), min(1.0, c + h))

def half_width(p_assumed, n, m=None):
    """Expected Wilson half-width at rate p_assumed with n checked out of m cited.
    m is the finite population: sampling without replacement shrinks the interval."""
    x = p_assumed * n
    lo, hi = wilson(x, n)
    h = (hi - lo) / 2
    if m is not None and n <= m:
        h *= sqrt((m - n) / (m - 1)) if m > 1 else 0.0
    return h

def n_needed(p_assumed, target, m=None, cap=100000):
    for n in range(2, cap):
        if m is not None and n >= m:
            return m
        if half_width(p_assumed, n, m) <= target:
            return n
    return None

print("cited claims in report: m = 120")
print(f"{'assumed rate':>12} {'target +/-':>11} {'k to check':>11} {'fraction of m':>14}")
for p in (0.05, 0.10, 0.20):
    for t in (0.20, 0.10, 0.05):
        k = n_needed(p, t, m=120)
        print(f"{p:>12.2f} {t:>11.2f} {k:>11} {k/120:>14.0%}")

cited claims in report: m = 120
assumed rate  target +/-  k to check  fraction of m
        0.05        0.20           8             7%
        0.05        0.10          22            18%
        0.05        0.05          52            43%
        0.10        0.20           9             8%
        0.10        0.10          29            24%
        0.10        0.05          66            55%
        0.20        0.20          13            11%
        0.20        0.10          40            33%
        0.20        0.05          81            68%

The lesson is uncomfortable and worth stating plainly: to bound the rate to within a few percentage points you end up checking most of the report. If you are unwilling to do that, do not cite the report’s claims as though you had.

When the audit turns up zero problems, quote a one-sided upper bound rather than “no errors found”. The rule of three, $3/k$, is the familiar approximation; the exact Clopper--Pearson bound is a one-liner and slightly tighter:

from math import comb

def upper_bound_zero_events(n, alpha=0.05):
    """Exact one-sided Clopper-Pearson upper bound when x = 0 of n:
    largest p with P(X = 0 | n, p) >= alpha, i.e. p = 1 - alpha**(1/n)."""
    return 1 - alpha ** (1 / n)

def exact_upper(x, n, alpha=0.05, tol=1e-10):
    """Exact one-sided upper bound for any x: largest p with P(X <= x) >= alpha."""
    def tail(p):
        return sum(comb(n, i) * p**i * (1 - p)**(n - i) for i in range(x + 1))
    lo, hi = 0.0, 1.0
    while hi - lo > tol:
        mid = (lo + hi) / 2
        if tail(mid) >= alpha:
            lo = mid
        else:
            hi = mid
    return lo

print(f"{'k checked':>9} {'rule of 3/k':>12} {'exact 95% upper':>16} {'if x=1':>9}")
for n in (5, 10, 20, 40, 100):
    print(f"{n:>9} {3/n:>12.3f} {upper_bound_zero_events(n):>16.3f} "
          f"{exact_upper(1, n):>9.3f}")

k checked  rule of 3/k  exact 95% upper    if x=1
        5        0.600            0.451     0.657
       10        0.300            0.259     0.394
       20        0.150            0.139     0.216
       40        0.075            0.072     0.113
      100        0.030            0.030     0.047

3.4Recall of the retrieval stack¶

The complement of fabrication is omission, and it is measured differently. Build a seed set of papers a domain expert says the review must contain, then score the agent’s bibliography against it. Precision at $k$ is nearly useless here -- a report can pad indefinitely -- but recall against a fixed seed set is informative and cheap:

# A hand-built seed set of papers a domain expert says the review MUST contain,
# versus what the agent's bibliography actually contained, in the order it cited them.
seed = {"P01", "P02", "P03", "P04", "P05", "P06", "P07", "P08", "P09", "P10",
        "P11", "P12"}
retrieved = ["P03", "X41", "P07", "P01", "X19", "P11", "X08", "P02", "P05",
             "X33", "P09", "X72", "P04", "X15", "P12"]

print(f"{'k':>3} {'hits':>5} {'recall@k':>9} {'precision@k':>12}")
for k in (5, 10, 15):
    hits = sum(1 for r in retrieved[:k] if r in seed)
    print(f"{k:>3} {hits:>5} {hits/len(seed):>9.2f} {hits/k:>12.2f}")

missed = sorted(seed - set(retrieved))
print(f"\nseed papers never cited: {missed}")
print(f"final recall           : {1 - len(missed)/len(seed):.2f}")
print(f"non-seed citations     : {sum(1 for r in retrieved if r not in seed)}"
      f" of {len(retrieved)}")

  k  hits  recall@k  precision@k
  5     3      0.25         0.60
 10     6      0.50         0.60
 15     9      0.75         0.60

seed papers never cited: ['P06', 'P08', 'P10']
final recall           : 0.75
non-seed citations     : 6 of 15

Three seed papers were never cited at all. Whether that is a frame problem (not in the index) or a ranking problem (in the index, not surfaced) is diagnosable by searching the tool directly for each missed paper, and the answer changes what you do next.

3.5The cost of doing this properly¶

Verification is the dominant cost, not generation, and by a wide margin. The snippet below takes every rate as a variable so you can substitute your own invoice and your own reading speed:

# Every quantity here is a variable you set from your own project, not a claim
# about any vendor. Change the four inputs and re-run.
m_claims            = 140     # cited claims in the report
minutes_per_check   = 6.0     # median time to open a paper and judge support
sample_fraction     = 0.30    # fraction you will hand-check
hourly_rate         = 45.0    # analyst cost per hour, your currency

k = round(m_claims * sample_fraction)
hours = k * minutes_per_check / 60
print(f"claims cited          {m_claims}")
print(f"sampled for checking  {k}")
print(f"analyst hours         {hours:.1f}")
print(f"verification cost     {hours * hourly_rate:,.0f}")
print(f"cost per cited claim  {hours * hourly_rate / m_claims:,.2f}")

# Token-budget arithmetic for the generation side. The RATE is your input:
# read it off your own invoice or the current documentation, do not memorise it.
rate_per_million_in  = 3.00
rate_per_million_out = 15.00
searches, tokens_in_per_search, tokens_out_per_search = 40, 12_000, 900
report_out_tokens = 9_000

tin  = searches * tokens_in_per_search
tout = searches * tokens_out_per_search + report_out_tokens
gen_cost = tin / 1e6 * rate_per_million_in + tout / 1e6 * rate_per_million_out
print(f"\ninput tokens          {tin:,}")
print(f"output tokens         {tout:,}")
print(f"generation cost       {gen_cost:,.2f}")
print(f"verification / gen    {hours * hourly_rate / gen_cost:,.1f}x")

claims cited          140
sampled for checking  42
analyst hours         4.2
verification cost     189
cost per cited claim  1.35

input tokens          480,000
output tokens         45,000
generation cost       2.11
verification / gen    89.4x

The ratio in the last line is the number to internalise. Generation is cheap enough to be free at the margin; the human hours needed to make the output citable are not. Budget accordingly, and be suspicious of any workflow that treats the generation step as the expensive one.

4What to record in a paper¶

For a deep research run to be reproducible-in-principle, a methods section must state: the tool and interface used, with the vendor’s model or product identifier as displayed at run time; the date of the run, because the index changes underneath you; the verbatim prompt, and any configuration such as source restrictions, date filters or an uploaded corpus; the number of cited claims $m$ in the returned report; the number $k$ hand-verified, how those $k$ were drawn (random, stratified by section, or exhaustive), and by how many readers; the identifier-resolution result for all $m$ references, reported as a census; the estimated unsupported-claim rate with an interval and the interval method; and what role the output played -- whether it seeded the search, drafted prose that was then rewritten, or supplied claims that appear in the final text. If the answer to the last item is “supplied claims”, the audit is not optional. Attach the reference list and the audit spreadsheet as supplementary material; they are data.

Writing that paragraph is easier if it already exists. Under the manuscript-driven discipline of Section that section you draft the methods text and the audit table shell before the run, which fixes $k$ and the sampling scheme in advance rather than after you have seen how many problems the first few checks turned up.

5Exercises¶

1. Run a deep research tool on a question in your own field, then take its bibliography and run the census of Example 1 against the live Crossref and arXiv APIs instead of the stand-in dictionary. Report $m$, the number unresolvable, and the number resolving to a title that does not match the one cited. Comment on which of the three categories was hardest to automate.

2. Using the same report, draw $k=20$ cited claims at random and have two readers judge support independently. Compute observed agreement, Cohen’s $\kappa$, and the Wilson interval for the unsupported rate under both a strict rule (flagged if either reader flags) and a consensus rule. Explain which rule you would pre-specify and why the choice must be made before seeing the data.

3. Modify the sample-size code so that the audit is stratified: introduction and discussion sections are sampled at twice the rate of methods sections. Derive the stratified estimator of the overall unsupported rate and its variance, and compare the required total $k$ with the simple random sample of Example 4 at the same target half-width.

4. Take a systematic review published in your field with a documented search strategy. Use its included-studies list as a seed set and compute recall at $k$ for two different tools, as in Example 6. For every seed paper missed, determine by direct search whether it was absent from the index or present but not surfaced, and tabulate the two failure types by tool.

5. A colleague proposes reporting “we verified a sample of citations and found no errors” after checking eight of ninety. Using the one-sided bound of Example 5, write the two sentences you would send back, including the numerical upper bound their check actually supports and the $k$ they would need for an upper bound of 0.05.