Choosing and Evaluating a Research AI Tool

1How to Choose and Evaluate a Research AI Tool¶

This section is about choosing an instrument, not about which product is currently ahead. The tool market re-shuffles every few months: capabilities, interfaces and prices move, vendors ship and withdraw features, and any snapshot printed in a book is stale before the book is bound. Readers should check current documentation for what a tool can do today. What does not move is the evaluation frame: what job you are hiring the tool for, what evidence would show it does that job on your data, and what failure mode you must therefore watch. That frame is a measurement problem, and it is the part of this material where a statistician has an unfair advantage over most of the people currently writing tool reviews.

1.1The Evaluation Frame: Job, Evidence, Failure Mode¶

Three questions, asked in order, dispose of most tool decisions.

What job? Name the workflow step, not the aspiration. “Help with my research” is not a job; “screen 1{,}800 abstracts against inclusion criteria”, “refactor a simulation study so the estimator is swappable”, “draft the first pass of a methods paragraph I will rewrite” are jobs. A job has an input, an output, and a person who currently does it.

What evidence? The vendor’s evidence is a single-arm study on a task distribution you did not choose. Your evidence has to come from a pilot on your own material, with a hand-labelled subsample as the reference standard. This is the same logic as validating an assay: the manufacturer’s sensitivity is a starting prior, not a substitute for local validation. Reporting checklists developed for machine-learning experiments make the same demand of published results Pineau et al., 2021Dodge et al., 2019, and documentation conventions such as model cards and datasheets exist because capability claims without population descriptions are uninterpretable Mitchell et al., 2019Gebru et al., 2021.

What failure mode? Every tool has a characteristic way of being wrong, and the useful question is whether that way is detectable in your pipeline. A tool that fails loudly (a script that raises) is safer than one that fails quietly (a plausible number in a table). Rank candidate tools by how visible their errors are downstream, not by how often they are right.

A fourth question decides whether the tool is worth adopting at all: does it change the total time, including review? The arithmetic is small enough to do in your head, and doing it explicitly is more informative than it looks.

The break-even acceptance rate depends only on the ratio of review time to do-it-yourself time. Below that rate, a suggestion engine costs you time even when it is occasionally right.

t_manual = 12.0      # minutes to write the block yourself
t_review = 4.0       # minutes to read the suggestion and decide
p        = 0.55      # fraction of suggestions you end up keeping

net = p * (t_manual - t_review) - (1.0 - p) * t_review
p_star = t_review / t_manual
print(f"net minutes saved per suggestion: {net:+.2f}")
print(f"break-even acceptance rate: p* = t_review/t_manual = {p_star:.3f}")
for pp in (0.20, 0.35, 0.55, 0.80):
    print(f"  p={pp:.2f}  net={pp*(t_manual-t_review)-(1-pp)*t_review:+6.2f} min")
h = 0.05             # extra probability that a kept suggestion carries a defect
t_debug = 45.0       # minutes to find and fix that defect later
print(f"with a {h:.0%} defect rate costing {t_debug:.0f} min, net becomes "
      f"{net - p*h*t_debug:+.2f} min")

net minutes saved per suggestion: +2.60
break-even acceptance rate: p* = t_review/t_manual = 0.333
  p=0.20  net= -1.60 min
  p=0.35  net= +0.20 min
  p=0.55  net= +2.60 min
  p=0.80  net= +5.60 min
with a 5% defect rate costing 45 min, net becomes +1.36 min

The break-even rate is $p^\star = t_{\mathrm{review}}/t_{\mathrm{manual}}$, which is why suggestion tools pay off on boilerplate (large $t_{\mathrm{manual}}$, small $t_{\mathrm{review}}$) and lose on subtle statistical code, where reviewing a wrong suggestion costs nearly as much as writing the right one. The last line adds the term people leave out: a defect that survives review and is found later. Two careful studies of programmer productivity with AI assistance reach opposite directional conclusions --- a controlled task study reporting faster completion Peng et al., 2023 and a randomised trial with experienced open-source maintainers in which measured completion time and developers’ own impressions diverged Becker et al., 2025. The disagreement is the lesson: the sign of the effect depends on task, codebase familiarity and expertise, so it has to be measured locally rather than assumed.

1.2The Durable Axes¶

Products change; the axes below have been stable since these tools existed, and they predict behaviour better than any feature list. Treat them as the columns of a small design matrix over the tools you are considering.

• [Terminal-native agent] Category: agent harness in a shell. Reads and edits files in a repository and runs commands in your working directory. Fits: refactoring, pipeline surgery, anything where the unit of work is a repository rather than a file. Watch: it acts in your real filesystem --- work on a branch and require approval for irreversible commands.

• [IDE-native assistant] Category: editor integration. Completes and rewrites code in the buffer you are already looking at, with the open project as context. Fits: incremental authoring, test writing, local edits under continuous human gaze. Watch: continuous acceptance fatigues reviewers; acceptance is not verification.

• [Hosted chat or canvas] Category: managed conversational UI. A browser interface over an API, sometimes with file upload and a sandboxed interpreter. Fits: exploration, explanation, one-off transformations. Watch: the session is not a script; nothing is reproducible unless you export the prompt and the code.

• [Deep research agent] Category: search-and-synthesis loop. Runs many searches and returns a long report with inline citations. Fits: scoping an unfamiliar literature before you read it yourself. Watch: citation and statement hallucination, treated as a measurement problem in Section that section.

• [Local-weights runner] Category: on-premises inference. Runs open-weight models on your own hardware. Fits: governed data, reproducible pinning, unmetered batch jobs (Section that section). Watch: you now own the version pinning, the throughput and the evaluation that a vendor was implicitly doing for you.

• [Retrieval stack] Category: index plus generator. Answers from a corpus you supply rather than from parametric memory (Section that section). Fits: question answering over your own documents. Watch: the index is a sampling frame; recall failures are invisible in the output.

The axes that matter for each of these are: terminal versus IDE versus hosted; local weights versus a remote API; whether the tool can execute code; whether it cites at the sentence level or at the paragraph level; whether it is open source or managed; and whether it is scriptable in a batch job. Note what is not on the list: any leaderboard position. Two tools on the same side of all six axes will behave similarly on your work regardless of which is currently ahead on a public task set.

Where a tool implements one of the agentic workflow patterns --- an on-demand skill mechanism, a project context file, an MCP client, a plan-first mode --- that is a genuine axis too, and Section that section treats those patterns in depth. They belong here only as a reminder that the pattern is durable while the product that implements it is not.

1.3Measuring Two Tools Against Each Other¶

If the choice matters, run a pilot. The design is a paired comparison on a random sample of your own items, with a hand-labelled reference standard on a subsample. Two quantities are worth separating: how much the tools agree with each other, and how often they agree with the truth. They are not the same, and the first is routinely mistaken for evidence of the second.

Two abstract-screening passes are simulated with a shared difficulty component, so some items are hard for both. Agreement is summarised by Cohen's $\kappa$ [@cohen1960coefficient], which corrects raw concordance for chance.

import numpy as np
rng = np.random.default_rng(11)
n = 400
truth = rng.random(n) < 0.30                 # 30% of abstracts are truly includable
shared = rng.random(n) < 0.15                # hard items both tools misread the same way
a = np.where(shared, ~truth, rng.random(n) < np.where(truth, 0.85, 0.10))
b = np.where(shared, ~truth, rng.random(n) < np.where(truth, 0.83, 0.12))

both = (a & b).sum(); neither = (~a & ~b).sum()
po = (both + neither) / n
pe = (a.mean()*b.mean()) + ((1-a.mean())*(1-b.mean()))
kappa = (po - pe) / (1 - pe)
print(f"tool A includes {a.sum()}, tool B includes {b.sum()}, truth includes {truth.sum()}")
print(f"raw agreement A-B: {po:.3f}   Cohen's kappa: {kappa:.3f}")
print(f"accuracy vs truth   A: {(a==truth).mean():.3f}   B: {(b==truth).mean():.3f}")
agree = a == b
print(f"among the {agree.sum()} items where they agree, "
      f"{(a[agree]!=truth[agree]).mean():.3f} are jointly wrong")

tool A includes 151, tool B includes 149, truth includes 133
raw agreement A-B: 0.830   Cohen's kappa: 0.637
accuracy vs truth   A: 0.770   B: 0.745
among the 332 items where they agree, 0.190 are jointly wrong

Substantial agreement, and yet nearly a fifth of the items the two tools agree on are jointly wrong. Correlated errors are the reason: the tools share training data, share prompt phrasing and fail on the same hard items, so concordance between them is not independent replication. This is exactly why an inter-rater statistic cannot stand in for a validation subsample, and why “the second model agreed” is a weak defence.

1.4Verification Is a Sample-Size Problem¶

The single most common methodological error in reported AI-assisted analysis is the unspecified spot check. “We manually verified a few outputs” has no operating characteristics. Verification is estimation of a proportion, and it obeys the arithmetic of estimating a proportion.

Wilson intervals [@wilson1927probable] for an assumed 10% error rate, at the sample sizes people actually check.

from math import sqrt
def wilson(k, n, z=1.96):
    ph = k / n; d = 1 + z*z/n
    c  = (ph + z*z/(2*n)) / d
    hw = z*sqrt(ph*(1-ph)/n + z*z/(4*n*n)) / d
    return max(0.0, c - hw), min(1.0, c + hw)

print("checked  found-bad   95% Wilson interval   half-width")
for n in (5, 10, 40, 100, 400):
    k = round(0.10 * n)
    lo, hi = wilson(k, n)
    print(f"{n:>7} {k:>10}   [{lo:.3f}, {hi:.3f}]   {(hi-lo)/2:.3f}")

target = 0.05
n = 5
while (wilson(round(0.10*n), n)[1] - wilson(round(0.10*n), n)[0]) / 2 > target:
    n += 1
print(f"n needed for half-width <= {target:.2f} at an assumed 10% rate: {n}")
print(f"a clean spot check of 5 still admits rates up to {wilson(0, 5)[1]:.3f}")

checked  found-bad   95% Wilson interval   half-width
      5          0   [0.000, 0.434]   0.217
     10          1   [0.018, 0.404]   0.193
     40          4   [0.040, 0.231]   0.095
    100         10   [0.055, 0.174]   0.060
    400         40   [0.074, 0.133]   0.030
n needed for half-width <= 0.05 at an assumed 10% rate: 141
a clean spot check of 5 still admits rates up to 0.434

A clean check of five items is consistent with an error rate above forty percent. Bounding the rate to within five percentage points needs on the order of a hundred and forty checked items --- a number worth budgeting for before the project starts, not discovering afterwards. If a hundred and forty is unaffordable, the honest response is to report the interval you can afford, not to report the point estimate alone.

The same arithmetic settles the tool-comparison question. Paired designs help, because the same items are given to both tools and only discordant pairs carry information McNemar, 1947Dietterich, 1998.

from math import sqrt
def paired_ci(b, c, n, z=1.96):
    d = (b - c) / n
    se = sqrt((b + c - (b - c)**2 / n) / n**2)
    return d, d - z*se, d + z*se

n, b, c = 30, 7, 4          # b: A succeeds where B fails; c: the reverse
d, lo, hi = paired_ci(b, c, n)
print(f"n={n}: A-B difference {d:+.3f}, 95% CI [{lo:+.3f}, {hi:+.3f}]")
print(f"interval covers zero: {lo < 0 < hi}  -> this design cannot order the two")

def n_for(delta, disc, z_pow=0.84, z_a=1.96):
    return ((z_a + z_pow)**2 * disc) / delta**2

for delta in (0.05, 0.10, 0.20):
    print(f"  detect a {delta:.2f} paired difference at 80% power: "
          f"n >= {n_for(delta, 0.37):.0f} tasks")
print(f"discordance rate assumed: {(b+c)/n:.3f}  (measure yours; it drives n)")

n=30: A-B difference +0.100, 95% CI [-0.114, +0.314]
interval covers zero: True  -> this design cannot order the two
  detect a 0.05 paired difference at 80% power: n >= 1160 tasks
  detect a 0.10 paired difference at 80% power: n >= 290 tasks
  detect a 0.20 paired difference at 80% power: n >= 73 tasks
discordance rate assumed: 0.367  (measure yours; it drives n)

Thirty tasks cannot order two tools whose true difference is ten points. Public comparisons are routinely reported at that sample size and read as rankings. Before believing a claimed ordering --- yours or anyone else’s --- ask how many paired items produced it and what the discordance rate was; those two numbers determine everything.

1.5Coverage, Cost and the Two Budgets¶

Two more measurements complete the frame. The first is coverage: a retrieval stack, a search tool or a deep research agent can only surface what its index contains and its cutoff admits, and recall is not visible from the output.

import numpy as np
rng = np.random.default_rng(3)
N, n_rel = 500, 25
relevant = np.zeros(N, bool); relevant[rng.choice(N, n_rel, replace=False)] = True
score = rng.normal(0, 1, N) + 2.5*relevant       # a retriever that separates, imperfectly
order = np.argsort(-score)

print(" k   recall@k  precision@k  new hits")
prev = 0
for k in (1, 3, 5, 10, 20, 50):
    hits = int(relevant[order[:k]].sum())
    print(f"{k:>3}   {hits/n_rel:8.3f}  {hits/k:11.3f}  {hits-prev:8d}")
    prev = hits
k90 = next(k for k in range(1, N+1) if relevant[order[:k]].sum() >= 0.9*n_rel)
print(f"k for 90% recall: {k90}   precision there: "
      f"{relevant[order[:k90]].sum()/k90:.3f}")
print(f"a generator reading only the top 5 never sees "
      f"{n_rel - int(relevant[order[:5]].sum())} of the {n_rel} relevant documents")

 k   recall@k  precision@k  new hits
  1      0.040        1.000         1
  3      0.120        1.000         2
  5      0.200        1.000         2
 10      0.360        0.900         4
 20      0.600        0.750         6
 50      0.840        0.420         6
k for 90% recall: 76   precision there: 0.303
a generator reading only the top 5 never sees 20 of the 25 relevant documents

Recall climbs with $k$ while precision falls; a generator that reads only the top few passages is conditioning on a badly truncated sample of the evidence. Report the $k$ you used and the recall you measured on a seeded set of documents you know should be found.

The second is cost, which should be computed with the rate as a variable you supply from current documentation, never as a number copied from a textbook.

# Rates are YOUR inputs, read off your provider's current pricing page.
# Units: currency units per million tokens. Nothing here is a quoted price.
rate_in, rate_out = 1.0, 5.0
tok_in, tok_out   = 4200, 700          # per item, measured from your own logs
n_items           = 1800
retry             = 1.35               # observed retries + failed parses

per_item = (tok_in*rate_in + tok_out*rate_out) / 1e6
run = per_item * n_items * retry
print(f"per item: {per_item:.6f}   full pass: {run:.2f} currency units")
print(f"three passes for a self-consistency vote: {3*run:.2f}")
human_rate = 30.0                       # currency units per hour
minutes    = 2.5                        # minutes per item by hand
print(f"same corpus by hand: {n_items*minutes/60*human_rate:.2f}")
for r in (0.5, 1.0, 2.0, 4.0):
    print(f"  rate_in={r:<4} -> full pass {(tok_in*r + tok_out*rate_out)/1e6*n_items*retry:8.2f}")

per item: 0.007700   full pass: 18.71 currency units
three passes for a self-consistency vote: 56.13
same corpus by hand: 2250.00
  rate_in=0.5  -> full pass    13.61
  rate_in=1.0  -> full pass    18.71
  rate_in=2.0  -> full pass    28.92
  rate_in=4.0  -> full pass    49.33

The comparison that matters is not machine cost against zero, but machine cost plus verification cost against the human alternative --- and the verification cost is set by the sample size computed above. A pipeline that is cheap to run and expensive to trust has not saved anything.

1.6What to Record in a Paper¶

For AI tool use to be reproducible, a methods section must state, at minimum: the tool and its version (a release tag or commit hash, not a marketing name); the model or endpoint it was configured to call; the date of the run, because hosted endpoints change under a fixed name; the exact prompt, configuration file or script, deposited with the code rather than paraphrased; the sampling settings, including temperature or a statement that defaults were used; what was checked by hand, how many items were checked out of how many produced, and by whom; and where the raw logs and outputs are archived. If any generated content entered the manuscript, say which parts. The final example turns that list into a check you can run in continuous integration.

REQUIRED = ["tool_name", "tool_version", "model_or_endpoint", "date_run",
            "prompt_or_config", "temperature_or_sampling", "human_check",
            "n_checked", "n_total", "artifact_location"]

record = {
    "tool_name": "terminal coding agent",
    "tool_version": "v1.2.3 (git sha 4f1c2ab)",
    "model_or_endpoint": "vendor default, endpoint pinned in config.yaml",
    "date_run": "2026-03-14",
    "prompt_or_config": "prompts/screen_v3.txt (in repo)",
    "human_check": "dual independent screening of a random subsample",
    "n_checked": 60, "n_total": 1800,
    "artifact_location": "osf.io/<id>/logs",
}
missing = [f for f in REQUIRED if f not in record]
print(f"completeness: {len(REQUIRED)-len(missing)}/{len(REQUIRED)}")
print("missing:", missing)
print(f"verified fraction: {record['n_checked']/record['n_total']:.4f} "
      f"({record['n_checked']} of {record['n_total']})")
print("a methods sentence needs every line above; "
      f"this record is short {len(missing)}")

completeness: 9/10
missing: ['temperature_or_sampling']
verified fraction: 0.0333 (60 of 1800)
a methods sentence needs every line above; this record is short 1

The record above is missing its sampling settings, which is the field most often omitted and the one that most directly governs whether a rerun can reproduce the output. Keeping this dictionary next to the analysis and asserting its completeness in a test is a two-line habit that makes the methods paragraph writable at submission time instead of reconstructible from memory.

1.7Exercises¶

1. Using the break-even relation $p^\star = t_{\mathrm{review}}/ t_{\mathrm{manual}}$, compute the acceptance rate a tool must achieve on your own last week of work to be time-neutral. Then add a defect term as in the first example and state how large a latent-defect probability would erase the gain.

2. You will hand-check a subsample of an AI-assisted extraction to bound the error rate. Derive the number of items needed for a Wilson half-width of 0.03 under assumed true rates of 0.02, 0.10 and 0.30, and explain why the required $n$ is not monotone in the assumed rate over the whole unit interval.

3. Two colleagues report that their tool and yours “agree 90% of the time” on a classification task with 20% prevalence. Compute $\kappa$ and explain, in two sentences, what additional design would be needed before either agreement figure could support a claim about accuracy.

4. Computational. Take 60 items from a real task you perform, run two tools on all 60, and hand-label a random 20 as the reference standard. Report: raw agreement, $\kappa$, each tool’s accuracy against the reference with a Wilson interval, and the paired difference with its interval. State which of the four numbers your design was actually powered to estimate.

5. Computational. Write the methods-record dictionary of the last example for a real analysis of yours, add a test that fails when a field is missing, and generate the methods sentence from it automatically. Report which fields you could not fill in retrospectively --- those are the ones to capture at run time in future.