Coding and Analysis Agents

1Tutorial: Coding and Analysis Agents¶

A coding agent is a language model wired into a loop with file access, a shell, and an approval gate. For a statistician the practical consequence is that the agent’s output is not an answer but a change to the code that produces answers, which is the highest-leverage and least-inspected object in most research projects. A wrong number in a chat window is embarrassing; a wrong sign in a variance formula, committed and forgotten, contaminates every downstream result. This section describes the tool families by mechanism, then gives the measurements that make agent-assisted development auditable: what to read out of a tool-call log, how to size a diff so review actually catches things, what a context budget buys, and how to record a run so that it can be reproduced.

The evaluation frame of Section that section applies throughout: name the job, decide what evidence would show the tool does it on your codebase, and identify the failure mode. Here the failure mode is remarkably consistent across products: plausible code that runs. Nothing crashes. The statistics are subtly wrong.

1.1Terminal-Native Agents¶

Terminal agents run in a shell inside your repository. They read files, propose edits, execute commands, and observe the results before continuing. The durable distinctions among them are how much they read before they write, how they gate irreversible actions, whether they support an explicit plan-first mode, and what extension mechanism they expose --- an on-demand skill format, project context files, or an MCP client for reaching external tools (Section that section).

• [Claude Code] Terminal-native agent. Anthropic’s command-line agent: reads a repository, proposes edits, and runs commands under configurable approval. Fits: multi-file refactors, test-first work, and long tasks with checkpoints; supports project context files, an on-demand skill format, subagents, and MCP connections. Watch: it will happily run a long analysis script --- keep it on a branch and gate anything that writes outside the repository.

• [OpenAI Codex CLI] Terminal-native agent. OpenAI’s open-source command-line agent, with a sandboxed execution mode and both local and cloud-delegated task modes. Fits: contained tasks you want executed and verified without granting full filesystem reach; reads project context files. Watch: sandbox boundaries and network policy are configuration --- check what the current mode actually permits before pointing it at data.

• [Gemini CLI] Terminal-native agent. Google’s open-source terminal agent, extensible through MCP and project context files. Fits: repository work in shops already on Google infrastructure; scriptable non-interactive invocation for batch use. Watch: large-context retrieval encourages pointing the agent at whole trees, which mostly buys tokens --- see the context-budget example below.

The common operational advice for all three is the same, and it is not tool-specific: work on a branch, commit before you start so the diff is clean, require approval for commands that touch anything outside the working tree, and read the tool-call log rather than only the final patch. That log is data.

Agents emit a structured trace of what they read and wrote. Summarising it takes a few lines and tells you where to spend review attention. The log below is a fabricated fixture in the format these tools export.

import json
from collections import Counter

log = json.loads("""
[{"step":1,"tool":"grep","args":{"pat":"impute_"},"approved":null},
 {"step":2,"tool":"read","args":{"path":"src/impute.py"},"approved":null},
 {"step":3,"tool":"read","args":{"path":"tests/test_impute.py"},"approved":null},
 {"step":4,"tool":"edit","args":{"path":"src/impute.py"},"approved":true},
 {"step":5,"tool":"bash","args":{"cmd":"pytest -q"},"approved":true},
 {"step":6,"tool":"edit","args":{"path":"src/impute.py"},"approved":true},
 {"step":7,"tool":"bash","args":{"cmd":"pytest -q"},"approved":true},
 {"step":8,"tool":"edit","args":{"path":"src/model.py"},"approved":false},
 {"step":9,"tool":"read","args":{"path":"src/model.py"},"approved":null},
 {"step":10,"tool":"edit","args":{"path":"src/impute.py"},"approved":true}]
""")

kinds = Counter(e["tool"] for e in log)
reads  = kinds["read"] + kinds["grep"]
writes = kinds["edit"]
print("tool counts:", dict(sorted(kinds.items())))
print(f"read-before-write ratio: {reads/writes:.2f}")
gated = [e for e in log if e["approved"] is not None]
print(f"approval gates hit: {len(gated)}   declined: "
      f"{sum(1 for e in gated if e['approved'] is False)}")
touched = Counter(e["args"]["path"] for e in log if e["tool"] == "edit")
print("edits per file:", dict(touched))
churn = {f: c for f, c in touched.items() if c > 1}
print(f"files edited more than once (churn): {churn}")
print(f"blast radius: {len(touched)} file(s) modified out of "
      f"{len({e['args'].get('path') for e in log if 'path' in e['args']})} opened")

tool counts: {'bash': 2, 'edit': 4, 'grep': 1, 'read': 3}
read-before-write ratio: 1.00
approval gates hit: 6   declined: 1
edits per file: {'src/impute.py': 3, 'src/model.py': 1}
files edited more than once (churn): {'src/impute.py': 3}
blast radius: 2 file(s) modified out of 3 opened

Three readable signals. The read-before-write ratio says how much the agent looked at before it changed something --- a ratio near or below one on an unfamiliar codebase means it was guessing. The churn count says which file it revised repeatedly, which is where its uncertainty was concentrated and therefore where a reviewer should start. The blast radius says how many files a supposedly local change actually touched. None of these require the agent’s cooperation; they come from the log.

1.2IDE-Native Assistants¶

IDE assistants live in the editor and operate at the granularity of the buffer: inline completion, selection rewriting, chat over the open project. The mechanism difference from terminal agents is not intelligence but cadence --- the human sees every change as it is proposed, which is a strength for small edits and a liability for long ones, because continuous approval fatigues.

• [Cursor] IDE-native assistant (editor fork). An editor built around model-assisted editing, with project-wide context and a multi-file agent mode. Fits: iterative authoring where you want to see and accept each change; project rule files serve the role of a context file. Watch: agent mode edits many files at once while the interface still feels like autocomplete --- check the diff, not the buffer.

• [GitHub Copilot] IDE-native assistant (plugin). Inline completion and chat inside mainstream editors, with agent and code-review modes. Fits: boilerplate, docstrings, test scaffolds, and translation between languages you both know. Watch: completion is accepted with a keystroke and leaves no trace of having been machine-written; acceptance is not review.

Systematic evaluation of code-generating models began with functional-correctness testing --- generate a program, run it against unit tests, and score whether it passes Chen et al., 2021 --- and that framing is still the right one for a research group: the question is never whether the code looks right but whether it reproduces a known answer. The published evidence on assistants’ effect on people is genuinely mixed and worth reading rather than summarising: a controlled task study found faster completion with an assistant Peng et al., 2023, while a randomised trial with experienced maintainers on their own repositories found measured completion time moving in the opposite direction from participants’ own impressions Becker et al., 2025. Security-oriented studies point at a distinct hazard --- code produced with assistant help was more often insecure while its authors were more confident in it Perry et al., 2023, and a systematic audit of generated completions in security-sensitive scenarios found a substantial share containing known weakness classes Pearce et al., 2022. For statistical code the analogous hazard is not a buffer overflow but a formula that is quietly wrong.

The characteristic defect is code that runs and is subtly incorrect. Both functions below are the kind of thing an assistant produces and a hurried reviewer accepts; both pass a casual smoke test.

def median_ok(x):                      # correct
    y = sorted(x); n = len(y)
    return y[n//2] if n % 2 else 0.5*(y[n//2 - 1] + y[n//2])

def median_agent(x):                   # plausible-looking, wrong on even n
    y = sorted(x); n = len(y)
    return y[n//2]

def ci_ok(p, n, z=1.96):               # correct
    return z * (p*(1-p)/n) ** 0.5

def ci_agent(p, n, z=1.96):            # wrong: forgets the square root on n
    return z * (p*(1-p)) ** 0.5 / n

CASES = [([1, 2, 3, 4], None), ([5, 1, 3], None), ([2.0, 8.0], None)]
fails = [c for c, _ in CASES if abs(median_agent(c) - median_ok(c)) > 1e-12]
print(f"median: {len(fails)} of {len(CASES)} cases differ -> {fails}")

pn = [(0.5, 100), (0.1, 25), (0.3, 400)]
bad = [(p, n) for p, n in pn if abs(ci_agent(p, n) - ci_ok(p, n)) > 1e-9]
print(f"ci: {len(bad)} of {len(pn)} cases differ")
print(f"  at p=0.5,n=100 the agent's half-width is {ci_agent(.5,100):.5f} "
      f"vs correct {ci_ok(.5,100):.5f}")
smoke = [([1, 3, 5], 3.0)]             # the test a hurried reviewer writes
print(f"a single odd-length smoke test passes: "
      f"{all(median_agent(x) == v for x, v in smoke)}")
print("the defect is invisible to the smoke test and visible to the paired oracle")

median: 2 of 3 cases differ -> [[1, 2, 3, 4], [2.0, 8.0]]
ci: 3 of 3 cases differ
  at p=0.5,n=100 the agent's half-width is 0.00980 vs correct 0.09800
a single odd-length smoke test passes: True
the defect is invisible to the smoke test and visible to the paired oracle

The median error appears only on even-length input; the interval error is a missing square root and is off by an order of magnitude at $n=100$, which is exactly the sort of thing that produces a suspiciously narrow confidence interval in a simulation study. The defence is not more careful reading. It is a differential test against a reference implementation --- numpy.median, statsmodels, an analytic value --- run over a set of cases that deliberately includes the boundaries. Ask the agent for the reference comparison, not for reassurance.

1.3Open-Source Command-Line Tools¶

An open-source layer sits under both families: agents you can run against a model endpoint of your choosing, including a local one (Section that section). For research the relevant properties are auditability, pinnability, and the ability to keep code and data inside an institutional boundary.

• [Aider] Open-source terminal agent, git-centric. Pairs with you in a repository and commits each accepted change as its own git commit. Fits: incremental work where you want the version history to record exactly what the agent did. Watch: many small commits are easy to approve individually and hard to review as a whole --- read the squashed diff before merging.

• [Cline] Open-source IDE agent (extension). An editor-embedded agent with explicit plan-then-act separation and MCP support. Fits: work where you want to inspect and amend the plan before any file is touched. Watch: a plan approved and then not compared against the resulting diff is theatre; check the result against the plan.

• [OpenCode] Open-source terminal agent, provider-agnostic. A terminal agent designed to run against many model providers, including self-hosted endpoints. Fits: governed environments and reproducibility work where the endpoint must be pinned and recorded. Watch: behaviour changes substantially with the backing model, so a workflow validated on one endpoint must be re-validated on another.

Provider-agnostic tools make one thing explicit that managed products hide: the harness and the model are separate factors, and a result attributed to “the agent” is a result from a particular harness-model pair. Record both.

1.4Reviewing What the Agent Wrote¶

Agents make it cheap to produce large diffs, and large diffs are where review stops working. The relationship is worth writing down, because it converts a vague unease into a patch-size policy.

A detection model with parameters you estimate from your own review history. The point is the shape --- detection probability decays with diff size while defects accumulate linearly --- not the constants.

# All three constants are YOURS: estimate them from your own review history.
d0, alpha = 60.0, 0.95       # P(catch a defect) = alpha / (1 + lines/d0)
defect_rate = 0.012          # defects per changed line

print("diff size  P(catch)  expected defects  expected escapes")
for lines in (20, 50, 150, 400, 900):
    p = alpha / (1 + lines / d0)
    exp_def = defect_rate * lines
    print(f"{lines:>9}  {p:>8.3f}  {exp_def:>16.2f}  {exp_def*(1-p):>16.2f}")

def escapes(lines):
    return defect_rate*lines * (1 - alpha/(1 + lines/d0))

budget = 0.5                 # escaped defects tolerated per patch
m = 1
while escapes(m + 1) <= budget:
    m += 1
n_patch = -(-900 // m)
print(f"largest single patch meeting a per-patch budget of {budget}: {m} lines")
print(f"900 lines as one patch: {escapes(900):.2f} escapes; "
      f"as {n_patch} patches of {m}: {n_patch*escapes(m):.2f}")
print("splitting helps because P(catch) falls with diff size, "
      "not because fewer defects are written")

diff size  P(catch)  expected defects  expected escapes
       20     0.713              0.24              0.07
       50     0.518              0.60              0.29
      150     0.271              1.80              1.31
      400     0.124              4.80              4.21
      900     0.059             10.80             10.16
largest single patch meeting a per-patch budget of 0.5: 72 lines
900 lines as one patch: 10.16 escapes; as 13 patches of 72: 6.38
splitting helps because P(catch) falls with diff size, not because fewer defects are written

Expected escapes grow faster than linearly in patch size because two things move at once. Splitting the same 900 lines into small patches reduces expected escapes substantially without reducing the number of defects written, purely by restoring the reviewer’s detection probability. This is why “ask for one change at a time” is a measurement statement rather than a style preference.

The second review lever is knowing where to look, and the version-control metadata already answers that.

Per-file authorship and churn from a `git diff -{`-numstat}-shaped record, annotated with who authored each hunk.

numstat = """4\t0\tsrc/impute.py\tagent
120\t8\tsrc/model.py\tagent
6\t2\ttests/test_model.py\thuman
64\t60\tsrc/model.py\thuman
3\t0\tREADME.md\tagent
40\t35\tsrc/model.py\tagent
12\t1\ttests/test_impute.py\thuman"""

rows = []
for line in numstat.strip().split("\n"):
    add, dele, path, who = line.split("\t")
    rows.append((int(add), int(dele), path, who))

tot_add = sum(r[0] for r in rows)
ag_add = sum(r[0] for r in rows if r[3] == "agent")
print(f"lines added: {tot_add}   agent-authored: {ag_add} ({ag_add/tot_add:.1%})")

test_add = sum(r[0] for r in rows if r[2].startswith("tests/"))
print(f"test lines added: {test_add}  ratio test:source = "
      f"{test_add/(tot_add-test_add):.2f}")
agent_test = sum(r[0] for r in rows if r[3] == "agent" and r[2].startswith("tests/"))
print(f"of the agent's {ag_add} lines, {agent_test} are tests")

churn = {}
for add, dele, path, who in rows:
    churn[path] = churn.get(path, 0) + dele
print("rewritten lines per file:", dict(sorted(churn.items())))
print(f"src/model.py: {churn['src/model.py']} of "
      f"{sum(r[0] for r in rows if r[2]=='src/model.py')} added lines were "
      f"later deleted -- churn is where review effort belongs")

lines added: 249   agent-authored: 167 (67.1%)
test lines added: 18  ratio test:source = 0.08
of the agent's 167 lines, 0 are tests
rewritten lines per file: {'README.md': 0, 'src/impute.py': 0, 'src/model.py': 103, 'tests/test_impute.py': 1, 'tests/test_model.py': 2}
src/model.py: 103 of 224 added lines were later deleted -- churn is where review effort belongs

Two red flags in a small record. The agent contributed two thirds of the added lines and none of the tests, so the validation set was written by one author and the code by another --- which is the right way round, but only by accident here; if the agent had written both, the test suite would not be an independent check. And nearly half the lines added to src/model.py were subsequently deleted, marking it as the file the agent was least sure about. Compute these two numbers before reviewing, and read the churny file first.

1.5Context, Cost and the Budget That Bites¶

Agents are often given more context than they can use, on the theory that more is safer. It is not free, and the biggest files in a research repository are usually the least informative.

repo = {"src/model.py": 61000, "src/impute.py": 18000, "src/plots.py": 31000,
        "src/io_utils.py": 9000, "tests/test_model.py": 26000,
        "tests/test_impute.py": 9000, "data/raw_dump.csv": 480000,
        "notebooks/scratch.ipynb": 96000, "README.md": 7000, "AGENTS.md": 4000}
CHARS_PER_TOKEN = 4.0        # crude, but stable enough for budgeting
tok = {f: c / CHARS_PER_TOKEN for f, c in repo.items()}
total = sum(tok.values())
print(f"repo total: {total:,.0f} tokens across {len(tok)} files")

# `budget` is yours: read the usable context off your tool's documentation.
for budget in (6_000, 24_000, 96_000):
    fits, used = [], 0.0
    for f, t in sorted(tok.items(), key=lambda kv: kv[1]):
        if used + t <= budget * 0.5:      # keep half the window for reasoning
            fits.append(f); used += t
    print(f"budget {budget:>6,}: {len(fits):>2}/{len(tok)} files fit, "
          f"{used/total:>5.1%} of the tree; largest excluded = "
          f"{max(set(tok)-set(fits), key=tok.get, default='none')}")
logic = {f: t for f, t in tok.items() if f.endswith((".py", ".md"))}
print(f"code and docs only: {sum(logic.values()):,.0f} tokens "
      f"({sum(logic.values())/total:.1%} of the tree)")
print("the data dump and the scratch notebook are most of the bytes and none")
print("of the logic: exclude them before you reach for a bigger window")

repo total: 185,250 tokens across 10 files
budget  6,000:  2/10 files fit,  1.5% of the tree; largest excluded = data/raw_dump.csv
budget 24,000:  5/10 files fit,  6.3% of the tree; largest excluded = data/raw_dump.csv
budget 96,000:  8/10 files fit, 22.3% of the tree; largest excluded = data/raw_dump.csv
code and docs only: 41,250 tokens (22.3% of the tree)
the data dump and the scratch notebook are most of the bytes and none
of the logic: exclude them before you reach for a bigger window

A single data dump dominates the byte count and contributes nothing to the logic; excluding it and the scratch notebook buys more usable context than any change of tool. Write those exclusions into the repository’s ignore file for agent tooling and the problem disappears permanently. The related budget surprise is that a conversation’s input cost is not linear in the number of turns.

Rates below are variables you fill in from current documentation; nothing here is a quoted price.

sys_tok, turn_in, turn_out = 1500, 350, 500     # your own measured sizes
rate_in, rate_out = 1.0, 5.0                    # currency per million tokens

ctx, billed_in, billed_out, rows = sys_tok, 0, 0, []
for t in range(1, 13):
    ctx += turn_in
    billed_in += ctx
    billed_out += turn_out
    ctx += turn_out
    if t in (1, 2, 4, 8, 12):
        rows.append((t, ctx, billed_in,
                     (billed_in*rate_in + billed_out*rate_out)/1e6))
print("turn  context  cumulative input tokens  cumulative cost")
for t, c, b, cost in rows:
    print(f"{t:>4}  {c:>7,}  {b:>23,}  {cost:>15.4f}")
naive = 12 * (sys_tok + turn_in)
print(f"\nnaive 'input = 12 x prompt' estimate: {naive:,} tokens")
print(f"actual cumulative input: {billed_in:,} tokens "
      f"({billed_in/naive:.1f}x the naive figure)")
print("input grows quadratically in turns because the transcript is re-sent")

turn  context  cumulative input tokens  cumulative cost
   1    2,350                    1,850           0.0043
   2    3,200                    4,550           0.0095
   4    4,900                   12,500           0.0225
   8    8,300                   38,600           0.0586
  12   11,700                   78,300           0.1083

naive 'input = 12 x prompt' estimate: 22,200 tokens
actual cumulative input: 78,300 tokens (3.5x the naive figure)
input grows quadratically in turns because the transcript is re-sent

Cumulative input grows quadratically because the whole transcript is re-sent each turn, so the naive estimate understates the total by several fold in a short session and much more in a long one. The practical consequences are structural: start a fresh session for an unrelated task, keep long reference material in files the agent reads on demand rather than in the conversation, and delegate long subtasks to fresh-context children when the harness supports it (Section that section).

1.6Making an Agent-Assisted Analysis Reproducible¶

Agent-assisted code has more axes of variation than ordinary code, and recording only the random seed pins the least important one.

import hashlib, json

def fingerprint(script, env, seed, sampler):
    payload = json.dumps({"script": script, "env": env, "seed": seed,
                          "sampler": sampler}, sort_keys=True)
    return hashlib.sha256(payload.encode()).hexdigest()[:12]

base = dict(script="fit.py@a91c3f", env="py3.12/numpy2.1", seed=7,
            sampler="greedy")
runs = {
    "same everything":        dict(base),
    "seed changed":           dict(base, seed=8),
    "sampler stochastic":     dict(base, sampler="temp=0.7"),
    "env drifted":            dict(base, env="py3.12/numpy2.2"),
    "script edited by agent": dict(base, script="fit.py@d4e017"),
}
ref = fingerprint(**base)
for name, cfg in runs.items():
    fp = fingerprint(**cfg)
    print(f"{name:<24} {fp}  {'match' if fp == ref else 'DIFFERS'}")
print()
print("recording only the seed pins 1 of the 4 axes above;")
print(f"the fingerprint pins {len(base)} and is one line in a methods section")

same everything          2bc5db2afd10  match
seed changed             efa991b66a92  DIFFERS
sampler stochastic       2db7fec37d8f  DIFFERS
env drifted              3fa3f0e2282a  DIFFERS
script edited by agent   13da6f730737  DIFFERS

recording only the seed pins 1 of the 4 axes above;
the fingerprint pins 4 and is one line in a methods section

Four axes, four ways for a rerun to diverge, and a twelve-character fingerprint that distinguishes them. Emit it from the analysis script, write it into the log and into the figure metadata, and a disagreement between two runs becomes a one-line diagnosis instead of an afternoon.

1.7What to Record in a Paper¶

A methods section describing agent-assisted analysis must state: the harness and its version, as a release tag or commit hash; the model endpoint it was configured against, since harness and model are separate factors; the date of the run; whether the agent could execute code and with what permissions; the prompt, project context file, or plan document, deposited with the code rather than described; what the agent produced versus what the authors wrote, at least at file granularity, which version control already records if commits are attributed; how the generated code was verified --- reference implementations, differential tests against a validated result, or line-by-line review, naming which; and who reviewed it. If a generated function computes anything that appears in a table or figure, the verification of that specific function belongs in the supplement. The fingerprint from the previous example makes most of this a single recorded string.

1.8Exercises¶

1. Take the detection model of the third example and fit its two parameters crudely to your own experience: estimate your probability of catching a seeded defect in a 20-line and a 200-line patch, solve for $\alpha$ and $d_0$, then state the maximum patch size consistent with an escape budget of one defect per ten patches.

2. Explain why a test suite written by the same agent that wrote the code is not an independent validation, and describe two concrete designs that restore independence at different costs.

3. Using the token-budget example, derive a closed-form expression for cumulative input tokens after $T$ turns as a function of the system prompt size and the per-turn input and output sizes. Confirm it reproduces the printed cumulative figure at $T=12$, and use it to find the turn at which a fresh session becomes cheaper than continuing.

4. Computational. Ask an agent to implement a statistical routine you can verify independently --- a weighted quantile, a survival estimator, a bootstrap interval. Write a differential test against a reference implementation over at least 200 randomly generated inputs including boundary cases, and report the disagreement rate with a Wilson interval. Then report how many of the disagreements a line-by-line reading would plausibly have caught.

5. Computational. Parse a real tool-call log from an agent session in your own repository and report the read-before-write ratio, the churn distribution over files, and the blast radius. Then review the diff, recording which file you inspected first and where the defects actually were. Report whether churn predicted defect location on your session.