Agentic Analysis Environments

1Tutorial: Agentic Analysis Environments¶

An agentic analysis environment is a place where a language model can run your code, read your files, and keep what it produced. It is not a chat window with a copy-paste step in the middle. That difference is the whole subject of this tutorial, because once the model can execute, three things become true at once: the work gets faster, the number of analysis decisions made per hour rises by an order of magnitude, and the record of which decisions were made stops being something a human wrote down. The first is why these environments are worth using. The second and third are why a statistician has to use them differently from a software engineer. This section covers the three families of environment a researcher will actually meet --- notebook-resident agents, hosted analysis agents, and workspace agents that persist artifacts --- and then turns to the question the tools cannot answer for you: when the agent should drive the analysis and when it should not.

Two ground rules before the tools. Specific capabilities change quickly, so this book teaches an evaluation frame rather than a snapshot; check the current documentation for any product named here. And the patterns underneath these products --- skills, project context files, the Model Context Protocol, spec-driven working --- are treated in depth in \Sthat section; they are named here only where a tool implements one.

1.1The Trajectory Log Is Data, So Analyse It¶

Every agentic environment writes a record of what it did: which tool it called, with what arguments, whether the call succeeded, how long it took. This is the single most under-used artifact in the whole area. It is a structured dataset about your own analysis, and reading it takes seconds. The example below parses a short log and answers the questions a reviewer would ask: how many models were fitted, how many steps could have changed state, what was written to disk, and where the retries went.

import json
from collections import Counter

LOG = """
{"t":1,"tool":"read_file","args":{"path":"cohort.csv"},"ok":true,"ms":40}
{"t":2,"tool":"python","args":{"cell":"df.describe()"},"ok":true,"ms":900}
{"t":3,"tool":"python","args":{"cell":"df.groupby('arm').mean()"},"ok":true,"ms":700}
{"t":4,"tool":"python","args":{"cell":"fit1 = ols(...)"},"ok":false,"ms":300}
{"t":5,"tool":"python","args":{"cell":"fit1 = ols(...)"},"ok":true,"ms":800}
{"t":6,"tool":"write_file","args":{"path":"table1.csv"},"ok":true,"ms":30}
{"t":7,"tool":"python","args":{"cell":"fit2 = ols(... + age)"},"ok":true,"ms":810}
{"t":8,"tool":"python","args":{"cell":"fit3 = ols(... + age + site)"},"ok":true,"ms":830}
{"t":9,"tool":"write_file","args":{"path":"fig1.png"},"ok":true,"ms":50}
{"t":10,"tool":"web_search","args":{"q":"ols robust se"},"ok":true,"ms":1200}
"""
steps = [json.loads(l) for l in LOG.strip().splitlines()]
WRITES = {"write_file", "python"}          # anything that can change state

n = len(steps)
tools = Counter(s["tool"] for s in steps)
fails = [s["t"] for s in steps if not s["ok"]]
models = [s for s in steps if s["tool"] == "python" and "ols(" in s["args"]["cell"]]
artifacts = [s["args"]["path"] for s in steps if s["tool"] == "write_file"]

print(f"steps                 {n}")
print(f"tool mix              {dict(tools)}")
print(f"failed steps          {fails}")
print(f"state-changing steps  {sum(s['tool'] in WRITES for s in steps)}")
print(f"model fits attempted  {len(models)}")
print(f"artifacts written     {artifacts}")
print(f"wall time (s)         {sum(s['ms'] for s in steps)/1000:.2f}")
print(f"retry overhead        {sum(s['ms'] for s in steps if not s['ok'])/1000:.2f} s")

steps                 10
tool mix              {'read_file': 1, 'python': 6, 'write_file': 2, 'web_search': 1}
failed steps          [4]
state-changing steps  8
model fits attempted  4
artifacts written     ['table1.csv', 'fig1.png']
wall time (s)         5.66
retry overhead        0.30 s

Four model fits were attempted and one table was written. Unless the write-up says so, the reader will assume there was one model. That gap between what was run and what was reported is the recurring hazard of this whole section, and the log is the only place it is visible.

1.2Notebook-Resident Agents¶

The lowest-friction entry point is an agent that lives inside the notebook you already use: it reads the cells, proposes new ones, runs them in your kernel and sees the tracebacks. JupyterLab has extensions of this kind, the major cloud notebook services ship their own, and the IDE-based editors expose the same capability against a notebook file.

• [Notebook-resident agents] In-kernel assistant. Proposes, edits and executes cells inside a live notebook kernel, conditioning on the variables and errors already present. Fits: exploratory analysis and debugging, where the feedback loop is the point. Watch: the kernel is mutable and shared. An agent that runs cells out of order leaves a notebook whose saved outputs cannot be reproduced by reading it top to bottom.

That failure is worth making concrete, because it is invisible on screen. A notebook records an execution counter per cell; if those counters are not in document order, at least one cell saw a value that a clean rerun would not supply. The example replays a notebook twice --- once in execution order, once top to bottom --- and compares which cell produced each value that was read.

import json

NB = json.loads("""
{"cells": [
 {"id":"a","execution_count":1,"reads":[],     "writes":["df"] },
 {"id":"b","execution_count":4,"reads":["df"], "writes":["df"] },
 {"id":"c","execution_count":2,"reads":["df"], "writes":["tab"]},
 {"id":"d","execution_count":5,"reads":["df"], "writes":["fit"]},
 {"id":"e","execution_count":3,"reads":["tab"],"writes":["fig"]}
]}
""")["cells"]

def replay(cells):
    """For each cell, which cell produced each value it read."""
    made, prov = {}, {}
    for c in cells:
        prov[c["id"]] = {v: made.get(v, "MISSING") for v in c["reads"]}
        for v in c["writes"]:
            made[v] = c["id"]
    return prov

paper_order = [c["id"] for c in NB]
run_order = [c["id"] for c in sorted(NB, key=lambda c: c["execution_count"])]
as_run, clean = replay(sorted(NB, key=lambda c: c["execution_count"])), replay(NB)

print("cells as written  :", paper_order)
print("cells as executed :", run_order)

direct = {cid for cid in paper_order
          for v in as_run[cid] if as_run[cid][v] != clean[cid][v]}
for cid in sorted(direct):
    v = next(v for v in as_run[cid] if as_run[cid][v] != clean[cid][v])
    print(f"\ncell {cid} read '{v}' from {as_run[cid][v]}; "
          f"a clean rerun supplies {clean[cid][v]}")

tainted = set(direct)                      # propagate downstream
for c in NB:
    if any(clean[c["id"]].get(v) in tainted for v in c["reads"]):
        tainted.add(c["id"])
print(f"\ndirectly affected  {sorted(direct)}")
print(f"plus downstream    {sorted(tainted)}")
print(f"share of notebook  {len(tainted)/len(NB):.2f}")

cells as written  : ['a', 'b', 'c', 'd', 'e']
cells as executed : ['a', 'c', 'e', 'b', 'd']

cell c read 'df' from a; a clean rerun supplies b

directly affected  ['c']
plus downstream    ['c', 'e']
share of notebook  0.40

One out-of-order cell silently invalidates forty percent of this notebook: the table was built before the age filter was applied, and the figure was built from the table. Nothing errors, nothing looks wrong, and the numbers in the saved output are not the numbers the code produces. The mitigation is a discipline rather than a tool --- finish every agent-assisted notebook with a restart and a full rerun, and compare the artifacts --- and it is the single highest-value habit in this section.

1.3Hosted Analysis Agents¶

The second family runs the code for you, in the vendor’s sandbox: you upload a file, describe the analysis in prose, and receive plots, tables and a narrative with the code that produced them. The code-execution modes built into the major hosted assistants work this way, as do the standalone data-analysis products built on the same idea.

• [Hosted analysis agents] Managed sandbox. Executes analysis code on uploaded data inside the provider’s environment and returns code, output and prose together. Fits: quick descriptive work, format wrangling, first looks at an unfamiliar file. Watch: data governance and environment drift. The upload leaves your control, the package versions are not yours, and the sandbox is usually ephemeral, so the analysis is not re-runnable next year.

For a statistician the governance question comes first and is often decisive: protected health information, student records and unpublished collaborator data generally may not be uploaded, and no amount of convenience changes that. Where the data are public or synthetic, the practical limitation is reproducibility --- an environment you cannot pin is an environment you cannot report. Section that section covers the alternative when the data cannot leave your machine.

1.4Workspace Agents That Persist Artifacts¶

The third family sits between the other two. The agent works in a durable workspace with your files, your environment and your compute, executes code in long-lived kernels, and --- the distinguishing feature --- promotes selected outputs to versioned artifacts with recorded provenance, so a figure carries a pointer back to the code and inputs that made it. Claude Science is an instance of this pattern; so are agent-driven research environments built around a project database rather than a chat transcript.

• [Claude Science] Workspace agent with artifact lineage. Runs multi-language kernels against a persistent project workspace and stores figures, tables and datasets as versioned artifacts with their producing code; connects to external systems over MCP and loads procedures as skills (\Sthat section). Fits: multi-session analyses where provenance and re-execution matter, and work that must reach a cluster or an institutional data source. Watch: recorded lineage is a claim about how an artifact was made, not a guarantee that the analysis was correct; a perfectly reproducible wrong model is still wrong.

The advantage over a notebook is that the unit of record is the artifact rather than the transcript, which is what a methods section actually needs. The advantage over a hosted sandbox is that the environment is yours and can be pinned. Neither advantage touches the statistical content of the analysis, which is the subject of the rest of this section.

1.5Adaptivity Is the Statistical Hazard¶

Give a competent analyst a faster tool and they will try more specifications. That is the point of the tool and it is also the problem: the reported estimate is now selected from a family the reader never sees. This is the garden of forking paths, and agentic environments industrialise it, because the cost of one more adjustment set falls to a few seconds and the agent will cheerfully propose the next one. The classical treatments of selective and post-selection inference Benjamini & Hochberg, 1995Berk et al., 2013Taylor & Tibshirani, 2015 and of adaptive data analysis Dwork et al., 2015 apply directly and without modification.

The arithmetic is worth seeing under correlation, because analysis variants are not independent --- they share the outcome, the sample and most covariates.

import numpy as np
from scipy.stats import norm

rng = np.random.default_rng(0)
S = 200_000
ALPHA = 0.05
crit = norm.ppf(1 - ALPHA / 2)

def family_error(m, rho):
    """m analysis variants, equicorrelated at rho, all under the null."""
    shared = rng.normal(size=(S, 1))
    idio = rng.normal(size=(S, m))
    z = np.sqrt(rho) * shared + np.sqrt(1 - rho) * idio
    return np.mean(np.abs(z).max(axis=1) > crit)

print("agent quietly tries m specifications and reports the significant one")
print("  m   rho=0.0  rho=0.5  rho=0.9   Bonferroni bound")
for m in (1, 3, 5, 10, 25):
    row = [family_error(m, r) for r in (0.0, 0.5, 0.9)]
    print(f"{m:3d}   " + "  ".join(f"{v:6.3f} " for v in row)
          + f"  {min(1.0, m * ALPHA):6.3f}")

agent quietly tries m specifications and reports the significant one
  m   rho=0.0  rho=0.5  rho=0.9   Bonferroni bound
  1    0.050    0.050    0.051    0.050
  3    0.144    0.126    0.084    0.150
  5    0.227    0.183    0.101    0.250
 10    0.399    0.286    0.126    0.500
 25    0.723    0.466    0.165    1.000

Ten quietly-tried specifications take a nominal five percent error rate to forty percent when the variants are unrelated and to thirteen percent when they are strongly correlated. Correlation helps, but not enough to ignore, and the agent does not tell you $m$ unless you read the log. The defence is procedural, not statistical: fix the primary specification before the agent runs, log every variant it tries, and label everything else as exploratory.

A second form of adaptivity appears when an agent is used as a measuring instrument --- screening abstracts, coding free text, extracting variables. The tempting quality check is to run the pass twice and see whether the two agree. The example shows what that check does and does not buy when the two passes share a prompt, and therefore share a bias.

import numpy as np
from sklearn.metrics import cohen_kappa_score

rng = np.random.default_rng(11)
n = 300
truth = rng.random(n) < 0.30                    # 30% of abstracts are eligible

def run(sens, spec, bias):
    """One agent screening pass. bias = shared prompt-induced error."""
    keep = np.where(truth, rng.random(n) < sens, rng.random(n) < 1 - spec)
    return keep | bias                          # both passes inherit the bias

bias = (~truth) & (rng.random(n) < 0.10)        # same systematic false positive
a = run(0.90, 0.92, bias)
b = run(0.90, 0.92, bias)

agree = (a == b).mean()
kappa = cohen_kappa_score(a, b)
acc_a = (a == truth).mean()
both_wrong = ((a == b) & (a != truth)).mean()

print(f"raw agreement between the two passes   {agree:.3f}")
print(f"Cohen's kappa                          {kappa:.3f}")
print(f"accuracy of pass A against truth       {acc_a:.3f}")
print(f"accuracy of pass B against truth       {(b == truth).mean():.3f}")
print(f"items where they AGREE and are WRONG   {both_wrong:.3f}")
print(f"recall  A {(a & truth).sum() / truth.sum():.3f}   "
      f"union of A,B {((a | b) & truth).sum() / truth.sum():.3f}")

raw agreement between the two passes   0.870
Cohen's kappa                          0.729
accuracy of pass A against truth       0.850
accuracy of pass B against truth       0.840
items where they AGREE and are WRONG   0.090
recall  A 0.897   union of A,B 0.979

Nine percent of items are ones on which the two passes agree and are both wrong, and no amount of re-running finds them. Self-agreement measures stability, not validity; it is the split-half reliability of the instrument, and reliability has never implied accuracy. Validity requires a labelled subsample, which is the next subsection.

1.6Verification Is a Sample-Size Problem¶

Suppose the agent’s report states $M$ numbers and you can afford to check $k$ of them by hand. That is a survey-sampling problem with an exact answer, and “I spot-checked a few” is not a method. If the audit finds no errors, the Clopper--Pearson upper bound converts $k$ into a defensible statement about the whole report Wilson, 1927.

from statsmodels.stats.proportion import proportion_confint

M = 240          # numeric claims the agent's report makes
print("hand-check k of the M=240 numbers the agent reported; 0 errors found")
print("  k   Clopper-Pearson 95% upper bound on the error rate   expected # bad in M")
for k in (5, 10, 20, 40, 80):
    hi = proportion_confint(0, k, alpha=0.05, method="beta")[1]
    print(f"{k:3d}   {hi:.3f}                                          "
          f"{M * hi:6.1f}")

print()
print("now suppose the audit does find errors")
for k, e in ((10, 1), (40, 2), (40, 6), (120, 6)):
    lo, hi = proportion_confint(e, k, alpha=0.05, method="wilson")
    print(f"{e:2d}/{k:3d} wrong -> rate {e/k:.3f}  95% CI ({lo:.3f}, {hi:.3f})"
          f"   half-width {100*(hi-lo)/2:4.1f} pts")

hand-check k of the M=240 numbers the agent reported; 0 errors found
  k   Clopper-Pearson 95% upper bound on the error rate   expected # bad in M
  5   0.522                                           125.2
 10   0.308                                            74.0
 20   0.168                                            40.4
 40   0.088                                            21.1
 80   0.045                                            10.8

now suppose the audit does find errors
 1/ 10 wrong -> rate 0.100  95% CI (0.018, 0.404)   half-width 19.3 pts
 2/ 40 wrong -> rate 0.050  95% CI (0.014, 0.165)   half-width  7.6 pts
 6/ 40 wrong -> rate 0.150  95% CI (0.071, 0.291)   half-width 11.0 pts
 6/120 wrong -> rate 0.050  95% CI (0.023, 0.105)   half-width  4.1 pts

Checking five numbers and finding them all correct is consistent with a fifty-percent error rate; it licenses no claim at all. Twenty checks bound the rate below seventeen percent, and eighty below five. Choose $k$ from the claim you intend to make, and state $k$ in the paper. The same logic applied to citations rather than numbers is developed in \Sthat section, which also draws the distinction between a reference that does not exist and a real reference that does not support the sentence attached to it.

1.7When the Agent Should Drive, and When It Should Not¶

The useful decision rule is not about the model’s ability. It is about whether a cheap verifier exists. Where a step can be checked mechanically --- code that runs or does not, a query that returns rows, a merge whose row count is known in advance, a figure that must reproduce a published number --- the agent can be wrong often and the system still works, because errors are caught and retried. Where verification requires judgement --- choosing an estimand, deciding what confounding matters, interpreting a null result, writing the discussion --- there is no recovery mechanism and the output is a draft for a human to adjudicate. Fully autonomous end-to-end pipelines have been demonstrated Lu et al., 2024, and reading such a demonstration carefully is the fastest way to see where the verifier runs out.

Human review is the fallback, and it is not free or constant. Reviewers fatigue across gates, and batching many steps into one approval dilutes attention per step. Both effects are in the example, which counts how many bad steps reach the manuscript under different approval batch sizes.

import numpy as np

rng = np.random.default_rng(3)
N, STEPS, P_BAD = 20_000, 60, 0.10

def audit(k, d0=0.95, fatigue=0.97, dilution=0.93):
    """Approve in batches of k: fewer gates fatigue less, bigger batches dilute."""
    gates = STEPS // k
    d = d0 * fatigue ** np.arange(gates) * dilution ** (k - 1)
    bad = rng.random((N, gates, k)) < P_BAD
    survived = bad & (rng.random((N, gates, k)) > d[None, :, None])
    return survived.sum(axis=(1, 2)).mean(), d.mean(), gates

print(f"{STEPS} agent steps; {P_BAD:.0%} of them would corrupt the analysis, "
      f"so {STEPS * P_BAD:.0f} bad steps on average")
print(" batch k  gates  mean detection  E[bad steps reaching the manuscript]")
for k in (1, 2, 3, 5, 10, 20, 60):
    e, dbar, g = audit(k)
    print(f"{k:7d}  {g:5d}  {dbar:13.3f}  {e:9.2f}")
print(f"{'none':>7}  {0:5d}  {0.0:13.3f}  {STEPS * P_BAD:9.2f}")

60 agent steps; 10% of them would corrupt the analysis, so 6 bad steps on average
 batch k  gates  mean detection  E[bad steps reaching the manuscript]
      1     60          0.443       3.33
      2     30          0.588       2.45
      3     20          0.625       2.27
      5     12          0.604       2.39
     10      6          0.459       3.24
     20      3          0.232       4.59
     60      1          0.013       5.94
   none      0          0.000       6.00

The optimum is interior and shallow: approving every single step is nearly as bad as approving none, because sixty consecutive approvals exhaust the reviewer before the end. Under these assumptions --- which are yours to change --- the best available review still lets a third of the bad steps through. Review is a mitigation, not a control, and the real lever is reducing the number of steps that need reviewing.

The last piece is the budget. The choice between an agent-run and a hand-run analysis is a decision problem with three costs: the agent, the reviewer, and the errors that survive. Everything below is a variable you set.

# All rates are YOUR numbers, declared here as variables. Substitute your own.
ANALYST_HOUR = 60.00      # currency units per hour of a qualified analyst
REVIEW_MIN = 4.0          # minutes to check one agent-produced step by hand
AGENT_STEP = 0.05         # variable cost of one agent step, whatever you pay
STEPS = 120               # steps the task needs
P_BAD = 0.10              # steps that would be wrong if unchecked
COST_OF_ERROR = 2000.0    # cost of one wrong step reaching the manuscript

manual_hours = STEPS * 6.0 / 60          # 6 minutes per step done by hand
manual = manual_hours * ANALYST_HOUR

print(f"{'coverage':>9} {'review h':>9} {'agent':>8} {'labour':>8} "
      f"{'E[errors]':>10} {'risk':>9} {'total':>9}")
for cov in (0.0, 0.10, 0.25, 0.50, 1.0):
    checked = STEPS * cov
    hours = checked * REVIEW_MIN / 60
    labour = hours * ANALYST_HOUR
    agent = STEPS * AGENT_STEP
    errs = STEPS * P_BAD * (1 - 0.9 * cov)     # review catches 90% of what it sees
    risk = errs * COST_OF_ERROR
    print(f"{cov:9.2f} {hours:9.1f} {agent:8.2f} {labour:8.2f} "
          f"{errs:10.2f} {risk:9.0f} {agent + labour + risk:9.0f}")

print(f"\nfully manual: {manual_hours:.1f} h, cost {manual:.0f}, "
      f"E[errors] {STEPS * 0.02:.1f}, total "
      f"{manual + STEPS * 0.02 * COST_OF_ERROR:.0f}")

 coverage  review h    agent   labour  E[errors]      risk     total
     0.00       0.0     6.00     0.00      12.00     24000     24006
     0.10       0.8     6.00    48.00      10.92     21840     21894
     0.25       2.0     6.00   120.00       9.30     18600     18726
     0.50       4.0     6.00   240.00       6.60     13200     13446
     1.00       8.0     6.00   480.00       1.20      2400      2886

fully manual: 12.0 h, cost 720, E[errors] 2.4, total 5520

Under these particular numbers the agent’s own cost is negligible and the answer is driven entirely by the cost of an error: unreviewed agentic analysis is the worst option available, fully reviewed agentic analysis beats doing it by hand, and partial coverage is worse than either. Change COST_OF_ERROR to something small --- an internal exploratory plot --- and the ordering reverses completely. That is the actual decision rule, and it depends on the stakes of the output rather than on the sophistication of the tool.

1.8What to Record in a Paper¶

For agent-assisted analysis a methods section must state: the environment and its version or build, the model or models used and whether they were hosted or local, the date range over which the work was done, whether the agent executed code or only proposed it, and the location of the trajectory log or notebook history. It must state the primary specification and whether it was fixed before the agent ran, together with the number of alternative specifications examined. It must state what was checked by hand --- how many items, drawn how, with what error rate and interval --- and it must state that the final analysis was re-executed from a clean state and reproduced the reported artifacts. Prompts, project context files and skills belong in the supplement as files, not as paraphrase. A reproducibility checklist of the kind now standard in machine learning conferences Pineau et al., 2021 covers most of this and adapts directly; documentation conventions for data and models Gebru et al., 2021Mitchell et al., 2019 cover the rest. The honest one-line test: could a reader with your data and your supplement obtain your Table 1 without asking you a question?

1.9Exercises¶

1. Take a trajectory log from your own agentic session and compute, as in the first example, the number of model fits attempted, the number of artifacts written, and the ratio between them. Write the sentence you would need to add to a methods section so that the ratio is not misleading.

2. Extend the notebook-provenance example so that each cell also records a hash of its output. Show that comparing hashes after a restart-and-run-all detects exactly the tainted set the example identifies, and construct a case where it detects a cell the provenance analysis misses.

3. Using the family-error example, find the number of correlated specifications ($\rho = 0.5$) at which the probability of at least one spurious result exceeds one half. Then state what you would have to log, and report, for a Bonferroni or Benjamini--Hochberg correction to be legitimate after the fact.

4. Computational. An agent screens 2000 abstracts. You can afford to hand-label 150. Design the audit: decide how to draw the subsample (simple random, or stratified on the agent’s own decision), compute the resulting interval for sensitivity and for specificity, and show which design gives the narrower interval for recall at the same labelling cost.

5. Computational. Re-run the review-gate example with detection probability that does not fatigue but with dilution that is stronger, and with an error cost that is incurred per surviving bad step. Find the batch size that minimises total expected cost, and report how sensitive the optimum is to the fatigue parameter.