Writing Code With a Model

“Vibe coding” is the practice of writing software by describing what you want in natural language and letting a large language model produce the code, iterating on the description rather than on the source. For a statistician the appeal is immediate: much of our code is glue --- reshaping a data frame, wiring up a plot, translating a formula from a paper into a fitting routine --- and a model that writes plausible glue on demand removes a real tax on research time. The risk is equally immediate, and it is a statistical one. A language model is a conditional distribution over token sequences trained to be fluent, not correct, so its output is a draft with an unknown and unreported error rate, and the errors that matter most are the silent ones: a routine that runs to completion, returns a number of the right magnitude, and computes the wrong quantity Brown et al., 2020Ouyang et al., 2022. This section treats generated code the way we would treat any other measurement instrument, by asking what its error distribution looks like, how to detect its failures, and what verification the analyst must supply. The organising principle is that the human retains the specification and the verification; only the typing is delegated.

1What the Models Are Actually Good At¶

The empirical picture is uneven in a way that is predictable from how these models are trained. They are strong on code that is heavily represented in public repositories and weakly constrained by context: standard data manipulation, plotting, file wrangling, boilerplate around a well-known library, and translation between languages. They are weak where correctness depends on something not stated in the prompt --- the exact parameterisation of a distribution, whether a matrix is column- or row-major, what a package’s default actually is in the installed version --- and weakest of all where an error produces output that looks reasonable. That ordering, rather than any general claim about capability, is what should determine how much verification a given piece of generated code receives. It also interacts with the interface: an inline completion in an editor is accepted a few tokens at a time, whereas a terminal agent returns a multi-file diff, and the amount of generated code a researcher accepts per unit of reading grows by orders of magnitude between the two.

1.1The Prompt as a Specification¶

The useful mental shift is to treat the prompt as a model specification rather than a request. A statistical model written in notation is unambiguous about its estimand, its parameterisation and its assumptions; a prose prompt is not, and every ambiguity is resolved silently by the model in a way that does not appear in the code it returns. In practice this means stating the estimand explicitly, naming the parameterisation (scale versus rate, log-odds versus probability), fixing the expected input and output shapes, and saying what the function should do at the boundaries. A prompt written this way is longer than a request and is worth the extra minute, because what it buys is the ability to check the answer against something you wrote down first.

1.2Verification is Not Optional¶

Generated code needs the verification a statistician already knows how to do, applied in a specific order. First, check the implementation against a case with a known analytic answer --- a closed-form estimator on a small example, a limiting case, a symmetry the output must obey. Second, simulate: generate data from a process you control and confirm the routine recovers the parameters you put in, which is the single most effective check available and the one most often skipped because the code appears to work on the real data. Third, verify against an independent implementation if one exists. Tests written by the same model that wrote the code will share its misunderstandings, so the tests that matter most --- the ones encoding what the answer should be --- are the ones you write yourself. (Property-based testing libraries such as Hypothesis fit this step well, because what you supply is the invariant and what the tool supplies is a search for inputs that violate it.)

beta_hat = fit_logistic(X, y) # the generated routine under test assert np.allclose(beta_hat, beta_true, atol=0.1), beta_hat

1.3Reproducibility When Part of the Analysis Was Generated¶

Sampling makes the generator itself a random object: the same prompt run twice can produce different programs, both of which may pass a cursory reading. The consequence for reproducible research is direct, and it is the opposite of what the interactive workflow suggests. The prompt is not the artefact; the generated code is. An analysis is reproducible when the exact source that produced the numbers is committed, pinned to package versions and rerunnable, and no amount of prompt archiving substitutes for that Donoho, 2017. Figure the figure sets out the loop that keeps this property: specify, generate, verify against a known answer, and only then commit, with a failed check sending you back to the specification rather than forward to a patch.

1.4Where This Leaves the Statistician¶

The honest summary is that these tools shorten the distance to a first draft and do almost nothing for the distance to a correct result, which means the value they deliver depends entirely on whether the verification discipline is in place. That is a comfortable position for our field, because the discipline in question --- state the estimand, check against a known answer, simulate to confirm recovery --- is one statisticians already teach. The failure mode to watch for is not the model writing obvious nonsense; it is the analyst gradually accepting code they have not checked because the last twenty suggestions were fine.

1.5Tools in practice¶

The tools that generate code for you differ along one axis that actually matters for the verification discipline above: how much of your project the model reads before it writes, and how much it can change without you looking. A completion that appears inline as you type is a suggestion you accept token by token; a terminal agent given a repository and a task will edit a dozen files and hand you a diff. The second is enormously more productive and enormously easier to accept without reading, and the failure mode of this section --- the analyst who gradually stops checking because the last twenty suggestions were fine --- scales with exactly that. Specific capabilities and pricing move fast enough that any number printed here would be wrong before the book reached a shelf; check the current documentation for those, and use the structural distinctions below, which do not move.

• [GitHub Copilot] IDE-native completion. Suggests the next lines inline from surrounding file context, accepted or rejected keystroke by keystroke. Fits: boilerplate, and the glue code where the model is strongest. Watch: the acceptance decision is made in under a second and the suggestion looks like your own code once accepted; this is where silently wrong index arithmetic enters a project.

• [Cursor / Windsurf] AI-first editor. Indexes the whole repository and applies multi-file edits, showing a diff for approval. Fits: refactors and cross-file changes where the specification is clear and the change is mechanical. Watch: approving a large diff is an unblinded review under time pressure. Require the test suite to pass before the diff is read, not after.

• [Claude Code / Codex CLI / Gemini CLI] Terminal agents. Run in a shell with filesystem and command access, iterating against a real interpreter and test suite until the task is done. Fits: tasks with a machine-checkable definition of done --- make the tests pass, reproduce this figure, port this script. Watch: the loop optimises whatever check it can see, so a weak test suite is an instruction to write code that satisfies a weak test suite. Work on a branch, in a container, with the data mounted read-only.

• [Jupyter AI / Colab AI] Notebook-embedded assistants. Generate and edit cells with access to the live kernel state. Fits: exploratory analysis, where the cost of a wrong cell is low and the next cell reveals it. Watch: notebooks execute out of order, so a cell that worked when generated may not work in a fresh kernel. Restart-and-run-all before anything from a notebook becomes a result.

• [Hypothesis] Property-based testing. Searches for inputs violating a property you state, and shrinks any counterexample to a minimal case. Fits: the verification half of the loop in Figure the figure, and the one tool here a model did not write. Watch: it checks the property you wrote, so the property must encode what you meant rather than what the code does.

• [Ruff / mypy] Static analysis. Catch undefined names, unused results, type mismatches and calls that cannot succeed, before execution. Fits: a zero-cost first filter on generated code --- an invented function or a wrong keyword argument fails here in milliseconds. Watch: they establish that the code is well formed, which is the cheapest and weakest of the checks in this section; passing them says nothing about the estimand.

Static checks and a smoke test are free and catch the least dangerous errors. The check that finds the dangerous ones is a metamorphic test: an invariant the correct answer must satisfy, stated independently of the implementation. The example below applies four checks in increasing order of strength to a plausible generated routine --- a rolling mean meant to summarise the previous $w$ observations, of the kind that becomes a predictor in a time-series model.

import numpy as np

# generated for: "a feature giving the mean of the previous w observations"
def rolling_mean(x, w):
    out = np.empty(len(x))
    for t in range(len(x)):
        out[t] = x[max(0, t - w + 1):t + 1].mean()
    return out

rng = np.random.default_rng(0)
n, w = 400, 5
x = rng.normal(size=n)
f = rolling_mean(x, w)

print(f"smoke        runs, shape ok {f.shape == x.shape}, "
      f"all finite {bool(np.isfinite(f).all())}")

hand = rolling_mean(np.array([1.0, 2.0, 3.0, 4.0]), 2)[3]
print(f"known answer f[3] on [1,2,3,4] with w=2 is {hand:.1f}, "
      f"spec says mean(2,3) = 2.5")

x2 = x.copy()
x2[200] += 100.0                      # perturb the present only
f2 = rolling_mean(x2, w)
print(f"metamorphic  f[200] moved by {abs(f2[200] - f[200]):.2f} when x[200] "
      f"moved; a past-only feature must move by 0.00")

print(f"leakage      corr(f_t, x_t) = {np.corrcoef(f, x)[0, 1]:.3f}; "
      f"the correct feature gives "
      f"{np.corrcoef(np.roll(f, 1)[1:], x[1:])[0, 1]:.3f}")

smoke        runs, shape ok True, all finite True
known answer f[3] on [1,2,3,4] with w=2 is 3.5, spec says mean(2,3) = 2.5
metamorphic  f[200] moved by 20.00 when x[200] moved; a past-only feature must move by 0.00
leakage      corr(f_t, x_t) = 0.451; the correct feature gives 0.003

The routine includes the current observation in the window --- “previous” resolved as inclusive rather than exclusive, a genuine ambiguity in the prompt resolved silently and invisibly in the returned code. The smoke test passes. The known-answer check is the first to fail, and the metamorphic check localises why: a feature that is supposed to depend only on the past moves when the present moves. The last line prices the bug in the currency the analyst cares about, a correlation of 0.45 between the feature and the contemporaneous outcome where the correct feature gives 0.003. Fitted on this, a model would have reported skill it does not have --- and every diagnostic downstream would have looked clean, because nothing about a leaking predictor shows in a residual plot. This is the failure the section warned about, and it took a stated invariant rather than a careful read to find it.

1.6Exercises¶

1. Write a one-line request and a specification-style prompt for the same estimator. List every modelling decision the first prompt leaves to the model.

2. Explain why a test suite generated from the same prompt as the code under test provides weaker evidence than a hand-written check against a closed-form answer.

3. (Compute) Ask a language model for a function computing the standard error of a sample median. Verify it by simulation against the bootstrap; report whether it agrees and, if not, what it computed instead.

4. (Compute) Generate the same routine three times from an identical prompt at a nonzero temperature. Diff the implementations and check whether they agree numerically on a simulated dataset.

5. (Compute) For a generated feature-engineering routine, write down one metamorphic invariant the correct output must satisfy --- a perturbation of the input that must leave a given output element unchanged --- and test it. Report whether the invariant, a smoke test, or a known-answer check found the bug first.

6. (Compute) Take a generated data-cleaning script and construct an input on which it fails silently --- runs to completion and returns a wrong answer. What check would have caught it?

7. Argue for or against the claim that a prompt should be archived alongside the generated code in a reproducible analysis, given that regeneration is stochastic.