Structured Data Extraction

import json

SCHEMA = {"study_id": str, "n_participants": int, "effect": float, "outcome": str}

candidates = [
    '{"study_id": "S1", "n_participants": 240, "effect": 0.31, "outcome": "mortality"}',
    '{"study_id": "S2", "n_participants": "two hundred", "effect": 0.4, "outcome": "mortality"}',
    '{"study_id": "S3", "n_participants": 88, "effect": 0.12}',
    '```json\n{"study_id": "S4", "n_participants": 51, "effect": 0.9, "outcome": "LOS"}\n```',
]

def parse(raw):
    s = raw.strip()
    if s.startswith("```"):                       # models love fencing their JSON
        s = "\n".join(l for l in s.split("\n") if not l.startswith("```"))
    try:
        obj = json.loads(s)
    except json.JSONDecodeError as e:
        return None, f"unparseable: {e.msg}"
    missing = [k for k in SCHEMA if k not in obj]
    if missing:
        return None, f"missing fields: {missing}"
    for k, typ in SCHEMA.items():
        if typ is float and isinstance(obj[k], int):
            obj[k] = float(obj[k])
        elif not isinstance(obj[k], typ):
            return None, f"field {k!r} is {type(obj[k]).__name__}, expected {typ.__name__}"
    return obj, "ok"

for raw in candidates:
    obj, msg = parse(raw)
    tag = obj["study_id"] if obj else "--"
    print(f"{tag:<4} {msg}")

S1   ok
--   field 'n_participants' is str, expected int
--   missing fields: ['outcome']
S4   ok

0.1Data Extraction¶

A great deal of the world’s scientific information is locked in prose: clinical notes, pathology reports, published tables, protocol documents, free-text survey responses, and the methods sections of ten thousand papers you would like to meta-analyse. Language models are startlingly good at turning that prose into a rectangular data frame, and this is arguably their highest-value application in research --- it converts a task that used to cost months of trained abstractor time into an afternoon of compute. It is also the application where a statistician’s instincts pay off fastest, because the output of an extraction pipeline is not data. It is a measurement of data, with a sensitivity, a specificity, and an error process that will propagate into every estimate computed downstream. The central message of this section is that this problem has been solved before, under different names: measurement error, misclassification, and two-phase sampling with a validation subsample. If you extract a variable with $90\%$ accuracy and then regress an outcome on it as though it were observed, you have not produced a slightly noisy estimate; you have produced a systematically attenuated one, and you can say by how much.

0.1.1The pipeline, and where the statistics enter¶

An extraction pipeline has five stages: acquire documents, segment them into passages, prompt a model for a structured record, validate the record against a schema, and then analyse. Figure the figure draws it with the validation subsample entering where it belongs --- as a designed second phase, not as a spot check bolted on at the end. The single most consequential design decision is to reserve labelling effort for a random validation sample rather than spending all of it on prompt iteration, because the validation sample is what converts an unquantified pipeline into a measurable one. The acquisition stage is usually a document-conversion problem before it is a language problem (GROBID turns scholarly PDFs into structured TEI with the reference list and section boundaries already delimited, and general layout converters such as Docling or Marker do the same job for reports and forms), and where the corpus must itself be assembled by an agent that searches and synthesises, the verification machinery of Section that section applies to the corpus before any extraction runs on it.

0.1.2Attenuation: the number every collaborator needs to see¶

Take the cleanest case. The true binary exposure $X \in \{0,1\}$ has prevalence $\pi$, the extracted version $X^\ast$ has sensitivity $\mathrm{se} = \Pr(X^\ast=1 \mid X=1)$ and specificity $\mathrm{sp} = \Pr(X^\ast=0\mid X=0)$, and errors are non-differential, meaning $X^\ast \perp Y \mid X$. Write $\pi^\ast = \pi\, \mathrm{se} + (1-\pi)(1-\mathrm{sp})$ for the prevalence of the extracted variable. For the linear model $\EX[Y\mid X] = \beta_0 + \beta_1 X$, regressing $Y$ on $X^\ast$ instead estimates

so the naive slope is the true slope multiplied by an attenuation factor $\lambda$ that depends only on $\pi$, sensitivity and specificity. The consequences are worth stating baldly. At $\pi = 0.4$ and $\mathrm{se}=\mathrm{sp}=0.90$ --- an extraction accuracy most people would call excellent --- the estimated effect is about $79\%$ of the truth, and no amount of additional data repairs it: the bias is in the estimand, and more documents only shrink the standard error around the wrong value.

0.1.3Validation is a design problem, not a QA step¶

The temptation is to label a convenience batch of records, quote an accuracy, and proceed. Two design refinements pay for themselves. First, stratify the validation sample: oversample records the model flagged positive, since sensitivity is estimated only from true positives and those are scarce when prevalence is low, then reweight. Second, measure the human ceiling by double-labelling a subset, because an extraction pipeline cannot be shown to exceed an accuracy that your gold standard itself does not attain. (Annotation platforms such as Label Studio or doccano exist to make double-labelling, adjudication and inter-rater statistics a routine part of the design rather than something reconstructed from a spreadsheet afterwards.) A useful supplementary tool is conformal prediction, which converts any confidence score into extraction sets with finite-sample coverage under exchangeability Springer-Verlag, 2005Angelopoulos & Bates, 2021 --- for a categorical field, the model returns a set of candidate values guaranteed to contain the truth $90\%$ of the time, and the singleton sets can be accepted automatically while the rest route to a human. :::{note} Author note Note the exchangeability caveat sharply: documents from a new hospital, a new year, or a new template are not exchangeable with the calibration set, so coverage degrades exactly when you most want it. Cross-reference the uncertainty section rather than repeating the conformal machinery. :::

0.1.4Implementation: schema-constrained extraction¶

The practical mechanics are simple enough to show in a dozen lines. Define the target record as a typed schema, require the model to emit that schema, and let the parser --- not the prose --- decide whether an output is admissible. Constrained decoding removes parse failures entirely; it removes no factual errors at all. (The constraint is enforced in one of two mechanically different ways --- a grammar-constrained decoder such as outlines, or the GBNF grammars built into llama.cpp, masks off-schema tokens so an invalid record is unreachable, whereas a validate-and-retry wrapper such as instructor re-prompts until the parser is satisfied --- and the difference matters to a statistician, because retrying until acceptance discards the hard documents non-randomly and quietly changes the population your data frame describes.)

class ChartRecord(BaseModel): patient_age: Optional[int] = Field(None, ge=0, le=120) smoking_status: Literal[“never”, “former”, “current”, “unknown”] biopsy_performed: bool evidence_span: str # verbatim quote supporting the fields above

1Ask the model for JSON conforming to ChartRecord.model_json_schema(),¶

2then parse. A parse failure is a rejected record, not a silent NaN.¶

record = ChartRecord.model_validate_json(model_output)

2.1Tools in practice¶

The tooling for extraction splits cleanly along the pipeline of Figure the figure, and the split is worth internalising because each stage has its own error process. Document conversion decides what text the model ever sees; constrained decoding decides what shape the output can take; the serving layer decides whether the instrument is stable enough to re-run; and the annotation layer decides whether you can measure any of it. Two habits are worth adopting before any of these are installed. First, measure the cheap baseline: a rule-based or dictionary-based extractor built with spaCy, or its biomedical variants scispaCy and medspaCy, often recovers structured fields such as dates, dosages and identifiers at an accuracy the language model must then be shown to beat, and it is deterministic and free. Second, remember that none of these tools estimates sensitivity or specificity for you; that number comes only from the validation subsample, and it is the number the rest of this section runs on.

• [GROBID] Document structuring. Converts scholarly PDFs into structured TEI XML with the header, sections, tables and parsed reference list delimited as separate elements. Fits: stage one of a meta-analysis or literature-scale extraction, before any model call. Watch: conversion errors are silent and correlate with layout, so older scans and two-column tables lose content that then simply never appears in the data frame.

• [Docling / Marker] Document structuring. General-purpose layout-aware converters that turn PDFs, office documents and scans into markdown or JSON with tables and reading order preserved. Fits: the same stage for non-scholarly sources --- reports, forms, regulatory submissions. Watch: table reconstruction is the weakest part of every such converter, and a merged or transposed table produces plausible numbers rather than an obvious failure.

• [Outlines / llama.cpp grammars] Constrained decoding. Restrict generation to a formal grammar or JSON schema so that off-schema tokens are never sampled. Fits: the extraction call itself, when every record must parse. Watch: a schema-valid record is not a correct record; constraining the output space raises the admission rate and can lower accuracy per admitted record, which is exactly what the example below measures.

• [Pydantic / Instructor] Schema and validation. Declare the target record as a typed model and reject anything that fails validation; instructor adds an automatic re-prompt on failure. Fits: the schema-validation stage, and the definition of the record itself. Watch: log every rejection and every retry, because an unlogged retry loop turns a parse failure into a silently substituted record.

• [vLLM] Local serving. Runs open-weight models over a document corpus with high batch throughput and supports schema-constrained output. Fits: extraction at corpus scale, and any setting where documents cannot leave the institution. Watch: throughput settings such as batch size alter floating-point reduction order, so the instrument is only as stable as the serving configuration you record alongside the weights.

• [Label Studio / doccano] Annotation. Interfaces for human labelling, adjudication and inter-annotator agreement over text spans and categorical fields. Fits: phase two --- the validation subsample and the double-labelled ceiling estimate. Watch: showing annotators the model’s proposed answer speeds labelling and biases it towards agreement, which inflates measured accuracy; label a fraction blind and compare.

The claim in the callouts that constrained decoding trades a parse-failure rate for a wrong-answer rate is easy to state and easy to get wrong, so measure it. The following replays stored outputs from two runs over the same six documents, one free-form and one schema-constrained, and applies the same admission gate to both. Nothing here calls a model; the responses are fixtures so that the arithmetic is reproducible.

SMOKING = (“never”, “former”, “current”, “unknown”)

def admit(raw): “”“Schema gate: parse, check types and levels. None = rejected.”“” try: r = json.loads(raw) except json.JSONDecodeError: return None ok = (set(r) == {“age”, “smoking”, “biopsy”} and isinstance(r[“age”], int) and 0 <= r[“age”] <= 120 and r[“smoking”] in SMOKING and isinstance(r[“biopsy”], bool)) return r if ok else None

def rec(a, s, b): return json.dumps({“age”: a, “smoking”: s, “biopsy”: b})

gold = [(61, “former”, True), (47, “never”, False), (73, “current”, True), (55, “never”, True), (38, “unknown”, False), (29, “never”, False)]

3Two runs over the same six documents, replayed from stored responses.¶

free = [rec(61, “former”, True), rec(47, “never”, False), ‘"json\n{"age": 73, ...}\n"’, # fenced: no parse rec(55, “never”, True), rec(38, “unknown”, False), ‘The note gives no age. {“smoking”: “never”}’] # prose: no parse constrained = [rec(61, “former”, True), rec(47, “never”, False), rec(73, “former”, True), # admissible, wrong rec(55, “never”, True), rec(38, “unknown”, False), rec(29, “never”, False)]

def audit(name, outs): keep = [admit(o) for o in outs] n_ok = sum(k is not None for k in keep) n_right = sum(k is not None and tuple(k.values()) == g for k, g in zip(keep, gold)) print(f"{name:12s} admitted {n_ok}/6 correct {n_right}" f" acc|admitted {n_right / n_ok:.2f}" f" per document {n_right / 6:.2f}")

audit(“free-form”, free) audit(“constrained”, constrained)

Read the last two columns against each other. Accuracy conditional on admission is the number a dashboard will show you, and it moves the wrong way: the free-form run looks perfect because every record it failed to parse was silently dropped from the denominator. Accuracy per document is the estimand your downstream analysis actually needs, because a dropped record is a missing value, not an absence of error --- and it is missing precisely on the documents the model found hardest. The general lesson is the one this section keeps making: report the denominator you started with, not the one that survived.

3.1Exercises¶

1. Derive the equation from $\beta_1^\ast = \mathrm{Cov}(Y, X^\ast)/\mathrm{Var}(X^\ast)$ under non-differential misclassification. Verify $\lambda = 1$ when $\mathrm{se}=\mathrm{sp}=1$ and $\lambda = 0$ when the extracted variable is independent of the truth.

2. With $\pi = 0.2$, compute the attenuation factor for $(\mathrm{se},\mathrm{sp}) = (0.95,0.95)$, $(0.99,0.90)$ and $(0.90,0.99)$. Explain why, at low prevalence, specificity matters far more than sensitivity for the bias in $\hat\beta_1$.

3. Construct an explicit differential-misclassification example in which the naive estimate is biased away from the null, and describe the feature of an LLM reading a full clinical note that makes this scenario realistic rather than contrived.

4. You will label $N_v$ records for validation. Write the expression for the standard error of $\widehat{\mathrm{se}}$ and find the $N_v$ needed for a $\pm 0.05$ margin at $\mathrm{se} = 0.9$. Then explain how stratifying on the model’s own prediction reduces that cost.

5. (Computational) Simulate the setting of Figure the figure: generate $X$, corrupt it to $X^\ast$ at a chosen accuracy, and compare the naive slope, the closed form the equation, and a regression-calibration estimate using a validated subsample. Report bias, Monte Carlo standard deviation, and coverage of the nominal $95\%$ interval for each.

6. (Computational) Build a small extraction pipeline over 200 public abstracts using a schema of three fields. Hand-label a random 50, estimate per-field sensitivity and specificity with Wilson intervals Wilson, 1927, and report how the estimated effect of one extracted field on another changes before and after correction.

7. (Computational) Take the model’s per-field confidence scores, assess their calibration with a reliability diagram, and construct a route-to-human rule that attains $98\%$ precision on the auto-accepted subset. Report the fraction of records requiring human review, and check whether the routed records differ systematically from the rest.