Screening and Systematic Review

Systematic review is the part of research synthesis with an explicit protocol, a reporting standard and a measurable error structure, which makes it an unusually tractable testbed for AI assistance: you can tell whether it worked. This section covers the platforms researchers actually screen in, the shape of a PRISMA-compatible workflow when a model is in the loop Page et al., 2021, and the agreement statistics that make dual review more than a ritual. The point running through all of it is that a machine screener is a diagnostic test applied to a low-prevalence population, and everything a statistician knows about that setting transfers directly. The literature on text mining for study identification is older and better developed than the current wave of tools suggests O'Mara-Eves et al., 2015Marshall & Wallace, 2019, and it is worth reading before adopting anything.

1Screening platforms¶

1.1Rayyan¶

• [Rayyan] Collaborative screening platform. Hosts title/abstract screening for a team, with blinding between reviewers and a relevance ranking learned from the decisions made so far Ouzzani et al., 2016. Fits: title/abstract screening once deduplication is done. Watch: the learned ranking is trained on your own early decisions, so an early systematic mistake is amplified rather than corrected. Blinding must be switched on deliberately; unblinded dual screening produces agreement statistics that mean nothing.

1.2Covidence¶

• [Covidence] End-to-end review workflow. Carries a review from import through dual screening, full-text assessment, extraction and risk-of-bias, emitting the counts a PRISMA flow diagram needs. Fits: the whole pipeline when the team wants one audit trail. Watch: the exported flow numbers are only as good as the deduplication that preceded them. Reconcile the stage counts arithmetically before they go into the diagram.

1.3DistillerSR¶

• [DistillerSR] Regulated-workflow review platform. Adds configurable multi-stage forms, quality-control sampling and an audit log aimed at regulatory and guideline submissions. Fits: reviews that must be defensible to an external auditor. Watch: configurability is the risk. A workflow assembled ad hoc during screening is a protocol amendment, and it must be logged as one.

1.4ASReview¶

• [ASReview] Open-source active-learning screener. Ranks the unscreened pool after every human decision, so relevant records surface early; being open source, the model and the stopping rule are inspectable Schoot et al., 2021. Fits: large title/abstract pools where full screening is infeasible and the ranking has to be auditable. Watch: it never tells you what remains in the unscreened tail. The recall you claim comes from your stopping rule, not from the software.

The distinction that matters across these four is not feature count but mechanism: whether the tool merely organises human decisions, whether it learns a ranking from them, and whether that learned component is inspectable. A hosted platform with a proprietary ranker and an open-source active learner can produce identical screening decisions and are not equally reportable.

2Dual review as a measurement design¶

Two independent screeners with adjudication is the standard defence against single-rater error, and it produces the data for an agreement statistic as a by-product. Build the two-by-two first and read several statistics off it, not just $\kappa$.

# Dual screening: build the 2x2, then read more than kappa off it
import random
random.seed(11)
n = 400
truth = ["I" if random.random() < 0.09 else "E" for _ in range(n)]

def screen(truth, p_miss, p_fp, seed):
    rng, out = random.Random(seed), []
    for t in truth:
        if t == "I":
            out.append("E" if rng.random() < p_miss else "I")
        else:
            out.append("I" if rng.random() < p_fp else "E")
    return out

A = screen(truth, p_miss=0.10, p_fp=0.03, seed=1)   # cautious
B = screen(truth, p_miss=0.25, p_fp=0.01, seed=2)   # stricter

n11 = sum(a == b == "I" for a, b in zip(A, B))
n00 = sum(a == b == "E" for a, b in zip(A, B))
n10 = sum(a == "I" and b == "E" for a, b in zip(A, B))
n01 = sum(a == "E" and b == "I" for a, b in zip(A, B))

po = (n11 + n00) / n
pA, pB = (n11 + n10) / n, (n11 + n01) / n
pe = pA * pB + (1 - pA) * (1 - pB)
print("            B:incl  B:excl")
print(f"A:include {n11:>8} {n10:>7}")
print(f"A:exclude {n01:>8} {n00:>7}")
print(f"n={n}  include rate A={pA:.3f} B={pB:.3f}")
print(f"agreement {po:.3f}  expected {pe:.3f}  "
      f"kappa {(po - pe) / (1 - pe):.3f}")
print(f"positive agreement {2*n11/(2*n11+n10+n01):.3f}  "
      f"negative {2*n00/(2*n00+n10+n01):.3f}")
print(f"prevalence index {abs(n11-n00)/n:.3f}  "
      f"bias index {abs(n10-n01)/n:.3f}  PABAK {2*po-1:.3f}")

union = sum(a == "I" or b == "I" for a, b in zip(A, B))
both = sum(t == "I" and a == "E" and b == "E"
           for t, a, b in zip(truth, A, B))
solo = sum(t == "I" and a == "E" for t, a in zip(truth, A))
print(f"true includes {truth.count('I')}; union flags {union}; "
      f"both missed {both}")
print(f"reviewer A alone would have missed {solo}")

            B:incl  B:excl
A:include       25      25
A:exclude        6     344
n=400  include rate A=0.125 B=0.077
agreement 0.922  expected 0.817  kappa 0.577
positive agreement 0.617  negative 0.957
prevalence index 0.797  bias index 0.048  PABAK 0.845
true includes 37; union flags 56; both missed 0
reviewer A alone would have missed 4

Three things in that output deserve attention. First, the union of two reviewers misses nothing here while reviewer A alone would have dropped four true includes -- that is what dual screening buys. Second, $\kappa=0.577$ looks mediocre while observed agreement is 0.922; the gap is almost entirely the prevalence index, which is high because most records are excludable. Third, positive agreement (0.617) and negative agreement (0.957) are wildly different, and reporting a single $\kappa$ hides that the reviewers agree easily on exclusions and disagree substantially on inclusions -- which is the disagreement that changes the review.

The prevalence problem is worth isolating, because it is the single most common misreading of a reported $\kappa$ in a screening paper Byrt et al., 1993.

# kappa falls with prevalence even when agreement is held fixed
def kappa(n11, n10, n01, n00):
    n = n11 + n10 + n01 + n00
    po = (n11 + n00) / n
    pA, pB = (n11 + n10) / n, (n11 + n01) / n
    pe = pA * pB + (1 - pA) * (1 - pB)
    return po, pe, (po - pe) / (1 - pe)

print(f"{'incl rate':>9} {'n11':>4} {'n00':>5} "
      f"{'p_o':>6} {'p_e':>6} {'kappa':>7}")
for n11, n00 in [(90, 810), (45, 855), (18, 882), (9, 891), (4, 896)]:
    n10 = n01 = 50                    # 100 disagreements every time
    tot = n11 + n00 + n10 + n01
    po, pe, k = kappa(n11, n10, n01, n00)
    print(f"{(n11+n10)/tot:>9.3f} {n11:>4} {n00:>5} "
          f"{po:>6.3f} {pe:>6.3f} {k:>7.3f}")

incl rate  n11   n00    p_o    p_e   kappa
    0.140   90   810  0.900  0.759   0.585
    0.095   45   855  0.900  0.828   0.418
    0.068   18   882  0.900  0.873   0.211
    0.059    9   891  0.900  0.889   0.099
    0.054    4   896  0.900  0.898   0.021

Observed agreement is held at exactly 0.900 in every row and the number of disagreements is held at 100; only the marginal distribution moves. $\kappa$ falls from 0.585 to 0.021. A reviewer pair that would be called “moderate” in a broad review is called “slight” in a narrow one for no reason connected to their performance. Report the two-by-two, the prevalence index and the bias index alongside $\kappa$ Cohen, 1960Byrt et al., 1993, and interpret the descriptive bands Landis & Koch, 1977 with that in mind.

A machine second reader is tempting here, and the design question is whether its errors are independent of the human’s. Two instruments that both key on title words fail on the same records, and correlated errors defeat the whole point of dual review. The defensible deployment is machine-plus-human with the machine never permitted to exclude a record on its own; the Cochrane evaluation of a randomised-trial classifier adopted exactly that safety-first arrangement Thomas et al., 2021.

3Where the classifier sits, and at what threshold¶

A screening classifier is a diagnostic test applied to a population with two percent prevalence. Judge it the way you would judge any such test.

# A screening classifier is a diagnostic test at 2% prevalence
n, prev = 2000, 0.02
n_pos = int(n * prev)
n_neg = n - n_pos
print(f"corpus {n}, includable {n_pos} ({prev:.0%})")
print(f"{'thresh':>7} {'sens':>6} {'spec':>5} {'flagged':>8} "
      f"{'PPV':>6} {'FN':>4} {'read':>6}")
for thr, sens, spec in [(0.10, 1.000, 0.55), (0.25, 0.975, 0.78),
                        (0.50, 0.900, 0.93), (0.75, 0.750, 0.98)]:
    tp = sens * n_pos
    fp = (1 - spec) * n_neg
    flagged = tp + fp
    print(f"{thr:>7.2f} {sens:>6.3f} {spec:>5.2f} {flagged:>8.0f} "
          f"{tp/flagged:>6.3f} {n_pos-tp:>4.0f} {flagged/n:>5.1%}")

corpus 2000, includable 40 (2%)
 thresh   sens  spec  flagged    PPV   FN   read
   0.10  1.000  0.55      922  0.043    0 46.1%
   0.25  0.975  0.78      470  0.083    1 23.5%
   0.50  0.900  0.93      173  0.208    4  8.7%
   0.75  0.750  0.98       69  0.434   10  3.5%

The bottom row is ninety-eight percent specific and would be reported as a success by an accuracy-minded evaluation. It also loses ten of the forty includable studies, which would sink the review. The top row misses nothing and still halves the reading. This is why review-automation work reports recall and workload rather than accuracy, and why any threshold you adopt must be quoted with the recall it was chosen to achieve O'Mara-Eves et al., 2015.

Active learning changes the accounting: instead of a fixed threshold you screen down a continuously re-estimated ranking and stop somewhere. The stopping rule is then the entire inferential content of the procedure.

# Active-learning screening: what does each recall target cost?
n_records, n_rel = 2000, 40
# rank position of each relevant record in the classifier's ordering
pos = [1,2,3,4,5,6,8,9,11,13,14,17,19,22,25,28,31,35,40,46,
       52,60,69,80,92,106,122,141,163,188,217,251,290,335,387,
       447,517,700,1100,1850]
assert len(pos) == n_rel

print(f"{'recall':>7} {'screened':>9} {'% corpus':>9} {'WSS':>7}")
for target in (0.50, 0.80, 0.90, 0.95, 1.00):
    need = int(-(-target * n_rel // 1))     # ceiling
    stop = pos[need - 1]
    frac = stop / n_records
    wss = (1 - frac) - (1 - target)         # work saved over sampling
    print(f"{target:>7.2f} {stop:>9} {frac:>8.1%} {wss:>7.3f}")

 recall  screened  % corpus     WSS
   0.50        46     2.3%   0.477
   0.80       251    12.6%   0.675
   0.90       447    22.4%   0.676
   0.95       700    35.0%   0.600
   1.00      1850    92.5%   0.075

Half the relevant records arrive in the first two percent of the corpus, and the last two cost more screening than the first thirty-eight. That convexity is the whole story of active-learning screening: workload savings are real and they evaporate at the recall targets a review actually needs. Note also that work saved over sampling peaks in the middle and is meaningless quoted without its recall target.

Since you cannot see the unscreened tail, a stopping rule has to be backed by a validation draw: sample from what the machine excluded, screen those by hand, and report an interval on the missed-include rate.

# How big must a validation draw be to bound the missed-include rate?
from math import sqrt, ceil

def wilson(x, k, z=1.96):
    p, d = x / k, 1 + z * z / k
    c = (p + z * z / (2 * k)) / d
    h = z * sqrt(p * (1 - p) / k + z * z / (4 * k * k)) / d
    return max(0.0, c - h), min(1.0, c + h)

print("re-screen k machine-excluded records; x are true includes")
print(f"{'k':>5} {'x':>3} {'p-hat':>7} {'95% Wilson':>20} {'width':>8}")
for k, x in [(5,0), (10,0), (40,0), (100,0), (100,1), (300,2), (1000,3)]:
    lo, hi = wilson(x, k)
    print(f"{k:>5} {x:>3} {x/k:>7.3f}  [{lo:>6.4f}, {hi:>6.4f}] "
          f"{hi - lo:>8.4f}")

print("rule of three: zero events in k gives an upper bound near 3/k")
for target in (0.05, 0.02, 0.01, 0.005):
    print(f"  to claim a rate below {target:<5} on a clean sample, "
          f"k >= {ceil(3 / target)}")

re-screen k machine-excluded records; x are true includes
    k   x   p-hat           95% Wilson    width
    5   0   0.000  [0.0000, 0.4345]   0.4345
   10   0   0.000  [0.0000, 0.2775]   0.2775
   40   0   0.000  [0.0000, 0.0876]   0.0876
  100   0   0.000  [0.0000, 0.0370]   0.0370
  100   1   0.010  [0.0018, 0.0545]   0.0527
  300   2   0.007  [0.0018, 0.0240]   0.0221
 1000   3   0.003  [0.0010, 0.0088]   0.0078
rule of three: zero events in k gives an upper bound near 3/k
  to claim a rate below 0.05  on a clean sample, k >= 60
  to claim a rate below 0.02  on a clean sample, k >= 150
  to claim a rate below 0.01  on a clean sample, k >= 300
  to claim a rate below 0.005 on a clean sample, k >= 600

This is the arithmetic that ends the “I spot-checked a few” era. Ten clean records bound the missed-include rate only below twenty-eight percent, which is no bound at all for a review. Three hundred clean records buy a bound near one percent. The rule of three is the version to memorise: with zero events in $k$ draws, the upper ninety-five percent bound is about $3/k$ Wilson, 1927. Decide the bound you need before screening starts, because $k$ is then determined and has to be budgeted.

4Extraction, and why cell accuracy flatters¶

Once studies are included, fields have to be pulled out of them. Language models do this well enough to be useful and badly enough to need checking, and the checking has a structure.

# Extraction accuracy: three fields from four papers, all hand-checked
truth = {
 ("P01", "n_rand"): 240, ("P01", "months"): 12, ("P01", "effect"): 0.82,
 ("P02", "n_rand"): 118, ("P02", "months"): 6,  ("P02", "effect"): 1.14,
 ("P03", "n_rand"): 96,  ("P03", "months"): 24, ("P03", "effect"): 0.55,
 ("P04", "n_rand"): 512, ("P04", "months"): 18, ("P04", "effect"): 0.91,
}
got = dict(truth)
got[("P02", "n_rand")] = 128    # digits transposed
got[("P03", "months")] = 2      # units: 2 years recorded as 2
got[("P04", "effect")] = None   # field not found

papers = sorted({p for p, f in truth})
fields = ["n_rand", "months", "effect"]
print(f"{'field':>8} {'checked':>8} {'exact':>6} {'wrong':>6} "
      f"{'missing':>8}")
for f in fields:
    cells = [(p, truth[(p, f)]) for p in papers]
    miss = sum(got[(p, f)] is None for p, v in cells)
    hit = sum(got[(p, f)] == v for p, v in cells)
    print(f"{f:>8} {len(cells):>8} {hit:>6} "
          f"{len(cells) - hit - miss:>6} {miss:>8}")

errs = [(p, f) for (p, f), v in truth.items() if got[(p, f)] != v]
for p, f in sorted(errs):
    print(f"  {p} {f}: truth {truth[(p,f)]!r}, got {got[(p,f)]!r}")
print(f"cell-level accuracy  {1 - len(errs)/len(truth):.3f}")
print(f"paper-level accuracy "
      f"{1 - len({p for p, f in errs})/len(papers):.3f}")

   field  checked  exact  wrong  missing
  n_rand        4      3      1        0
  months        4      3      1        0
  effect        4      3      0        1
  P02 n_rand: truth 118, got 128
  P03 months: truth 24, got 2
  P04 effect: truth 0.91, got None
cell-level accuracy  0.750
paper-level accuracy 0.250

Cell-level accuracy is 0.750 and paper-level accuracy is 0.250: three of four studies carry at least one bad cell. Errors cluster within a paper -- a study reported in an unusual layout defeats several fields at once -- so the independence that cell-level accuracy implicitly assumes does not hold. Report the paper-level figure, since a meta-analysis consumes rows, not cells. Note the character of the three errors: a transposition, a units error, and a silent None. Only the last is self-announcing, which is an argument for prompting extraction to emit an explicit “not reported” token and for range-checking every numeric field against what is plausible.

5PRISMA compatibility is an arithmetic property¶

Nothing in the PRISMA 2020 statement forbids machine assistance Page et al., 2021. What it requires is that the process be described and that the counts reconcile. That makes the flow diagram a testable object, and it should be generated from the record log rather than typed by hand.

# Rebuild the PRISMA flow from the log and check the arithmetic closes
log = {
    "db_medline": 1420, "db_embase": 1180, "db_scopus": 640,
    "reg_trials": 55, "other_citation_chasing": 38,
    "dupes_removed": 1105,
    "excl_screen": 1798,
    "excl_not_retrieved": 12,
    "fulltext_assessed": 418,
    "excl_fulltext": {"wrong population": 141, "wrong comparator": 96,
                      "no outcome of interest": 88,
                      "conference abstract": 51, "duplicate cohort": 12},
    "included_studies": 30,
}
found = sum(v for k, v in log.items()
            if k.startswith(("db_", "reg_", "other_")))
after_dupes = found - log["dupes_removed"]
sought = after_dupes - log["excl_screen"]
assessed = sought - log["excl_not_retrieved"]
excl_ft = sum(log["excl_fulltext"].values())
included = assessed - excl_ft

print(f"records identified           {found:>6}")
print(f"after duplicates removed     {after_dupes:>6}")
print(f"reports sought               {sought:>6}")
print(f"assessed for eligibility     {assessed:>6}  "
      f"(log: {log['fulltext_assessed']})")
print(f"excluded at full text        {excl_ft:>6}")
print(f"studies included             {included:>6}  "
      f"(log: {log['included_studies']})")
checks = {
    "assessed matches log": assessed == log["fulltext_assessed"],
    "included matches log": included == log["included_studies"],
    "no stage is negative": min(after_dupes, sought, assessed,
                                included) >= 0,
}
for name, ok in checks.items():
    print(f"[{'ok  ' if ok else 'FAIL'}] {name}")

records identified             3333
after duplicates removed       2228
reports sought                  430
assessed for eligibility        418  (log: 418)
excluded at full text           388
studies included                 30  (log: 30)
[ok  ] assessed matches log
[ok  ] included matches log
[ok  ] no stage is negative

This is a spec-drift check applied to a review: the declared shape of the output -- the flow diagram’s stages -- is written down first and the pipeline is asserted against it. That is the same discipline as writing the manuscript’s table shells before the analysis exists, developed in Section that section. In a review it is close to free, because the stage counts are already required for reporting.

6What to record in a paper¶

The methods section must state, for every automated component: the tool and its version, and the date the screening ran; the model or classifier used, and whether it was pre-trained, fine-tuned or learned from the review’s own decisions; the exact prompt or configuration, quoted verbatim in the supplement; the operating threshold or stopping rule and the recall target it was chosen to achieve; whether the machine could exclude a record alone or only rank it; the dual-screening design, including whether reviewers were blinded to each other, with the two-by-two table, $\kappa$, the prevalence and bias indices, and how disagreements were adjudicated; and the size, sampling scheme and result of every hand-validation draw, with an interval. If a language model extracted data, report per-field and per-paper accuracy against a validated subsample. The general principle: anything a human reviewer would have to declare, a machine reviewer must declare too.

7Exercises¶

1. Re-run the dual-screening simulation across include rates from 0.02 to 0.30 while holding both reviewers’ sensitivity and specificity fixed. Plot $\kappa$, PABAK and positive agreement against prevalence. Write the two-sentence caption you would put under that figure in a methods paper.

2. Using the operating-point table, suppose your review can afford to read four hundred of the two thousand records. Find the highest recall attainable within that budget under the four operating points given, then state what additional information you would need to decide whether to spend the budget on machine-ranked screening or on a second human reviewer.

3. Take the active-learning example and implement two stopping rules: stop after 50 consecutive irrelevant records, and stop when the estimated remaining relevant count falls below one. Report the recall each achieves on the given ranking, then explain why neither rule can be validated without the sampling draw from the previous example.

4. Modify the extraction example so that errors are induced independently per cell at the same overall cell-level rate. Compare the resulting paper-level accuracy with the clustered version. Quantify how badly an independence assumption would mislead someone budgeting verification effort for a hundred-study review.

5. Corrupt the PRISMA log by changing one stage count so the arithmetic no longer closes, and extend the checking code to localise which stage is inconsistent rather than merely reporting a failure. Then write the three assertions you would add to catch the three most common real errors: double-counted duplicates, records excluded at two stages, and full texts never retrieved.