fact-check-prose — verify claims, reasoning, and computed values in prose

Prose review usually stops at style. This skill goes further: it checks whether the prose is actually true and its logic actually holds — per the standing rule in Morrison-Lab/ai-config’s shared/writing/fact-check-prose.md, which is the canonical statement of this policy (edit that file, not this skill’s procedure, if the policy itself changes).

When this fires

  • “fact-check this”, “check the math”, “verify this reasoning”, “check this proof”, “is this accurate”, “does this claim check out”
  • As part of any PR/MR review where the diff touches narrative prose — docs, READMEs, lecture notes, textbook chapters, commit/PR descriptions making a substantive claim. Run it alongside (not instead of) style review (use-preferred-style, find-ai-tells) and terminology review (challenge-ambiguous-terminology, folded into ard/ardi).

Procedure

1. Identify the target and extract checkable units

A PR/MR diff (gh pr diff <N> / glab mr diff <N>), a file, or pasted text. Read the changed prose and pull out three kinds of units to check:

  • Factual claims — statements that cite a value, a behavior, a result, an external fact (“X was introduced in version Y”, “the estimator is unbiased under Z”).
  • Undefended claims — a factual claim with no internal reasoning and no citation backing it, even if it turns out to be accurate. Flag it separately from an accuracy check (below) — a claim can pass accuracy and still fail this one.
  • Reasoning chains — an argument the document makes, not just its individual facts. This includes formal mathematical reasoning (derivations, proofs, algebraic manipulations — check every step follows from the last, check dimensional/unit consistency, check edge cases) and informal reasoning (X therefore Y, a causal claim, a design justification).
  • Computed/rendered values — a number, table entry, or figure the prose describes as the output of code, a model fit, or a render.
  • Citations to external works — identify all sources cited that will need to be checked.
  • Config list and allowlist criteria — when prose describes why an entry belongs on an allowlist or exclusion list, or what its entries have in common, verify the claim against the defining comment or documentation in code.

2. Pre-flight Readiness Gate (Hard Stop)

Before evaluating any claim against an external source, check whether the repository’s local source corpus (sources/ or equivalent) actually contains the needed sources: (Note: If the target document contains zero citations to external works, skip this gate and proceed to Step 3).

  1. Are required sources missing from the local corpus?
  2. Is the required source present but unconverted? For example, a .pdf, .docx, or .epub file without a corresponding Markdown conversion.

If either of these conditions hold, refuse to run. - Do NOT silently narrow scope. - Do NOT partially check around the gap. - Do NOT fall back to memory or the open web for the missing sources.

Return Pre-flight: NOT READY instead of a report, and print remediation instructions. For example, “Populate the sources/ directory with Markdown conversions of the cited works so they can be verified locally.”

Only proceed to Step 3 if the gate passes.

3. Check factual claims

For each factual claim, check it against domain knowledge first. Where it’s checkable against an external source — a package’s own docs, a spec, a paper, a dataset’s documentation — fetch that source from the local sources/ corpus. Do not fall back to the open web (WebFetch) if the local source is missing. If you discover during verification that a local source is incomplete (e.g. missing a chapter), abort and return Pre-flight: NOT READY. Confirm or refute the claim against the local source, not against a secondhand summary. Don’t wave a plausible-sounding claim through unchecked — see challenge-ambiguous-terminology for the sibling rule on ambiguous phrasing specifically.

Separately from accuracy, check that the claim is defended: does the surrounding text give reasoning for it, or does it carry a citation? A claim can be accurate and still undefended — flag both kinds of gap, not just outright inaccuracy.

4. Verify document-internal reasoning

Walk each reasoning chain step by step:

  • Formal math — re-derive or spot-check each step; confirm dimensions and units are consistent at every operation; check boundary/edge cases the claim implies; confirm a cited theorem’s hypotheses actually hold in this application (e.g. don’t apply a result that requires independence to correlated data without noting it).
  • Informal arguments — does the stated reasoning actually support the conclusion drawn, or is there a gap (an unstated assumption doing the real work, a non sequitur, an overgeneralization from one example)?

5. Cross-check computed values and figures against the rendered output

Don’t trust the source prose’s own description of a computed value — verify it against what was actually produced:

  1. Check whether the project has a PR-preview deploy. Look for a sticky preview-link comment on the PR, or check the project’s CLAUDE.md / .github/workflows/ for a preview workflow (e.g. preview.yml + preview-deploy.yml built on rossjrw/pr-preview-action, common in this lab’s Quarto repos — those deploy to a gh-pages branch under pr-preview/pr-<N>/<path>).
  2. If a live preview exists, fetch the rendered page (WebFetch) and read the actual value/figure off it.
  3. If no preview covers the changed page yet, fall back to the project’s render command (e.g. quarto render <chapter.qmd> --to html in an R/ Quarto repo) and check the local render instead of skipping the check.
  4. Compare what’s rendered against what the prose claims. A mismatch is a finding even if the prose “sounds” right — the rendered output is ground truth, not the sentence describing it.

6. Report

For every unit checked:

| Claim/step | Verdict | Basis (source checked) |
|------------|---------|-------------------------|
| "the estimator is unbiased" | Inaccurate | Biased under this design per [source/derivation]; see step 3 below |
| Derivation step 4 → 5 | Confirmed | Re-derived; algebra holds, dimensions consistent |
| "Table 2, row 3 = 4.7" | Inaccurate | Rendered preview (pr-preview/pr-N/chapter.html) shows 4.9 |
| "adoption has grown steadily" | Undefended | No citation or reasoning given; claim itself checks out but needs a source or an explicit basis |

Then, separately, list additional citations/references worth adding — proactively, anywhere a pointer to a source would help a reader, not only where a claim is currently flagged as uncited.

7. Offer to apply corrections

On request, fix the flagged inaccuracies and reasoning gaps in place with Edit, preserving the author’s intent and voice — same posture as find-ai-tells’s apply step. Don’t silently rewrite without being asked; report first.

Relationship to other skills

  • challenge-ambiguous-terminology — catches phrasing whose meaning is unresolved; this skill catches claims and reasoning that are resolved but wrong. Run both during review.
  • use-preferred-style / find-ai-tells — style and AI-tell scans; this skill is the accuracy/reasoning counterpart, not a replacement.
  • ard / ardi — when a reviewer (human or bot) flags a factual or reasoning error, ard handles the response disposition; this skill is how you (as reviewer) find that class of error in the first place.
  • prose-fact-checker agent — the read-only, parallelizable version of step 2–4 above, for fanning claim-checking out across a Workflow when a document has many independent claims to verify at once.
  • check-info-quality (ciq) — this skill verdicts a claim Accurate / Inaccurate / Undefended; ciq’s check C catches a claim that’s accurate by this skill’s own standard yet still misleads (missing context, cherry-picking, a citation that only weakly supports it). ciq also covers two failure modes this skill doesn’t: stale claims (check A) and irrelevant content (check B). Run both.

Anti-patterns

  • Don’t accept a claim because it sounds authoritative — check it.
  • Don’t skip a derivation step because “it’s probably fine” — re-derive it or flag it as unverified, not silently pass it.
  • Don’t trust the prose’s own description of a computed value when a rendered artifact (preview or local render) is available to check it against directly.
  • Don’t limit citation suggestions to claims already flagged as uncited — suggest them anywhere they’d help a reader, per the fragment’s policy.
Back to top