VALIDATION · APRIL 2026

sourcebook check — Validation Results

30 real diffs from open-source repos. Rules-based Layer A and AI-powered Layer B evaluated separately. Methodology, results, and known limitations.

Overview

sourcebook check analyzes a git diff for completeness using structural signals: git co-change coupling, test file mappings, import graph siblings, and hub file blast radius.

Layer A: Rules-based, no LLM, <1 second
Layer B (--ai): Sends diff + dependency context to Claude Sonnet, ~$0.012/run

This page documents the validation methodology and results for both layers.

Methodology

30 real diffs from open-source repositories
Each diff manually classified: complete (all companion files changed) or incomplete (missing companions)
sourcebook check run on each diff
Measured: true positives (correctly flagged incomplete), true negatives (correctly passed clean diffs), false positives, false negatives
Layer A and Layer B evaluated separately

Layer A results (rules-based)

Metric	Result
Completeness gate accuracy	100% (30/30)
False positive rate (clean diffs)	0%
Test file detection	73%
Sibling file detection	71%
Hard semantic detection	33%

Completeness gate: If the diff is actually incomplete, Layer A always detects it
Zero false positives: Clean diffs never flagged
Test file detection limited by non-standard naming and locations
Sibling detection covers import graph and same-directory patterns
Semantic gaps (field renames needing migrations, etc.) are structurally invisible to Layer A

Layer B results (AI-powered)

Catches semantic relationships Layer A misses
“This config change probably needs a migration”
“The validation logic assumes the old field name”
Cost: ~$0.012/run (Claude Sonnet, ~800 input tokens, ~200 output tokens)
Combined Layer A + B catch rate significantly higher on semantic cases

What it catches

Four signal types power the completeness analysis:

Co-change coupling — Files that historically change together in git commits. “auth.ts changed → session.ts usually changes too.”
Test file coverage — Source file modified, corresponding test file not. Detected via naming convention and co-change history.
Import graph siblings — Files that import or are imported by the changed file. Structural dependencies.
Hub file blast radius — Modified file has 50+ dependents. High-risk change.

Known limitations

Semantic relationships with no structural signal (cross-subsystem, cross-extension dependencies)
Test files in non-standard locations not covered by naming conventions
Co-change data limited by git history depth (shallow clones reduce signal)
Layer B requires ANTHROPIC_API_KEY (user provides their own)

Try it

$ npx sourcebook check # free, no API key needed

$ npx sourcebook check --ai # deeper semantic analysis, requires ANTHROPIC_API_KEY

$ npx sourcebook init # sets up Claude Code hooks for automatic checking

$ npx sourcebook init

VIEW_ON_GITHUB ALL_RESEARCH

MORE_FROM_SOURCEBOOK

Benchmark methodology and full results arrow_forward sourcebook vs Repomix arrow_forward Blog: We benchmarked AI context files on real GitHub issues arrow_forward