Case Study · Developer Tooling
Forespec
Catches what your AI coder didn't know to ask — the atomic stock hold, the tenant isolation, the webhook signature. 42 checkpoints your kind of app actually requires, surfaced before you build and graded against the code after — so the expensive discoveries surface in week one, not month three.
The 30-second version
- Problem
- AI coding tools build what you ask, not the non-obvious thing your kind of app requires — an atomic stock hold, tenant isolation, webhook authenticity. You find out in month three, doing surgery on a live flow.
- Solution
- An engine that surfaces those requirements before you build, hands your coding tool a gotcha-aware spec, then grades what actually got built and tracks how each part moves run over run. Inside Claude Code it runs as a plugin — no API key.
- Proof
- 0 false-greens across 152 critical-bad trials on the labelled corpus — a ≤2.0% miss rate at 95% confidence, both runs agreeing on all 133 cases.
- Outcome
- Public on npm at v0.2.0 and installable inside Claude Code today — so the expensive discoveries land in week one instead of month three.
The problem
The failure isn't bad code. It's foresight arriving too late.
AI coding tools answer the literal ask. Nothing in the moment forces a stop to interrogate the non-obvious requirements of a feature. Depth is available but not automatic — and under time pressure it biases even harder toward “just answer it.”
You ask for a checkout. You don't know to ask for an atomic stock hold, so it isn't added. Months later, after Stripe is wired and taking real money, the race condition bites.
Now you're doing surgery on a live payment flow to retrofit something that would have been trivial to design in on day one. Forespec's job isn't more horsepower — it's making that interrogation forced, structured, and live.
The loop it keeps live
Point → build → verify → remember — and the foresight stays alive the whole way.
A prompt forgets and walks away after planning. The library compounds and the verifier keeps foresight live— that's the difference between a wrapper and an engine.
Architecture
Five components, one spine.
Refuses to just plan a feature. First interrogates what it touches, the non-obvious requirements, and the ordering that prevents rework — the forced foresight pass that can never be skipped.
Turns the interrogation into a sequenced plan with the dangerous, foundational pieces first — atomic hold before Stripe wiring, data-model shape before features built on it.
The unit of work is a feature with checkable completion criteria. Each gotcha becomes a concrete, verifiable checkpoint with a stable, namespaced id that is a permanent contract.
Reads the real repo and reconciles it against the checkpoints. A reasoning layer catches the fuzzy things; a deterministic assertion layer backstops what is mechanically checkable.
Every gotcha is saved as a transferable pattern, walled off from the never-leaves-the-project instance data. The pattern compounds across features and projects; the instance never travels.
Does the verifier tell the truth?
The whole tool rests on one number: how often the grader lets a bad thing through.
A labeled good/bad fixture corpus measures the reasoning verifier's precision, recall, and — the one that matters — its false-green rate. Both roads to a real grade answer to it: the ANTHROPIC_API_KEYpath that runs in CI, and the plugin path that runs on a Claude Code subscription — held to the same fixtures and the same launch gate rather than taken on trust because it is more convenient. “Is the verifier trustworthy?” becomes a number, not a hope.
But a corpus of snippets can't exercise the thing the plugin exists for: grep, read, and follow an import into the code that supposedly holds the guard. So the grader was pointed at eight public repositories it had never seen, and every finding was checked against the source by hand — 147 verdicts, not one of them without a file:line citation, and no fabricationsamong those audited. It came back clean on the clean repos, which is the property that makes the rest mean anything. The expected weak spot didn't appear either: a 4,332-file Python codebase, where the JS/TS-shaped keyword selector should have degraded, graded best of the whole run — because the agent greps the repository rather than leaning on the selector.
That ledger is a hand-audited field report rather than a second error rate, and it says so. It also records the defects the run found in Forespec itself — a reporting bug that made a good run look like a failed one, and an archetype detector that reads a document-signing SaaS as an AI app on the strength of a single dependency. A proof run that produced only good news would not have been a proof run.
You don't have to take that on faith. Inside Claude Code the plugin installs straight from the repository and grades on the subscription you already pay for — no API key, no second bill to set up. The standalone CLI is one command away with npx forespec, and forespec demo gives you a graded run in about 20 seconds against a bundled vulnerable-checkout example — rendered through the same code path as a live verify, so the demo can never drift from real output. The PR gate runs on a subscription too, which was the last place Forespec still asked for a metered key.
A composable standard
One shared checkpoint library. Every archetype composes it.
Checkpoints are authored once in the library and composed per archetype. A fix to a shared checkpoint lifts every archetype at once, and each new archetype reuses instead of copies. The portfolio archetype is the proof: a whole standard with nothing new to author.