Six stages, each ending in a gate that produces a durable artifact the next stage reads cold. Specification through governed build, automated UAT, hybrid code review, and out to a PR contract that says what was actually proven.
/speckit-specify writes specs/NNN-slug/spec.md from a natural-language
description. What makes the rest of the workflow possible is that the acceptance scenarios
aren't prose decoration — review/criteria.py parses them into ordered criterion
IDs (CR-01, CR-02…) that later stages can prove against.
specs/003-duplicate-account-check/ spec.md ← acceptance scenarios, parsed into CR-ids plan.md research.md data-model.md quickstart.md tasks.md uat-checklist.md contracts/ checklists/requirements.md
[UI] tag. A scenario marked [UI] cannot be
proven by a view or test-client proxy. It demands a passing browser-layer test —
the rendered page, or nothing.Wave 3 of the build loop hand-derived spec 003's numbering from the raw markdown and the parser matched exactly. Counts were then hand-verified on specs 005 (16 criteria, decoy list correctly excluded) and 008 (10). All nine specs parsed at the time: 106 criteria.
/speckit-plan produces the design artifacts. The
speckit workflow puts an explicit human gate on both sides of it —
review-spec before planning and review-plan
before tasks — and both are on_reject: abort. You cannot fall
forward into implementation.
specify ──► [gate] review-spec ──► plan ──► [gate] review-plan ──► tasks ──► implement on_reject: abort on_reject: abort
Constraints live where the machinery can read them, not only in prose:
.specify/memory/constitution.md for principles,
review/risk_tier_paths.json for what counts as dangerous, and
.cursor/rules/*.mdc for workflow conventions. One authoritative list
each — decision D7 exists specifically to stop two divergent notions of "dangerous
path" from living in the repo at once.
/speckit-build merges task breakdown and implementation into one session,
because splitting them makes the implementer re-derive intent the planner already had.
It runs unattended, serially, and bounded — and it never marks its own work done.
Per task: 3 failed attempts.
Per run: one forward pass through the stories, plus at most one rework pass.
Repeating an approach that already failed is not progress.
CONVERGED — every story passed the verifier, suite green. The only success state.
BLOCKED — environment or coordination problem. Migration divergence lands here by rule.
EXHAUSTED — hit the boundary without converging.
NO_PROGRESS — two consecutive attempts cleared nothing.
The builder never marks a story done on its own judgment; a separate verifier decides.
A run that ends in prose alone has not finished — it must classify its outcome into
build-receipt.json. That receipt is what the dashboard and the next stage read.
Waves 1–4 of the harness were themselves built under this protocol: a fresh builder agent in its own worktree makes one bounded change; a fresh reviewer agent in a detached checkout of the builder's SHA reruns every gate from scratch, then proves the test by reverting the change (must go red) and mutating it (must go red).
| Tier | Means | Result |
|---|---|---|
| T1 proven | All gates green + revert proof red + mutation proof red | 4 of 4 waves |
| T2 gated | Gates green, proof partial — reason logged | 0 |
| P parked | Human required; evidence attached | 0 |
The reviewer chose its own mutations, not the builder's. Across the four waves, 16 of
17 independent mutations were killed. The one survivor was honest signal, not a gap:
disabling the duplicate-check service left the behavior blocked anyway, because
User.email unique=True still holds — CR-02 turned out to be enforced three
times over. Recorded as a finding, not patched away.
R1/R2/R3 grade a change. T1/T2/P grade how well a completed wave was proven. Different axis, different vocabulary — the docs call this out explicitly because the two were being conflated.
This is a different question from the build's own gates. The build asks "do my tests pass?"
The review phase asks "is every required acceptance criterion in spec.md
actually proven?" — and it asks from a fresh context that never saw the build.
spec.md ──parse──► criteria.py ─┐ │ (CR-ids, [UI] flags) review-marked ──pytest──► conftest collector ──► results.json tests (markers) │ ▼ packet.py: compute_coverage + verdict │ │ review-packet.md review-summary.json (human-facing) (machine-facing, SSOT) │ ┌───────────────────────────────────────────────┬──────┴────────┐ ▼ ▼ ▼ run.py (two-lane, gate exit code) loop.py build-dash make review / review-det (control plane) Review screen
Service, view, model and admin-layer checks. Browser-free by construction: pytest
addopts pins -m "not integration" so the gate can never
depend on a browser being installed.
Playwright browser proofs, run explicitly via -m "review and integration".
A flake here downgrades the verdict to 🟡; it never fails the gate. Screenshots and
video land in review/artifacts/browser/.
A green suite is not a green verdict. packet.py only reports pass when every
required criterion is proven. An unproven criterion is treated as risk, not as
test hygiene — it downgrades the verdict to 🟡 Incomplete and says so loudly in the packet,
while exiting 0. Only a check that ran and failed exits non-zero.
| Spec | Required | Verified | State |
|---|---|---|---|
| 003-duplicate-account-check | 9 | 4 | 5 unproven |
| 013-transcript-reminders | 16 | 16 | complete |
| 014-course-creation-core | 10 | 10 | complete |
review/loop.py decides the next action —
FIX_RED before PROVE_UNVERIFIED —
and decides when to stop: CONVERGED,
PARKED, NO_PROGRESS,
EXHAUSTED. A genuine pass is checked first, so a green
run at the budget edge is never mislabelled as exhausted.
The decider is not the fixer. loop.py never edits application code.
The agent that fixes is an injected, pluggable callback — the speckit-review
skill. Demonstrated live moving spec 003 from 3/9 to 4/9 proven.
An agent reads every line of a large diff and never gets bored on line 400 — but it cannot decide what matters, and unconstrained it invents findings that evaporate under scrutiny. A human decides what matters — but at this volume cannot read everything, and pretending otherwise produces approvals that look like coverage. Each half covers the other's failure mode.
review/tier.py derives R1/R2/R3 from the changed-file list against
review/risk_tier_paths.json. Highest match wins: one R3 path in an otherwise
docs-only PR makes the whole PR R3. A dev may escalate freely; lowering requires a named
approver and a recorded reason.
| Requirement | R1 | R2 | R3 |
|---|---|---|---|
| Deterministic checks (§4.1) | ✓ | ✓ | ✓ |
| LLM judgment pass | — | ✓ | ✓ |
| Failure-path tests for new behavior | — | ✓ | ✓ |
make review incl. Playwright | — | if spec'd | ✓ |
| Mutation proof on new tests | — | — | ✓ |
| Human review | — | 1 | 1 + independent 2nd agent pass |
| Preview environment exercised | — | — | ✓ |
Path globs couldn't express "anything Stripe" — apps/applications/views.py
holds StripeWebhookView and the PaymentIntent creator but has no "stripe" in
its filename, so money code was grading R2. The fix was content regexes matched against
file bodies, targeting SDK calls and settings keys rather than the bare word so prose
doesn't trip them. Erring toward R3 is deliberate: tiering money code down is
the unsafe direction.
Six commands, all six must pass, and none is redundant.
review/deterministic.py shells out and records real subprocess output with
the SHA it ran against.
| Command | Catches what nothing else does | Result |
|---|---|---|
ruff check . | Lint violations | pass |
black --check . | Formatting drift | pass |
pytest --cov | Behavioral regressions | pass |
manage.py check | Django config errors — pytest passes with a misconfigured app registry | pass |
makemigrations --check | Model/migration drift — pytest structurally cannot catch this | pass |
lintmigrations | Backwards-incompatible migrations — "is this safe to deploy?", not "is one missing?" | pass |
f64a6c27 — "Green the deterministic gate" — took this table from four
reds to six greens. ruff/black were the easy ones: 2 lint
errors and 4 unformatted files, all auto-fixable, plus one real modernization
(typing.Callable → collections.abc.Callable). The other two
reads were hiding something worse than a lint violation:
lintmigrations was crashing, not reporting (TD-16). It ran
sqlmigrate over every third-party migration and died on a Wagtail one —
wagtailcore_grouppagepermission. Scoping it to our own apps stopped the
crash and surfaced 3 real errors in our own migrations that the crash had been
hiding: NOT NULL constraints on columns in migrations already applied to
prod, which can't be rewritten after the fact. Baselined by name rather than
disabling the rule — verified live by removing a baseline entry and watching it go
red again.DJANGO_MIGRATION_LINTER; the package reads
MIGRATION_LINTER_OPTIONS — zero references to the old name anywhere in
the installed package. ignore_name_patterns and
warnings_as_errors had never applied to anything, silently, until this
commit renamed it.fingerprint_for()
claimed to be a pure disk-state read but went through MigrationLoader,
which pytest-django's --nomigrations patches to report zero migrations
once any test sets up a database. Passed in isolation, failed in the full suite — every
fingerprint was silently collapsing to the same empty-string hash. Fixed by reading the
migrations package off disk directly, pinned by a django_db regression
test that forces the exact failure condition.
Deliberately not maximal. all_warnings_as_errors stayed off — two
already-applied migrations carry unfixable CREATE INDEX warnings, and failing
the gate on day one over something nobody can act on is exactly how a gate earns being
ignored. Live right now, re-run at be2ca08: 1,108 passed, 4 deselected;
tests/review/ 281 passed; tree clean at commit.
Green checkmarks above weren't always trustworthy in the way they look. Until
be2ca08, code_review.py's prepare command
returned 0 unconditionally, discarding the verdict it had just
computed — and the hook wrapped the call in if, which disarms
set -e, so a genuinely red gate printed nothing at all, not even to
stderr. Two swallow points stacked on top of each other. Fixed and proven by injecting
a ruff error and watching prepare move from exit 0 to exit 1, with the
hook reporting it instead of going quiet. The table above is the first version of
this panel where a red row was structurally guaranteed to be seen.
The makemigrations --check line is the one most easily dropped and most
expensive to lose: addopts pins --nomigrations, so pytest builds
the test schema straight from the models and never executes a migration. A green suite
proves nothing about whether the migration files match the models.
Fill a behavior→proof matrix by lookup: behavior changed, test that proves it, red without the change, failure path. You cannot name a test without confirming it exists. Empty cells are the findings.
A defect report where every finding carries Location, Input, Expected, Actual, Repro, Severity, Route. Anything that can't fill Input/Actual/Repro goes to a separate Questions for the human list that is explicitly not severity-ranked.
That separation is the single most important implementation detail. Without it, uncertainty leaks into the findings list as hedged bugs and the output loses credibility by week three.
| Bucket | Fixable by | Handling |
|---|---|---|
| Code defect | Engineering | Fix in this PR if in scope, else a bd issue |
| Spec contradiction | Product owner | Engineering cannot fix it — two documented positions disagree |
| Phantom implementation | Whoever owns status | Correct the tracker first; planning is running on it |
| Test infrastructure | Engineering | Fix before trusting any pass rate, including this review's |
| Coverage gap | Engineering | Failure paths before happy paths |
| Doc rot | Anyone | Cheap — batch them |
| Dismissed | Author | To the contract's disposition field, with the reason |
Routing a spec contradiction to engineering wastes a sprint. It comes back as "blocked, needs decision" no matter who picks it up.
review/contract.py emits the machine half from real results. The human writes
only three fields: Intent, Dispositions, and Reviewer focus areas. Everything else is
filled or it isn't there.
Risk tier: R3 — High computed from changed paths. Not author-declared. Size: 6,417 hand-written lines · 0 generated · 49 files ⚠ over the ~250 guideline — consider splitting Intent ✍ human Deterministic 6 of 6 checks pass, vs be2ca08a · clean tree Spec/UAT evidence → review-packet.md — 0 proven, 6 unverified Test evidence 18 changed test files, all "unproven — no behavior→proof matrix on disk" Dispositions ✍ human — what you did NOT act on, and why Focus areas ✍ human — where you want eyes, and where you don't
Read what a green run still refuses to do. All six checks pass now — but the contract does not read that as "done." Seventeen changed test files still come back unproven, with the reason stated: no behavior→proof matrix exists on disk yet, because that matrix is B3's job, not the deterministic gate's. It will not let a size warning pass silently at 6,417 lines. Green deterministic checks and a proven spec are different claims, and the contract keeps them in separate rows on purpose.
Approving asserts three things and nothing more: findings adjudicated, intent right, you can explain it. It does not assert "I read every line." On R2 the agent asks a substantive "why" and the human answers in the PR; on R3 the human walks through the change and the agent challenges the explanation against the code. "The agent wrote it and the tests pass" is not an explanation.
Bitbucket Cloud has no PR label primitive, so the description is the only place the tier is
both durable and visible without opening a tool. The packet copy is what survives if the
description is edited. As of the last commit (85abac8), a human no longer pastes it:
review/pr.py finds the open PR for the current branch and upserts the contract
between HTML-comment markers — idempotent, so re-running replaces the block instead of
appending another copy, and preserves anything a human wrote outside the markers.
make code-review-publish previews the write; code-review-publish-apply
sends it. Nothing reaches the PR without that explicit flag — an outward-facing write
doesn't get a silent default.
tdd-gate.sh on preToolUse. code-review-prepare.sh on
beforeSubmitPrompt and stop — fast path skips pytest for latency, the stop path runs it.
understand-auto-update.sh keeps the knowledge graph fresh. The collector
hook feeds build-dash on every event.
Every wave claims a bead before work and closes it after the merge lands — an ordering
learned the hard way when bd close auto-committed onto a branch mid-merge.
No TodoWrite, no markdown TODO lists.
An event reducer over the hook stream with parsers per stage: spec, plan, tasks,
research, build-receipt, review. It renders review-summary.json and
re-implements none of its logic.
Per-agent databases with --create-db. Parallel agents on a shared
--reuse-db deadlock and then report infrastructure noise as product failures.
These aren't aspirations written after the fact. Each one is a lesson already paid for, and dropping any of them is how the workflow becomes noise.
Never ask an agent to find bugs.
A bug hunt asks the model to be right about something it invented. Give it a document to produce that requires citations, and one hard rule: every factual claim must be verified against the code first. Findings then arrive as a side effect of not being allowed to write the comfortable sentence.
docs/code-review-spec.md §2 — inherited from the tech-debt sweepA defect claim must come with a reproduction, not a suspicion.
State the concrete input, the exact line, and the wrong result. If you cannot demonstrate the failure it is not a finding — downgrade it to a question for the human, or drop it. Never report it as a probable bug.
docs/code-review-spec.md §2 — the rule the sweep didn't haveNever lead the witness.
The reviewing agent gets the diff and the spec. It does not get the author's explanation of why the code is fine. In the review phase the same rule runs the other way: to close a red finding you fix the real behavior — never edit or weaken the check to turn it green.
code-review-spec.md §4 · speckit-review SKILL.md · LOOP-STATE.md hard rulesNo borrowed verdicts.
If an exemplar carries a strong claim — "9/9 proven" — agents will copy that shape onto work that earned nothing of the sort. It has to be forbidden by name: the exemplar says X because it had Y; your unit has no Y; state your actual evidentiary basis. Without this, review launders confidence, which is worse than no review.
docs/code-review-spec.md §4The decider is not the fixer.
The convergence control plane decides what to fix and when to stop, and never edits application code. The agent that fixes is an injected, pluggable callback. A review that fixes things is unreviewable and unbounded — write-scope confinement is a constraint in the prompt, not a convention.
review-phase-architecture.md §3 tenet 8 · code-review-spec.md §4Single source of truth in Python; the dash only renders.
All coverage and verdict logic lives in packet.py, which emits a
structured review-summary.json. The dashboard maps that JSON onto a screen
and never re-implements the logic. The same discipline puts tier paths in one JSON file
that both the review skill and the build loop's park rule read.
The workflow's own verdict on itself, read from
review-summary.json. Everything below is the independently audited
column of docs/code-review-spec.md §8 — not a self-report.
| Step | What | State | The gap |
|---|---|---|---|
| P1 | pytest_plugins collection errors | met | 739/743 collected, 0 errors, no tests deleted to get there |
| P2 | Per-agent DB isolation | met | Same suffix twice still collides; every run leaks a database |
| B1 | Failure-path requirement in the build skill | met | Closed (be2ca08) — review/failure_paths.py could measure but was structurally incapable of failing: no threshold, every code path returned 0. Added --changed/--base (diff-scoped, not the global 26.7%) and --fail-on-zero, wired into prepare via make review-failure-paths-gate. Deliberately scoped to the diff — gating on legacy zero-coverage files would fail forever and teach people to disable it. Caught a real offender on its first run: test_program_cta_service.py was happy-path only |
| B2 | Tier computation | met | Closed since the earlier read — R100\told\tnew now parses as the three tab-separated fields git actually emits, so a rename into an R3 directory grades correctly. Stripe coverage is by content pattern, not filename, so money code can't hide behind a name with no "stripe" in it. 41 tests |
| B3 | The code-review skill — both lanes | partial | Still the real gap: never run on a real PR — that clause is exercise, not code, and it was deliberately not softened to match what got built. What did move (be2ca08): the Step 2 conformance block was checked in as a literal <run> placeholder, not runnable, and its *.md glob flagged pr-contract.md itself as malformed unit output — unit files now land in latest/units/ with nullglob so an empty run is silent. Two of the doc's own complaints were struck as false rather than fixed: the routing table has carried a "Fixable by" owner column all along, and the conformance grep covers 6 of 6 mandated headings, not 4 |
| B4 | Execution capability | met | Closed since the earlier read — all 6 commands now produce real results. Pytest no longer dies at argparse with zero tests run; a check that could not execute is recorded as error, kept structurally distinct from fail so an infrastructure gap can never masquerade as a red suite. HEAD is captured before and after, tree dirtiness is recorded, and the contract carries the ⚠️ warning when results describe uncommitted code |
| B5 | ruff/black back in an automatic gate | met | Closed (be2ca08) — two swallow points, not one. prepare returned 0 unconditionally, discarding the verdict it had already written to the manifest; it now returns a result and exits non-zero on any red check. The hook wrapped the call in if, which disarms set -e, so a failing prepare printed nothing — not even stderr; it now captures the exit status explicitly. Proven, not asserted: injecting a ruff error moves prepare from exit 0 to exit 1 and the hook reports it. bd lsuoce-registration-d0h (move lint/format into a review skill) is still open — the gate exists now, but the skill migration it originally asked for hasn't happened |
| B6 | Contract emission | met | Closed since the audit (85abac8) — review/pr.py resolves the open Bitbucket PR from origin and upserts the contract into its description between HTML-comment markers, so re-running replaces rather than appends and human prose outside the block survives. Nothing is written without --apply. 29 tests, all offline |
| B7 | Automatic triggering | met* | Closed (be2ca08), * a stated boundary below. Cursor hooks are shell-only — there's no agent/prompt field in hooks.json — so the fan-out can't be invoked directly. The hook now surfaces a red gate via additional_context instead of staying silent, following the repo's own precedent in understand-auto-update.sh ("tell the developer, do not silently run it yourself"). It nudges; a human still triggers /code-review and still runs code-review-publish-apply. The status cell says "with a boundary" rather than claiming more than that |
| B8 | Independent second pass for R3 | partial | Ranking and corroboration were already fixed — corroborated-first within severity band, matched on file:line/token overlap, pass 2 as a sibling directory. New this round (be2ca08): runs made before that fix had left a stale latest/second-pass/ child directory whose own INSTRUCTIONS.md still taught the layout the fix forbids — an artifact actively instructing the wrong thing, worse than no artifact. init_second_pass now detects and removes it. Still open: no live dual-pass run has ever exercised any of this |
Named in the spec so they get checked early rather than discovered late:
Failure-path coverage is 26.7% — a real number rather than an alarming one. The actionable
target is the 13 files with zero, not the aggregate.
make review-failure-paths ARGS=--zero names them.
The previously published figure was 2.7% and it was wrong. A shell grep counted assertion
tokens and missed this codebase's dominant idiom — assert not result —
scoring a password-reset file with 8 failure paths of 16 as zero. It also counted the wrong
unit: the standard is a property of a test, not of an assertion. The measurement now lives
in review/failure_paths.py as AST analysis per test function, so it can be
re-checked rather than re-argued.
# Stage 04 — automated UAT make review # both lanes, incl. Playwright — before any PR with user-facing criteria make review-det # deterministic lane only — no browsers, safe anywhere make review-loop # one review + the next action + the terminal state # Stage 05 — hybrid code review make review-tier # R1/R2/R3 from the diff — nobody types it make code-review-prepare # tier + the six §4.1 checks + contract → artifacts/latest/ make code-review-deterministic make code-review-contract # Measurement make review-failure-paths ARGS=--zero # the 13 files with none make test-isolated DB_NAME_SUFFIX=review_unit ARGS='apps/foo/'
None of these run in CI, deliberately. The standard is dev-side because a failure a developer can act on immediately is worth more than one that surfaces after the fact — but dev-side does not mean manual. The hooks run the machine half; the developer sees results. A standard that depends on human diligence gets skipped under deadline, and its being skipped is invisible. That is precisely how the security scan and the PR Validation step both went dark for months.