A test must fail without the fix: the independent verifier

The idea

The independent verifier is the part of Crewshal that checks an AI worker's pull request out of the worker's reach. It does not accept "the tests pass" as proof. It also puts the fix back to the old code and checks that a new test then fails, so the test shows the bug. The verifier runs in the lab; it is off in production.

The problem
A worker can say its change works, and the tests can be green, while the change does not fix the bug.
Why it matters
Work counts as done only after a check that the worker cannot change, written on the pull request where the team reads it.
Where it stands
Built, off in production as of An owned lab run on November 16, 2026 showed it approve a real fix and reject a wrong patch. It is switched off in production, with the worker computers it checks.

Crewshal team · · 3 min read

Recorded run · Owned lab, real model for the checks · November 16, 2026

A real fix and a wrong patch

The platform’s verifier checks two pull requests for the same bug. Both pass the team’s tests; only one has a test that fails without its fix.

Step 1 of 6Pull requests

Lab scriptTwo pull requests for one bug

Issue #1: the search misses issues when the letter case differs. A lab script plays the worker and opens two pull requests. #3 changes the search and adds a test. #4 adds only a test, and that test also passes on the old code.

Input#3: the fix

diff --git a/src/search.js b/src/search.js
index 26f0b86..f0e42bb 100644
--- a/src/search.js
+++ b/src/search.js
@@ -1,8 +1,9 @@
 /**
  * The issues that match a search: the query is part of the title, or it is one of the labels.
- * An empty query matches every issue.
+ * Letter case and the spaces around the query do not matter. An empty query matches every issue.
  */
 export function matchIssues(issues, query) {
-  if (!query) return issues
-  return issues.filter(issue => issue.title.includes(query) || issue.labels.includes(query))
+  const wanted = (query ?? '').trim().toLowerCase()
+  if (!wanted) return issues
+  return issues.filter(issue => issue.title.toLowerCase().includes(wanted) || issue.labels.some(label => label.toLowerCase() === wanted))
 }
diff --git a/test/search.test.js b/test/search.test.js
index b24b17d..11aa5d9 100644
--- a/test/search.test.js
+++ b/test/search.test.js
@@ -14,3 +14,8 @@ test('a word of a title finds that issue', () => {
 test('a label finds every issue with that label', () => {
   assert.deepEqual(matchIssues(ISSUES, 'auth').map(issue => issue.id), [1, 4, 7])
 })
+
+test('letter case and spaces around the query do not matter', () => {
+  assert.deepEqual(matchIssues(ISSUES, ' Login ').map(issue => issue.id), [1, 7])
+  assert.deepEqual(matchIssues(ISSUES, 'AUTH').map(issue => issue.id), [1, 4, 7])
+})

Input#4: the wrong patch

diff --git a/test/search.test.js b/test/search.test.js
index b24b17d..e8e2b73 100644
--- a/test/search.test.js
+++ b/test/search.test.js
@@ -14,3 +14,7 @@ test('a word of a title finds that issue', () => {
 test('a label finds every issue with that label', () => {
   assert.deepEqual(matchIssues(ISSUES, 'auth').map(issue => issue.id), [1, 4, 7])
 })
+
+test('login finds the rate limit issue', () => {
+  assert.ok(matchIssues(ISSUES, 'login').some(issue => issue.id === 7))
+})

Where it ran. An owned lab with a Git forge and a worker computer in Docker. A script played the worker and pushed both pull requests; the verifier ran on its own, and only its issue checks called a real model (Opus). The verifier runs in the lab; it is off in production.

The recorded files behind this replay
  • Run record: stages, steps, gates, model calls sha256 bab9d08305d3
  • Pull request #3: the fix sha256 3181896d39bf
  • Pull request #4: the wrong patch sha256 c955c2299328
  • Verifier's verdict on pull request #3 sha256 62827735cf02
  • Verifier's verdict on pull request #4 sha256 1dc15f5974b3

Why it matters

A worker can say that its change works, and the team's tests can all be green, while the change does not fix the bug. A new test that passes on the old code proves nothing about the fix.

The verifier runs on its own, with its own account, in a container that the worker cannot change. Its verdict is one of the gates that the work must pass before it counts as done, and it is written on the pull request where the team can read it.

How it works

  1. Read the pull request. The verifier reads the target branch, the branch name and the commits from the forge.
  2. Test the old code. It runs the team's commands on the code before the change. If they fail there, a later result says nothing about the change.
  3. Write a check from the issue alone. A model sees only the issue text and the old code, and writes one check. The check must fail on the old code. The verifier records its hash before it fetches the change, so the change cannot shape it.
  4. Test the change. It fetches the change and runs the team's commands on it.
  5. Revert proof. It puts the code files back to the old version and runs each new or changed test. At least one must fail. A change that edits only tests fails this step.
  6. Issue check on the change. It runs the recorded issue check on the change.
  7. Verdict. It approves the pull request on the forge, or requests changes with the step and the reason.

What the run showed

The replay above is a recorded run from November 16, 2026 (UTC), in an owned lab. A script played the worker and opened two pull requests for the same bug. Only the verifier's issue checks used a real model.

  • Pull request #3 changed the search code and added a test. Pull request #4 added only a test, "login finds the rate limit issue", and that test also passes on the old code.
  • Both passed the team's tests on the old code. The verifier wrote each issue check before it fetched the change, and both changes then passed the team's tests.
  • At the revert proof, #3's test failed without the fix. #4 changed no code, so its run without the change was the change itself, and its test passed.
  • The verifier approved #3 and requested changes on #4, with the reason. The run passed 53 of 53 checks. The verifier took 24 s for #3 and 14 s for #4; the two issue checks cost USD 0.12 at list price.

Limits

  • Lab, not production. The verifier needs the worker's own computer, and worker computers are off in production.
  • A script played the worker here. This run tested the verifier. The run in the chapter Correct once shows the same verifier on pull requests that a real model wrote.
  • A page check needs a page port. The lab gave the verifier no page port, so the page step was skipped for #3. For #4 it did not run, because the verifier stopped at the revert proof.
  • One check per verification. Each verification writes its own issue check from the issue text, with a model; a check can miss a part of the issue that the text does not state.

Frequently asked questions

Can the worker change the verifier's result?

No. The verifier runs in its own container with its own forge account. In this lab, the team's branch rule counted only the verifier's approval. The product checks that main refuses direct pushes, needs at least one approval, and that the verifier cannot push; if not, the work does not count as done.

Why is a passing test not enough?

A test that also passes on the old code does not show the bug. The revert proof runs the new test without the fix; only a test that then fails shows that the fix matters.

What happens when the verifier rejects a change?

It requests changes on the pull request and names the step and the reason. The worker gets the same finding and must fix the change before it can finish the task.

  • Correct once

    When you correct an AI worker once, Crewshal turns the correction into a lesson, and the worker gets that lesson again when later work touches the same part of the code.

    Recorded walkthrough

  • Revert proof

    A new or changed test must fail when the fix is reverted.

  • Nine-stage lifecycle; the builder never judges itself

    Each item records which stages apply (context, architecture, design, spec, tests, build, standards, verification, delivery), why, who decided and which gate enforces each. A verdict from the builder's own context counts as Unverified.

  • Standalone autonomous AI employee

    A separate offering: one AI employee with its own private persistent computer and enrolled accounts that onboards itself, does a real engineering task alone and learns from owner corrections.

  • Ship: repository to live fix

    A person pastes a GitHub link and an issue; the platform indexes, finds, fixes on the person's own subscription, proves with revert proof and the sealed upstream test, deploys in one switch and checks the live site.

  • Worker's private persistent computer

    An admin issues a worker its own Linux computer that persists like a laptop until revoked, offboarded or expired, with full control inside its guest, an approved egress allowlist and isolation at the guest boundary (today a Docker container for a single tenant).

    Recorded walkthrough

Turn the numbers into shipped work.

Crewshal runs these choices for you: a persistent AI worker with memory and rules, on your Claude and Codex subscriptions.