A test must fail without the fix: the independent verifier
The idea
The independent verifier is the part of Crewshal that checks an AI worker's pull request out of the worker's reach. It does not accept "the tests pass" as proof. It also puts the fix back to the old code and checks that a new test then fails, so the test shows the bug. The verifier runs in the lab; it is off in production.
- The problem
- A worker can say its change works, and the tests can be green, while the change does not fix the bug.
- Why it matters
- Work counts as done only after a check that the worker cannot change, written on the pull request where the team reads it.
- Where it stands
- Built, off in production as of An owned lab run on November 16, 2026 showed it approve a real fix and reject a wrong patch. It is switched off in production, with the worker computers it checks.
Crewshal team · · 3 min read
Recorded run · Owned lab, real model for the checks · November 16, 2026
A real fix and a wrong patch
The platform’s verifier checks two pull requests for the same bug. Both pass the team’s tests; only one has a test that fails without its fix.
Lab scriptTwo pull requests for one bug
Issue #1: the search misses issues when the letter case differs. A lab script plays the worker and opens two pull requests. #3 changes the search and adds a test. #4 adds only a test, and that test also passes on the old code.
Input#3: the fix
diff --git a/src/search.js b/src/search.js
index 26f0b86..f0e42bb 100644
--- a/src/search.js
+++ b/src/search.js
@@ -1,8 +1,9 @@
/**
* The issues that match a search: the query is part of the title, or it is one of the labels.
- * An empty query matches every issue.
+ * Letter case and the spaces around the query do not matter. An empty query matches every issue.
*/
export function matchIssues(issues, query) {
- if (!query) return issues
- return issues.filter(issue => issue.title.includes(query) || issue.labels.includes(query))
+ const wanted = (query ?? '').trim().toLowerCase()
+ if (!wanted) return issues
+ return issues.filter(issue => issue.title.toLowerCase().includes(wanted) || issue.labels.some(label => label.toLowerCase() === wanted))
}
diff --git a/test/search.test.js b/test/search.test.js
index b24b17d..11aa5d9 100644
--- a/test/search.test.js
+++ b/test/search.test.js
@@ -14,3 +14,8 @@ test('a word of a title finds that issue', () => {
test('a label finds every issue with that label', () => {
assert.deepEqual(matchIssues(ISSUES, 'auth').map(issue => issue.id), [1, 4, 7])
})
+
+test('letter case and spaces around the query do not matter', () => {
+ assert.deepEqual(matchIssues(ISSUES, ' Login ').map(issue => issue.id), [1, 7])
+ assert.deepEqual(matchIssues(ISSUES, 'AUTH').map(issue => issue.id), [1, 4, 7])
+})
Input#4: the wrong patch
diff --git a/test/search.test.js b/test/search.test.js
index b24b17d..e8e2b73 100644
--- a/test/search.test.js
+++ b/test/search.test.js
@@ -14,3 +14,7 @@ test('a word of a title finds that issue', () => {
test('a label finds every issue with that label', () => {
assert.deepEqual(matchIssues(ISSUES, 'auth').map(issue => issue.id), [1, 4, 7])
})
+
+test('login finds the rate limit issue', () => {
+ assert.ok(matchIssues(ISSUES, 'login').some(issue => issue.id === 7))
+})
Lab scriptTwo pull requests for one bug
Issue #1: the search misses issues when the letter case differs. A lab script plays the worker and opens two pull requests. #3 changes the search and adds a test. #4 adds only a test, and that test also passes on the old code.
The record for step 1
Input#3: the fix
diff --git a/src/search.js b/src/search.js index 26f0b86..f0e42bb 100644 --- a/src/search.js +++ b/src/search.js @@ -1,8 +1,9 @@ /** * The issues that match a search: the query is part of the title, or it is one of the labels. - * An empty query matches every issue. + * Letter case and the spaces around the query do not matter. An empty query matches every issue. */ export function matchIssues(issues, query) { - if (!query) return issues - return issues.filter(issue => issue.title.includes(query) || issue.labels.includes(query)) + const wanted = (query ?? '').trim().toLowerCase() + if (!wanted) return issues + return issues.filter(issue => issue.title.toLowerCase().includes(wanted) || issue.labels.some(label => label.toLowerCase() === wanted)) } diff --git a/test/search.test.js b/test/search.test.js index b24b17d..11aa5d9 100644 --- a/test/search.test.js +++ b/test/search.test.js @@ -14,3 +14,8 @@ test('a word of a title finds that issue', () => { test('a label finds every issue with that label', () => { assert.deepEqual(matchIssues(ISSUES, 'auth').map(issue => issue.id), [1, 4, 7]) }) + +test('letter case and spaces around the query do not matter', () => { + assert.deepEqual(matchIssues(ISSUES, ' Login ').map(issue => issue.id), [1, 7]) + assert.deepEqual(matchIssues(ISSUES, 'AUTH').map(issue => issue.id), [1, 4, 7]) +})Input#4: the wrong patch
diff --git a/test/search.test.js b/test/search.test.js index b24b17d..e8e2b73 100644 --- a/test/search.test.js +++ b/test/search.test.js @@ -14,3 +14,7 @@ test('a word of a title finds that issue', () => { test('a label finds every issue with that label', () => { assert.deepEqual(matchIssues(ISSUES, 'auth').map(issue => issue.id), [1, 4, 7]) }) + +test('login finds the rate limit issue', () => { + assert.ok(matchIssues(ISSUES, 'login').some(issue => issue.id === 7)) +})VerifierFirst, the old code and a check from the issue
The verifier works in its own container, out of the worker's reach. It reads each pull request from the forge and runs the team's tests on the old code. Then a model sees only the issue text and the old code, and writes one check for each pull request. Each check fails on the old code, as the bug says it should. The verifier records the check's hash before it fetches any change, so the change cannot shape it.
The record for step 2
Check#3
- Read the pull request from the forgePassed
- Run the team's tests on the old codePassed
- Write a check from the issue alone; it must fail on the old codePassed
Check#4
- Read the pull request from the forgePassed
- Run the team's tests on the old codePassed
- Write a check from the issue alone; it must fail on the old codePassed
DecisionIssue checks, recorded before the fetch
- #3 check
- sha256 7bf8882e14f9
- #4 check
- sha256 1a7ea8fb4f03
- Model calls
- 2, USD 0.12 at list price
VerifierBoth changes pass the team's tests
Only now does the verifier fetch each change and run the team's tests on it. Both changes pass. Green tests alone cannot tell the two apart.
The record for step 3
Check#3
- Fetch the changePassed
- Run the team's tests on the changePassed
Check#4
- Fetch the changePassed
- Run the team's tests on the changePassed
InputThe lab script's test run on the worker's computer, before each push
- #3
- 6 pass, 0 fail
- #4
- 6 pass, 0 fail
VerifierThe revert proof
The verifier puts the code files back to the old version and runs each new test. #3's test fails without the fix, so it shows the bug. #4 changed no code. Putting the old code back changes nothing, so its test runs on the same code and still passes: it does not show the bug.
The record for step 4
CheckRevert proof
- #3: "letter case and spaces around the query do not matter" fails on the old codePassed
- #4: "login finds the rate limit issue" passes on the old codeDid not pass
DecisionThe verifier's reason for #4
test/search.test.js › login finds the rate limit issue passes without the change, so it does not show the bug. The change edits only tests, so the run without the change is the head itself.
VerifierOne approved, one sent back
#3 also passes the issue check on the change, and the verifier approves it on the forge. For #4 the verifier stops at the revert proof and requests changes, with its reason. The page check is skipped for #3, because the lab gave the verifier no page port, and does not run for #4.
The record for step 5
Output#3
- Outcome
- Verified
- Review on the forge
- Approved
- Verifier time
- 24 s
Output#4
- Outcome
- Rejected at the revert proof
- Review on the forge
- Changes requested
- Verifier time
- 14 s
CrewshalMain stays protected
In this lab, the team's branch rule counted only the verifier's approval: a merge needed 1 approval, and nobody could push to main directly. The verifier's account cannot push to main either. The product checks that main refuses direct pushes, needs at least one approval, and that the verifier cannot push; if not, the work does not count as done.
The record for step 6
CheckMain branch
- Protected
- Yes
- Approvals needed
- 1
- Who could approve, in this lab
- The verifier's account only
- Verifier's account can push to main
- No
Result
The verifier approved the fix and sent back the patch whose test passed without it.
- Run checks passed
- 53 of 53
- Verifier time
- 24 s and 14 s
- Model cost of the checks
- USD 0.12
Where it ran. An owned lab with a Git forge and a worker computer in Docker. A script played the worker and pushed both pull requests; the verifier ran on its own, and only its issue checks called a real model (Opus). The verifier runs in the lab; it is off in production.
The recorded files behind this replay
- Run record: stages, steps, gates, model calls sha256 bab9d08305d3
- Pull request #3: the fix sha256 3181896d39bf
- Pull request #4: the wrong patch sha256 c955c2299328
- Verifier's verdict on pull request #3 sha256 62827735cf02
- Verifier's verdict on pull request #4 sha256 1dc15f5974b3
Why it matters
A worker can say that its change works, and the team's tests can all be green, while the change does not fix the bug. A new test that passes on the old code proves nothing about the fix.
The verifier runs on its own, with its own account, in a container that the worker cannot change. Its verdict is one of the gates that the work must pass before it counts as done, and it is written on the pull request where the team can read it.
How it works
- Read the pull request. The verifier reads the target branch, the branch name and the commits from the forge.
- Test the old code. It runs the team's commands on the code before the change. If they fail there, a later result says nothing about the change.
- Write a check from the issue alone. A model sees only the issue text and the old code, and writes one check. The check must fail on the old code. The verifier records its hash before it fetches the change, so the change cannot shape it.
- Test the change. It fetches the change and runs the team's commands on it.
- Revert proof. It puts the code files back to the old version and runs each new or changed test. At least one must fail. A change that edits only tests fails this step.
- Issue check on the change. It runs the recorded issue check on the change.
- Verdict. It approves the pull request on the forge, or requests changes with the step and the reason.
What the run showed
The replay above is a recorded run from November 16, 2026 (UTC), in an owned lab. A script played the worker and opened two pull requests for the same bug. Only the verifier's issue checks used a real model.
- Pull request #3 changed the search code and added a test. Pull request #4 added only a test, "login finds the rate limit issue", and that test also passes on the old code.
- Both passed the team's tests on the old code. The verifier wrote each issue check before it fetched the change, and both changes then passed the team's tests.
- At the revert proof, #3's test failed without the fix. #4 changed no code, so its run without the change was the change itself, and its test passed.
- The verifier approved #3 and requested changes on #4, with the reason. The run passed 53 of 53 checks. The verifier took 24 s for #3 and 14 s for #4; the two issue checks cost USD 0.12 at list price.
Limits
- Lab, not production. The verifier needs the worker's own computer, and worker computers are off in production.
- A script played the worker here. This run tested the verifier. The run in the chapter Correct once shows the same verifier on pull requests that a real model wrote.
- A page check needs a page port. The lab gave the verifier no page port, so the page step was skipped for #3. For #4 it did not run, because the verifier stopped at the revert proof.
- One check per verification. Each verification writes its own issue check from the issue text, with a model; a check can miss a part of the issue that the text does not state.
Frequently asked questions
Can the worker change the verifier's result?
No. The verifier runs in its own container with its own forge account. In this lab, the team's branch rule counted only the verifier's approval. The product checks that main refuses direct pushes, needs at least one approval, and that the verifier cannot push; if not, the work does not count as done.
Why is a passing test not enough?
A test that also passes on the old code does not show the bug. The revert proof runs the new test without the fix; only a test that then fails shows that the fix matters.
What happens when the verifier rejects a change?
It requests changes on the pull request and names the step and the reason. The worker gets the same finding and must fix the change before it can finish the task.