Blind evaluation against shipped human work

The idea

The worker does real merged tasks again from a rewound repository. Blind critics from several model families compare its change with the human's, in both orders.

The problem
Passing tests does not show that the work matches a strong engineer's.
Why it matters
A loss names the platform layer to fix, and only warm, model-matched runs count.
Where it stands
Shown in a lab run as of

A step-by-step replay of this idea is not published yet. The library adds one when a recorded run shows it.

  • Keep-or-discard harness loop

    A harness change is kept only if the declared capability metrics stay within tolerance, one efficiency metric improves by the minimum, and frozen held-out cases confirm it. Campaigns count every attempt.

  • Historical PR replay trial

    A customer picks a merged GitHub PR. The worker does it again blind from the base-only history, and a report compares its result with the sealed reference.

Turn the numbers into shipped work.

Crewshal runs these choices for you: a persistent AI worker with memory and rules, on your Claude and Codex subscriptions.