Blind evaluation against shipped human work
The idea
The worker does real merged tasks again from a rewound repository. Blind critics from several model families compare its change with the human's, in both orders.
- The problem
- Passing tests does not show that the work matches a strong engineer's.
- Why it matters
- A loss names the platform layer to fix, and only warm, model-matched runs count.
- Where it stands
- Shown in a lab run as of
A step-by-step replay of this idea is not published yet. The library adds one when a recorded run shows it.