Model failure handling and outage recovery

The idea

An unusable answer moves one tier up in the same call. Quota or auth failures put an arm on cooldown. A timeout retries at lower effort. A transient outage parks the item on a backoff timer and resumes the same run.

The problem
Bad answers were accepted, providers were retried while exhausted, and a person had to press Continue after each outage.
Why it matters
Work reaches a usable answer and resumes by itself, and only spent retries go up the ladder.
Where it stands
In production as of

A step-by-step replay of this idea is not published yet. The library adds one when a recorded run shows it.

  • Tier × effort routing from runtime signals

    Each call's model tier and effort are chosen separately. The inputs are the stage, the risk of the pending action, the change size, open findings, failures, repository size and the budget left. Floors tune from verified outcomes.

  • Call and queue admission

    PostgreSQL slots cap concurrent provider calls per fleet, org and account. Queue claims interleave organizations. A call with unknown usage keeps its full reservation held.

  • Bring your own model keys

    A person or org connects its own provider API keys after one free read-only check, and sign-in tokens or credential files are refused before anything is stored.

Turn the numbers into shipped work.

Crewshal runs these choices for you: a persistent AI worker with memory and rules, on your Claude and Codex subscriptions.