- Learn
207 ideas · 9 recorded walkthroughs · 29 benchmark terms · updated November 21, 2026
How Crewshal works, idea by idea.
Each idea starts with what it is and why it matters, in plain words. Nine ideas are shown on recorded runs that you can replay step by step: what went in, what the platform decided, and what came out. The rest are short definitions.
Start here
A short path through the ideas that make Crewshal different, in reading order.
Checks and proof
A receipt for every task
How do I know a task is really done, and what it cost?
Learning and improvement
Correct once
Do I have to repeat a correction in every new task?
Checks and proof
Independent verifier
Who checks the worker's work, and can the worker fool it?
Decisions and authority
The closed list
Which questions can still reach me?
Timeline
Source health
Can I trust that the Timeline is complete?
The AI organization
Named AI employees, a chief executive, managers, teams and people in one organization chart, and how they work together.
AI employees with names and roles
Each AI worker is a named employee with a role, persona, model, memory and tools, hired into the organization.
CEO does hands-on work, code included
The CEO also holds the engineer's code, git, command and check tools; it delegates first and works itself when that is the fastest correct path.
CEO front door over a worker hierarchy
The operator talks to one CEO worker; department heads and workers sit below it, and each level's budget and tool limits bind its whole subtree.
An admin turns a brief into a saved company plan of workers, reporting lines, instructions, outputs and a repeatable pipeline before any worker is created.
Delegation down the chain with fan-out
A manager splits work into parts, creates every child in one transaction with dependencies, and waits once for all of them.
An employee asks its manager for headcount, budget, a tool, a model or access with evidence; a queue full for 3 days raises a headcount request by itself.
Planning and doing the work
How a request becomes defined, planned, built, checked and delivered work.
A message with timing words becomes a card (what, when, next three runs, who, credits); Start creates a schedule or event trigger whose runs report back to the asker.
A campaign is a root item and all work under it, with limits, one-click hold and resume, one instruction sent to all open work, and a report of runs, pull requests, cost and follow-ups.
Code checks for code that AI workers operate
A check without a model holds worker code to module size, header, import cycles, colocated tests, index entries and typed exports. It runs at act exit, in verification and in the platform verifier.
A live console shows totals, a runtime map, a control desk and telemetry from stored records, and every page opens a stacked drill-down down to the step, with a why-trace and question threads.
Decisions and hard parts in task review
Task review starts with consequential choices and hard parts; people can support or ask to reconsider a choice, and that feedback reaches the worker without changing its evidence.
Before building, the worker defines what, why, for whom and how. It checks fit, knowledge and authority, and answers its own questions with research and with colleagues.
Context and memory
What a worker sees while it works, what it remembers, and how it finds what it needs.
Action fusion: edit plus check
An edit can carry its planned check. The runtime runs the check as its own gated call in the same model turn.
Typed decisions say which optional sections a turn needs and how to retrieve. Code plans the slots under a ceiling, with a reduction ladder that never cuts more than before, and limits the queries.
Area notes and knowledge that stays current
Research writes a note for each package or folder it touches, kept while the files do not change. Facts tied to a page or ticket go stale when it changes, and pushes to the default branch update ties and notes.
Byte-stable prefix, one session per ticket
The prefix stays reusable: one prompt and tool catalog per item, stage notes appended to the transcript, the model held within a stage, and cache breakpoints per provider. Claude Code runs as one live process.
The main line compacts at a finished plan step only when the estimated cache-read savings are larger than the cost to rewrite, fold and carry.
Code knowledge indexes for agents
A code index at the level of signatures finds code in milliseconds. An Atlas profile, computed by code with no model, gives each repository's stack, commands, rules and module map.
Learning and improvement
How a correction, a lesson or a past result reaches later work, and how the platform improves itself.
When you correct an AI worker once, Crewshal turns the correction into a lesson, and the worker gets that lesson again when later work touches the same part of the code.
Approved examples as style references
Worker outputs from merged or approved pull requests become examples. Later act and chat turns cite them as style references, never as instructions.
Blind evaluation against shipped human work
The worker does real merged tasks again from a rewound repository. Blind critics from several model families compare its change with the human's, in both orders.
Frozen context-recall baseline gate
A frozen offline set of 36 cases plants facts and records where each one shows in the next request and whether it can be read back. A change to context, memory or compaction must beat this baseline.
A customer picks a merged GitHub PR. The worker does it again blind from the base-only history, and a report compares its result with the sealed reference.
Any signed-in person can send feedback with a picture of the page. The operator's agent or a granted worker triages it and opens a work item, and the sender sees the outcome.
Checks and proof
How the platform proves that work is done, and refuses work that only looks done.
Every finished task gets a sealed receipt: what was done, which checks ran and how they ended, the time and the cost, built from records the platform already keeps.
The platform checks an AI worker's pull request in its own container, out of the worker's reach, and approves it only when a new test fails without the fix.
Acceptance criteria as executable checks
At plan time, each acceptance criterion maps to a test, script, query, HTTP check or screenshot check. Verification runs all of them in each round.
Check-before-finishing Stop hook
In a native Claude Code round, a Stop hook that uses no model holds the round once if code changed after the last passing check. A piped or masked status does not count as a pass.
New tests must run here or have a trace for each assertion. Claimed behaviour must be tested, and the same bug elsewhere fixed or ruled out. Planned changes must exist, steps tick only on evidence, and spend over budget needs a recorded decision.
Exact bytes behind every check
A file check binds the criterion, tenant, item, digest, size and producing action, and the bytes are hashed again before use. Selected command output is kept whole and checked again by identity and hash.
Models and routing
Which model or agent does each part of the work, and how cost and speed stay in check.
A harness around swappable agents
The parts of a worker's harness are pluggable: models, tools, context, rules, checks, decisions, verification, memory and evaluation. Each is set per organization, repository, person and run.
PostgreSQL slots cap concurrent provider calls per fleet, org and account. Queue claims interleave organizations. A call with unknown usage keeps its full reservation held.
Call manifests and layered observability
Each metered call records its role, model, layers, files, tokens, cache reads and stage budget, plus a digest of its input taken before the call. A task can be read at any level, from a glance down to these internals.
A scheduled run can skip when nothing new needs its work. After three semantic skips in a row, or 24 hours of unreviewed events, the next run opens a governed review.
Engineer pipeline with stage shares
Work follows an engineer's pipeline, and each stage gets a share of the time and cost. Review is one plan review and one holistic diff review, within about 10% of spend.
Fast path and step-boundary fan-out
At each step boundary, one call asks every relevant purpose. Low-risk reversible next steps can run without a model turn, and read-only steps run while the model thinks.
Channels and the front door
Where people reach the crew: the web, WhatsApp, Slack, Teams, the phone, and outside contacts.
AI receptionist for unknown senders
With public inbound on, an unknown WhatsApp or SMS sender or caller gets a fixed-template reply and each message becomes a Flow signal; asking for a person notifies the owner.
The crew runs only on the customer's own number, answers clear consumer-assistant use with one scoped reply, discloses it is AI, and keeps WhatsApp content out of other orgs and shared outputs.
Completion reports, digest and alert fallback
Completion reports follow a per-conversation level, a daily digest batches status, quiet hours hold non-urgent messages, and a critical alert open 15 minutes also goes by SMS or email.
Status questions get answers from runtime state through templates, with no model, and a receipt goes out before any model call. A separate job lane serves what a person waits on, and casual consumer chat starts no work.
Governed outreach to outside people
Workers contact prospects, customers and partners only through one gate that checks consent, limits, sending hours and content, and asks a person before a first contact.
Many workers on one WhatsApp number
One business number serves the team: each worker signs its messages, the operator addresses workers by name, replies stick to the last speaker, updates batch under Meta pacing, and a worker can get its own number.
Accounts and connections
Your own computers, subscriptions, keys and connected tools.
A person or org connects its own provider API keys after one free read-only check, and sign-in tokens or credential files are refused before anything is stored.
Native connectors and signed webhooks link GitHub, GitLab, Jira, Linear, Slack, Teams, Notion and Confluence for both worker actions and the Timeline.
Crewshal inside Claude and ChatGPT
Crewshal is an OAuth remote MCP server packaged as connectors and plugins, and a worker can fire the requester's own Claude routine or ChatGPT agent trigger after approval.
Guided setup and launch readiness
Sign-up walks the owner through readiness, preferences and connections, pairs their computer and subscription, names the CEO and reports what blocks the first task.
Own computer by default, one command
A person connects their own computer with one pre-approved command; the official Claude Code and Codex CLIs run there under the person's own login, which the platform never opens.
Plans, entitlements and credits
An org's plan sets its limits and credits. Credits pay for calls on platform keys, calls on the org's own accounts use no credits, and trial calls also need the pooled allowance.
The worker's own computer
The separate autonomous worker offering: its own private computer, accounts, browser and checks.
Worker's private persistent computer
An admin issues a worker its own Linux computer that persists like a laptop until revoked, offboarded or expired, with full control inside its guest, an approved egress allowlist and isolation at the guest boundary (today a Docker container for a single tenant).
Device-code sign-in for the worker's tools
An admin signs the worker's gh or Codex in on its computer with the vendor's device code, which reaches only the admin and is never stored; the doctor signs a wrong account out.
One approval by a second person for the computer and its sites
One approval by a second person (an owner, or an admin who did not issue the computer) binds the computer's install manifest and allowed hosts; onboarding runs as a resumable job with a doctor, snapshot and restore, and each added site needs another such approval.
Lab infrastructure on the operator's Docker host, with a forge, a mail server and a lab CA, where a worker proves self-onboarding with no outside account.
A root service on the host runs sandbox commands under signed, bounded grants from an issuer. It listens on a private socket and keeps a journal to clean up the Docker objects it owns.
Passwords, codes and captured tokens are stored as references; the platform fills or stages them into the guest and masks output, so no value reaches events, logs or model context.
Timeline
An automatic feed of what each person, outside agent, AI worker or automation did across connected systems. It needs no AI worker, and nobody reports into it.
Each webhook, collector and token shows current, late, stopped, never or unscheduled from recorded facts, and a late source or partial read marks the feed incomplete.
Credit the mover, at the action's time
A ticket move counts only for the mover the board history names, never the holder, and late or redelivered GitHub, Jira, Slack and Notion webhooks take their action's time, up to three days back.
When a webhook and collectors record the same GitHub change, the feed shows it once with a count of sources, while every record stays in the audit log.
A separate collector reads GitHub and Jira through a person's own signed-in tools with read calls only. It can preview without sending, sends when the person runs a send or installs the service, and runs as a background service that survives restarts without copies.
Current state of outside boards
Collectors report the current status of items on outside boards; a status change becomes one entry and feeds Today and Blockers in formats.
Formats over the timeline, standup included
Templates with slots turn timeline entries and state facts into a daily standup and other reports, copied as text or Markdown or posted to Slack on a schedule.
Data and trust
What is stored, who can read it, how secrets stay out of view, and how data is erased or taken away.
Append-only, hash-chained audit log
Every event is appended to a per-org hash chain that a database trigger keeps from change or deletion, with secrets redacted first.
Erasure by tombstone, log untouched
Owner-only calls erase an organization or a person in one transaction; the person becomes a tombstone and audit rows point at it, so the log is never edited.
Commits, PR text, comments, messages and reports go through rule-based checks (length, structure, banned phrases, attribution, secrets) and a style check against examples a person approved. Failing text gets a limited number of rewrites.
A written position lists stored data, subprocessors, each control with its code path and status, and known gaps, plus a register of infrastructure accounts and owners.
Only the operator's own words instruct
Only a verified sender's own lines are instructions; forwarded, quoted, fetched, partner and guest-command text is data, hidden characters are stripped, and actions after instruction-shaped content wait for a person.
An org can opt in to a read-only public status page that is redacted and aggregated with small-number suppression.
Running the platform
How the service stays healthy, recovers from failures and releases safely.
Only the process that holds a Postgres advisory lock runs background work. Other processes serve requests as warm standbys.
Changelog from commit trailers
The public changelog and its RSS feed are built from one-line commit trailers. A build fails when a commit has had no note for 7 days.
A local index says where X is, and traces an error, stack, URL or UI text to its code path and tests. Live docs get features from the code, and a daily check files doc drift.
Gated release with append-only migrations
One script builds the committed head, migrates in a throwaway container, swaps the image, checks health gates and rolls back on failure. Released migrations never change.
Health components and incident watch
Each module registers its own health checks, and /_health never calls a paid provider. A required component that fails past a grace period is filed once as an incident.
The build stage owner reads the CI result of each watched branch head through the gateway. A red head opens one urgent incident, and a green head records the recovery time.
Measuring AI work
The terms behind our public benchmarks: routing, caching, effort, intervals and inference cost, each shown on measured data.
Agent harness
What is an agent harness? The code around the model, measured
Agent harness is the code that wraps a language model and turns it into an agent. The model answers; the harness runs the loop, offers tools, builds prompts, keeps memory, manages context and checks the result. Two systems can share one model and still differ in time and tokens, so a benchmark scores a system, not a model.
Benchmark saturation
Benchmark saturation: when every model scores 100%
Benchmark saturation means most or all models score near the top of a test, so scores stop telling them apart. This is the ceiling effect. A perfect score means every observed call passed. It does not prove equal ability. Our sets hit ceilings while time, tokens and calculated list-price cost still differed.
Claude Code hook
Claude Code hooks, explained with a measured Stop hook
A Claude Code hook runs a command at an event, such as a tool call or session stop. Our Stop hook blocked finishing and sent standard-error feedback with exit code 2.
CLAUDE.md
What is CLAUDE.md? Agent memory files, measured
CLAUDE.md is a Markdown file in your project that Claude Code reads into its context at the start of every session. It holds facts the code cannot show, such as a team decision or the current test command, and it helped most there in our tests. Codex CLI reads AGENTS.md for the same job, but we measured Claude Code only.
CLI context tax
Tokens per call and the CLI context tax
Tokens per call is the number of input and output tokens one model request uses, and the CLI context tax is the part of the input that a coding CLI such as Claude Code or the Codex CLI adds on its own: its system prompt, tool definitions and environment context, sent with every request before your prompt. You pay for these tokens (or spend subscription quota on them), and the model must read them, even when your question is one line.
Context engineering
What is context engineering? What to put in the context, measured
Context engineering chooses what enters a model's context window on each call: instructions, memory, tools, retrieved files and history. It covers more than wording. In our tests, curated memory improved team-knowledge checks. Haiku used a stale command less often with consolidated notes than raw notes.
Cost per correct answer
Cost per correct answer: the LLM price that counts failures
Cost per correct answer is the total cost of every call, failed calls included, divided by the number of attempts that pass a check. An attempt can use one call or a full agent session. A lower token price can cost more per pass when a model writes more tokens or fails more often.
Eval sample size
How many runs does an LLM eval need? Sample size, with real intervals
Eval sample size counts the units behind a rate: tasks, attempts or checks. Choose independent cases for the precision you need. A narrow 95% interval does not guarantee power to detect a gap between systems.
Inference gateway
Inference gateways and OpenRouter, explained
An inference gateway is a service that puts many model providers behind one API: you send a request for a model to the gateway, and it forwards the request to one of the providers that serve that model, then returns the answer and one bill. OpenRouter is the best-known public example. A gateway gives you one key, one schema and fallback between providers; it can add a fee and a network hop, and it routes between providers of a model, not between models for a task.
Inference provider
What is an inference provider? Bedrock, Vertex, Azure and first-party prices
Inference provider is a company that sells API access to an AI model: the vendor, a cloud, a specialist host or a gateway. In our 2026-10-06 snapshot, Bedrock listed Claude Sonnet 5.5 at the same price as the Anthropic API: $2 input and $10 output per million tokens. For seven open-weight models, standard-tier price spreads ranged from 1.7x to 12.6x (a calculation). The largest spread covered 15 providers.
LLM as judge
LLM as judge: how blind review works, and where it fails
LLM as judge (or LLM-as-a-judge) is an evaluation method in which one or more language models grade or compare outputs, such as answers, essays or code changes, in place of human reviewers or fixed tests. In a blind review, the judge does not know which output came from which source, and in a pairwise setup it sees both outputs in both orders, so that labels and position cannot decide the verdict. It scales to tasks that have no exact answer, but its verdicts carry the judge's own biases.
LLM nondeterminism
Why the same prompt gives different answers: LLM nondeterminism, measured
LLM nondeterminism means a model can answer the same prompt differently. General causes include how the model samples each next token and how batching on shared servers can change tiny numeric details. In 90 repeated calls, the wording changed in 5 of 9 prompt-and-model pairs and the answer in 3 of 9.
LLM router
What is an LLM router?
An LLM router is the component that decides, before a request runs, which language model should answer it, at which reasoning effort and with how much context. The router can be a set of rules in code, a small model trained to make the decision, or a general LLM that you ask to choose. A good router sends easy work to cheap, fast models and hard work to strong ones, and adds less delay and cost than it saves.
McNemar test
The McNemar test: compare two models on the same cases
McNemar test, also called McNemar's test, compares two models on the same cases, such as two LLMs or two classifiers. It reads cases where exactly one model was right; the others cannot separate them. The exact version is a binomial test on those disagreements with a probability of 0.5.
Memory consolidation
Memory consolidation (dreaming) for AI agents, measured
Memory consolidation, also called dreaming, is a pass in which a model rewrites saved agent notes into one shorter file. It aims to keep current facts and drop stale notes and one-off events.
Open-weight model
Open-weight models: why the same model has many prices
Open-weight models are AI models whose trained weights are public, so anyone who accepts the license can download and run them. Many companies host the same open model. Our snapshot shows a price spread up to 12.6x (calculation, 15 standard-tier providers for DeepSeek V4 Flash 0423). All 15 closed models with 2 or more providers had one standard-tier price across those providers. This applies to our snapshot, not every closed model.
p95 latency
p95 latency, explained: median, tail and range for LLM calls
p95 latency marks the time at or below which about 95% of calls fall. About 1 call in 20 is slower. Small samples and tied times can change that share. It shows slow calls that the median (p50) hides. In 82 recorded routing calls, Claude Sonnet 5.5 through Claude Code had a median of 2.60 s and a p95 of 4.30 s (range 1.993 to 5.583 s).
Pareto frontier
The Pareto frontier: reading LLM cost against quality
Pareto frontier is the set of options that no rival matches or beats on every axis and beats on at least one. For large language models (LLMs), the axes are cost and quality, and sometimes speed. The frontier is your short list.
pass@k
pass@k explained: pass@1, pass@k and pass^k with real runs
pass@k is the chance that at least one of k tries at a task passes its check. pass@1 is the chance that one try passes, and pass^k (with a caret) is the chance that all k tries pass. So pass@k rewards a model that is sometimes right, and pass^k rewards one that is right every time.
Price per million tokens
LLM pricing per million tokens, explained with real token counts
Price per million tokens is an LLM vendor’s dollar rate per 1,000,000 tokens. Input, output and cached tokens have separate rates. Multiply each token count by its rate, divide by 1,000,000, then add the results. Our stored Sonnet 5.5 rates per million tokens are $2 for input, $10 for output and $0.20 for cache reads.
Prompt caching
Prompt caching, explained with measured sessions
Prompt caching is a feature of model APIs that stores the processed form of a prompt prefix, so that a later request that starts with the same prefix reads it from the cache instead of processing it again. A cache read is billed at a fraction of the normal input price, and the first request that writes the prefix may cost more than normal input. Caching saves money when the same long context, such as a system prompt, tool definitions or a document, is sent many times.
Quantization
Quantization and cheap inference, explained
Quantization is the practice of storing and running a model's weights (and sometimes its activations and cache) in fewer bits, such as 8-bit (fp8) or 4-bit (fp4) numbers instead of 16-bit (bf16 or fp16). A quantized model needs less memory and less compute per token, so a provider can serve it faster or cheaper on the same hardware. The trade is accuracy: a lower precision can change the model's outputs, by a little or a lot, depending on the model, the method and the task.
Reading an AI benchmark honestly
How to read AI benchmarks honestly
Reading an AI benchmark honestly means checking what was measured, on how many cases, under which configuration and with what uncertainty before you accept a ranking. A trustworthy result states its sample size, gives an interval for every rate and a range for every time, counts every failed attempt, separates measured runs from calculations, and names the setup (model, effort, route, harness) behind each number. A table that hides any of these can make a tie look like a win.
Reasoning effort
What is reasoning effort?
Reasoning effort is a setting on a reasoning model that controls how much internal work, usually in the form of hidden thinking tokens, the model does before it gives its answer. Typical levels are low, medium and high. A higher effort can help on hard problems, but it costs more output tokens, more time and more money per call, so the useful question is whether a higher effort changes the result on your tasks.
Reasoning tokens
Reasoning tokens, explained: the hidden output you pay for
Reasoning tokens are the tokens a reasoning model writes to think before it writes its visible answer. Vendors generally bill them as output tokens, and a tool may report their count without showing their text. A short answer can hide a long chain of thought that you pay for.
Strict grading
Strict grading and format misses: when a right answer fails
Strict grading requires the whole reply to pass the declared check. A right answer inside a code fence can fail. Lenient grading can count it after extraction. On our hard set, 139/152 passed strictly (95% interval 85.9% to 94.9%). With format misses included, 144/152 passed (90.0% to 97.3%). Here, "correct" means it passed the task validator, not a proof for all possible inputs.
SWE-bench Verified
SWE-bench Verified, explained
SWE-bench Verified is a benchmark of 500 real GitHub issues from popular Python repositories, each checked by people to have a clear problem statement and a fair test. A coding agent gets the repository and the issue text, writes a patch, and the patch counts as resolved only when the project's hidden tests that the real fix made pass now pass, and the tests that passed before still pass. The score is the share of instances resolved.
Time to first token
Time to first token (TTFT), explained with CLI and API timings
Time to first token (TTFT) is the wait from sending a model request to receiving its first reply token. Start-up, processing, queues and the network can add delay. We measure first useful output, a different clock from raw TTFT.
Wilson confidence interval
Wilson confidence intervals for AI benchmarks
A Wilson confidence interval (also called the Wilson score interval) is a range around a measured pass rate that shows which true pass rates are consistent with the result, given the number of trials. For a benchmark result of k passes in n attempts, the 95% Wilson interval is the range that would contain the true rate in about 95% of repeated experiments. Unlike the simple "plus or minus" interval, it stays inside 0% to 100% and stays honest at small n and at rates near 0% or 100%.