Build an Error Budget for an AI Feature

Calling an AI system “reliable” does not create a decision rule. A punctuation error, fabricated source and unauthorized irreversible action cannot share one average. An error budget translates accepted risk into observable thresholds and operational responses.

Start with severity

A practical taxonomy can use four levels:

• Minor: presentation defect without decision impact.
• Significant: incomplete information or correction required before use.
• Major: false recommendation, exposed sensitive data or incorrect but reversible business action.
• Critical: material harm, irreversible action, affected right or missing appeal.

Severity should reflect actual effect, not only technical deviation.

Define the denominator

A rate without a precise denominator is useless. Is it measured per response, decision, executed action, user or case? For an agent with many tool calls, message count may hide the unit carrying risk.

Choose the unit closest to the promise and impact: decisions prepared, payments proposed or cases closed.

Write thresholds and consequences

An error budget should state:

• maximum rate by severity;
• observation window and minimum volume;
• segments monitored separately;
• rule for any critical case;
• decision owner;
• action attached to each threshold.

An example response ladder is: below threshold, monitor; near threshold, limit rollout; above threshold, return to assistance mode; critical event, suspend the capability and review affected actions.

Numbers must come from context. There is no universally safe percentage.

Add detectability and reversibility

Two features with the same error rate can carry different risk. An error visible before sending and easy to correct differs from a silent error discovered after a decision.

Weight the budget with:

• probability of detection before effect;
• detection delay;
• ease of reversal;
• exposed population;
• quality of appeal.

This prevents the budget from becoming a simple accuracy target.

Connect evaluation and production

Offline evaluations check a version before release. Production reveals real data, unexpected behavior and context change. Use the same severity vocabulary in both environments.

Maintain a critical-case set that blocks promotion, then sample production with calibrated human review.

Avoid three traps

The reassuring average

A strong global score can hide a rare but dangerous segment.

A budget without action

A threshold that changes nothing is only a dashboard.

A permanent budget

Review thresholds when use, authority, population or model changes.

Fact, inference and hypothesis

Direct fact: the same error rate creates different risk depending on impact, detectability and reversibility.

Inference: a shared taxonomy connects product, evaluations, incidents and deployment decisions.

Hypothesis to test: an error budget with predefined actions shortens the time between degradation and capability limitation.

FAQ

Does an error budget deliberately allow errors?

It recognizes that no system is perfect and makes explicit what is tolerable, what requires correction and what requires shutdown.

What sample size is needed?

It depends on frequency and risk. Rare critical cases need targeted tests, not only a large average.

Can one threshold cover the whole product?

No. Separate actions, severities and segments with materially different impact.

Next step

Use the error-budget calculator on one capability and attach a decision to every threshold. For a review of severity and stopping rules, email contact@yumea.fr with the subject “ERROR BUDGET.”