OpenTelemetry for AI: What to Trace Without Logging Everything

OpenTelemetry can connect an AI journey across an interface, service, model, database and external tool. Its purpose is not to record everything. It should answer an operational question quickly: why did this action fail, where did latency appear or which version degraded the outcome?

Start with one critical workflow and a decision plan. Instrumenting the entire architecture before knowing what you are looking for creates cost, noise and privacy risk.

What OpenTelemetry provides

OpenTelemetry is an open framework for producing and transporting observability signals, including traces, metrics and logs. A trace represents an operation. Its spans describe steps and relationships.

In an AI workflow, one trace may connect the user request, context retrieval, model call, tool use, human review, final write and a business outcome observed later.

OpenTelemetry does not provide quality evaluation, a security policy or a correction interface by itself. It provides a correlation structure those capabilities can use.

Start with four questions

1. Which decision will this signal change?
2. Which journey requires end-to-end correlation?
3. Which data is necessary for diagnosis without exposing content?
4. Who may access the signal and for how long?

If an attribute has no associated decision, do not collect it by default.

A minimum AI trace schema

Technical identity

Trace ID, span ID and, where needed, a pseudonymous journey identifier. Avoid an email address or name as the correlation key.

Version

Application, workflow, prompt or configuration version, provider and model. Versioning supports before/after comparison and rollback.

Performance

Total latency, latency by step, external wait time, retries and approximate input/output size.

Cost

Tokens or billed units, estimated cost and call count. Later, measure cost per business outcome rather than only per request.

State

Technical success, error, timeout, refusal, fallback or human review. A successful model call may still produce an invalid business result, so keep those states distinct.

Linked quality

Evaluation identifier, criterion-based score, human correction and observed outcome. The trace carries the link; it does not replace the evaluation protocol.

Should prompts and responses be recorded?

Not by default. Content may contain personal data, secrets, health information, intellectual property or hostile instructions. It also increases cost and access surface.

Prefer template hash, prompt version, input category, document type, approximate size, review state, error code and aggregated outcome.

If content is essential for a quality sample, use a restricted environment, documented purpose, short retention, controlled access and redaction. Verify that content remains disabled elsewhere.

Example: case-file assistant

A user requests a draft based on three documents.

The trace shows document retrieval, rights filtering, generation, citation verification, human review and export. Citation verification times out. The system falls back to an unverified draft and blocks automatic export.

The useful trace does not need the documents themselves. It retains authorized identifiers, versions, durations, control state and fallback decision.

Connect technical and business outcomes

Technical observability answers “did the service run?” Product observability must also answer “did the user obtain the outcome?”

Link the trace to a business event without copying content: proposal approved, action completed, case resolved or correction requested. Correlation can be delayed and pseudonymized.

A two-hour pilot

1. choose a workflow where failure is costly;
2. draw five to eight steps;
3. define a correlation identifier;
4. add version, latency, state and cost;
5. simulate an error and fallback;
6. verify that no unexpected sensitive data appears;
7. write down the decision the signal will support.

Common mistakes

• tracing every call but no outcome;
• storing full prompts “for later”;
• mixing technical failure and poor quality;
• failing to version prompts and configuration;
• retaining signals indefinitely;
• granting broad access;
• never testing instrumentation itself;
• treating experimental semantic attributes as an immutable contract.

OpenTelemetry's GenAI semantic conventions continue to evolve. Pin the version in use and add a translation layer instead of scattering attribute names throughout the codebase.

FAQ

Does OpenTelemetry replace Sentry?

Not necessarily. Sentry focuses on the application error diagnosis experience. OpenTelemetry standardizes signal production and correlation. They may complement one another.

Is OpenTelemetry enough to evaluate a model?

No. It connects traces with evaluations. You still need cases, criteria, thresholds and human review.

Should a prototype use it immediately?

Only when a critical journey already crosses multiple components or trust requirements justify it. A small structured log may be enough at the earliest stage.

Next step

Apply the OpenTelemetry AI Workflow Checklist to one journey. If you cannot name the decision associated with a signal, remove it. For a schema review, email contact@yumea.fr with “OTEL” in the subject.

Sources

• OpenTelemetry, Semantic conventions for generative AI systems.
• OpenTelemetry documentation for traces, metrics and logs.
• CNIL recommendations on securing AI system development.