2.7 KiB
LLM failure diagnostics
Failed response processing now retains optional outcome.metadata and
outcome.diagnostics in the existing v1 llm/operations/.../round-NNNN.json
records. Old records without these fields remain valid and replay without API
calls. Previously discarded details cannot be recovered retrospectively.
For incomplete replies, diagnostics preserve:
- The API finish reason, restricted to known values; absent/unrecognized values
are labeled
missing/unknownrather than copying arbitrary provider text. - Boolean refusal/tool-call indicators, not their text or arguments.
- Input/output tokens and reasoning tokens when supplied by the API.
- Configured output-token limit, received response bytes and output character count.
Metadata records the configured provider/model, prompt version, elapsed seconds, known usage, previous unavailable-provider attempts, and estimated cost only when usage and operator-configured prices are available. Missing/invalid token values remain unknown, not zero. API output token usage may include reasoning, depending on the provider; do not infer reasoning usage when the API omits that field.
JSON parsing errors retain available envelope details. Schema rejection retains
usage and a safe validation keyword, without storing response contents or the
validation exception message. Diagnostic fields have their own allowlisted JSON
schema (llm-failure-diagnostics-v1.schema.json). No headers, API keys, provider
error bodies, refusal text, tool arguments or partial completion text are added.
The same safe details are visible in structured service llm_round log events.
Known usage/cost from failed replies now contributes to existing telemetry
counters. The connection-test CLI includes diagnostic/metadata fields on errors.
Existing retry, failover, validation limits, lifecycle and hypothesis budgets are
unchanged. Successful reply metadata and response contracts remain unchanged.
A finish_reason = length indicates an output-budget stop reported by the API;
compare the output token count with max_output_tokens, and inspect reasoning
usage if available. content_filter, refusal/tool-call indicators, or missing
finish reasons describe different failure conditions. An incomplete response
alone does not prove a transport timeout. Transport/availability failures still
use their existing codes and HTTP attempt records; usage may be unavailable.
Tests use local HTTP fixtures and model fixtures. They cover truncation, refusal/tool calls, malformed usage, unknown finish reasons, JSON/schema failures, usage persistence, replay without network calls, telemetry counters/logs and removal of free provider text. No paid model request is needed.