45 lines
2.7 KiB
Markdown
45 lines
2.7 KiB
Markdown
# LLM failure diagnostics
|
|
|
|
Failed response processing now retains optional `outcome.metadata` and
|
|
`outcome.diagnostics` in the existing v1 `llm/operations/.../round-NNNN.json`
|
|
records. Old records without these fields remain valid and replay without API
|
|
calls. Previously discarded details cannot be recovered retrospectively.
|
|
|
|
For incomplete replies, diagnostics preserve:
|
|
|
|
- The API finish reason, restricted to known values; absent/unrecognized values
|
|
are labeled `missing`/`unknown` rather than copying arbitrary provider text.
|
|
- Boolean refusal/tool-call indicators, not their text or arguments.
|
|
- Input/output tokens and reasoning tokens when supplied by the API.
|
|
- Configured output-token limit, received response bytes and output character count.
|
|
|
|
Metadata records the configured provider/model, prompt version, elapsed seconds,
|
|
known usage, previous unavailable-provider attempts, and estimated cost only when
|
|
usage and operator-configured prices are available. Missing/invalid token values
|
|
remain unknown, not zero. API output token usage may include reasoning, depending
|
|
on the provider; do not infer reasoning usage when the API omits that field.
|
|
|
|
JSON parsing errors retain available envelope details. Schema rejection retains
|
|
usage and a safe validation keyword, without storing response contents or the
|
|
validation exception message. Diagnostic fields have their own allowlisted JSON
|
|
schema (`llm-failure-diagnostics-v1.schema.json`). No headers, API keys, provider
|
|
error bodies, refusal text, tool arguments or partial completion text are added.
|
|
|
|
The same safe details are visible in structured service `llm_round` log events.
|
|
Known usage/cost from failed replies now contributes to existing telemetry
|
|
counters. The connection-test CLI includes diagnostic/metadata fields on errors.
|
|
Existing retry, failover, validation limits, lifecycle and hypothesis budgets are
|
|
unchanged. Successful reply metadata and response contracts remain unchanged.
|
|
|
|
A `finish_reason = length` indicates an output-budget stop reported by the API;
|
|
compare the output token count with `max_output_tokens`, and inspect reasoning
|
|
usage if available. `content_filter`, refusal/tool-call indicators, or missing
|
|
finish reasons describe different failure conditions. An incomplete response
|
|
alone does not prove a transport timeout. Transport/availability failures still
|
|
use their existing codes and HTTP attempt records; usage may be unavailable.
|
|
|
|
Tests use local HTTP fixtures and model fixtures. They cover truncation,
|
|
refusal/tool calls, malformed usage, unknown finish reasons, JSON/schema failures,
|
|
usage persistence, replay without network calls, telemetry counters/logs and
|
|
removal of free provider text. No paid model request is needed.
|