149 lines
8.6 KiB
Markdown
149 lines
8.6 KiB
Markdown
# Case Interpretation v1
|
||
|
||
This implements architecture sections 12–14: the first domain-specific LLM phase.
|
||
It compares supplied observations and intended behavior, records uncertainty and
|
||
prepares broad investigation directions. It implements no hypotheses, root-cause
|
||
diagnosis, Investigation Tools, patches or Change Requests. Architecture.md is unchanged.
|
||
|
||
## API and prerequisites
|
||
|
||
```python
|
||
from viano_bug_analysis.analysis import CaseInterpreter
|
||
from viano_bug_analysis.llm import GatewayConfig, LLMGateway
|
||
|
||
gateway = LLMGateway(GatewayConfig.load("config/llm.local.toml"))
|
||
interpreter = CaseInterpreter(store, gateway)
|
||
result = await interpreter.interpret(case_id, package_id)
|
||
```
|
||
|
||
CaseBuilder must already have confirmed the current analysis revision. The interpreter
|
||
verifies its checkpoint, manifest pointer, identities and current raw-input fingerprint
|
||
before preparing a request. It does not silently build an absent case. Changed raw inputs
|
||
require a new analysis revision. No identity or state is inferred from the ZIP filename.
|
||
|
||
The API returns COMPLETED, RETRY_PENDING, DEFERRED, BLOCKED or ERROR. Storage/input problems
|
||
raise InterpretationError with a diagnostic code; concurrent dispatches raise LockBusy.
|
||
The async API supports independent cases while a provider awaits its response. It is not
|
||
a continuously running worker daemon; the caller schedules due retries.
|
||
|
||
## Versioned prompt and bounded context
|
||
|
||
`llm/prompts/case_interpretation_v1.txt` is independent of provider and model. The domain
|
||
PromptBuilder deterministically serializes a projection of the canonical case. The
|
||
system prompt treats reports, comments, filenames, URLs and source excerpts as untrusted
|
||
data. It forbids root-cause assertions, new causal hypotheses, implementation guidance
|
||
and tool execution. There is no tool/command execution interface in this phase.
|
||
|
||
The context contains a reference catalogue whose entries retain their canonical JSON
|
||
pointers and original data/provenance:
|
||
|
||
| IDs | Contents |
|
||
| --- | --- |
|
||
| O / C / T / E / F / R | Existing observations, comments, tester hypotheses, expectations, facts and reproductions |
|
||
| B | Report titles and summaries |
|
||
| S | Explicit reproduction steps |
|
||
| N | Canonical entities |
|
||
| D | Data-quality issues |
|
||
| Q0001 | Expected-behavior completeness |
|
||
| X | Metadata for unread runtime, source, database and research references |
|
||
|
||
Aliases B/S/N/D/Q/X are scoped to this exact context and case hash. Existing record IDs
|
||
remain unchanged. Large raw payloads, original archives, arbitrary unclassified files
|
||
and source/database file contents are not loaded into the prompt. Facts represented
|
||
only by value_ref remain unread references. Inline facts and explicit expectations
|
||
retain their source authors/timestamps and any original-data or external-source provenance.
|
||
|
||
InterpretationLimits defaults are 8 MiB for the canonical case, 512 KiB context, 2048
|
||
catalogue entries, and 2 MiB each for accepted output and Markdown. They are configurable
|
||
operational safeguards, not permanent format bounds. Gateway/package limits also apply.
|
||
For cases built with custom CaseBuilderLimits, pass those as case_limits to CaseInterpreter.
|
||
|
||
## Output contract and validation
|
||
|
||
`case-interpretation-output-v1.schema.json` contains precisely these required sections:
|
||
observed_behavior, expected_behavior, tester_assumptions, confirmed_technical_facts,
|
||
behavior_gap, first_known_divergence, failure_stage, contradictions, unknowns,
|
||
relevant_entities, investigation_direction and interpretation_summary.
|
||
|
||
Statements contain text and source_ids. Gaps, divergence and failure stage may be null;
|
||
collections may be empty. Extra keys—including identities, root_cause, hypotheses, patch
|
||
or tool requests—are rejected. The controller supplies identities, analysis revision,
|
||
case/context hashes, actual LLM outcome reference and metadata in the persisted wrapper.
|
||
The model cannot overwrite those authoritative fields.
|
||
|
||
After gateway schema validation, the interpreter validates the whole response against
|
||
the actual reference catalogue:
|
||
|
||
- Every cited source exists, and every relevant entity is a real unique N alias.
|
||
- Intended behavior cites E records only; tester assumptions cite T/C records.
|
||
- Confirmed technical facts cite only inline F facts. Tester assumptions and external
|
||
expectations cannot become local technical proof.
|
||
- A gap/divergence cites both intended behavior and an observation/report/comment;
|
||
a reproduction step alone does not establish a divergence.
|
||
- Contradictions need at least two distinct supplied sources and cannot use unread payloads.
|
||
- Incomplete/missing/conflicting intended behavior remains an unknown citing Q0001.
|
||
With no E records, an invented expectation or justified gap cannot pass validation.
|
||
|
||
Completeness and tester/external_research/combined provenance are copied from the case,
|
||
not decided by the model. External intended behavior remains distinct from local
|
||
implementation evidence. An invalid response is rejected atomically: no interpretation
|
||
pointer or summary is published. Its raw gateway outcome and a separate rejection report
|
||
remain available for audit. The phase stays INTERPRETING_CASE; invalid output is not
|
||
misclassified as unreachable infrastructure or exhausted diagnostic investigation.
|
||
|
||
Schema/reference validation cannot prove that free-text paraphrases are true, detect
|
||
every hidden causal assertion, or prevent all semantic prompt injection. The prompt and
|
||
typed source constraints bound the task; actual model adherence must be evaluated with
|
||
real providers and known cases. No heuristic word blacklist pretends to provide that proof.
|
||
|
||
## Durable lifecycle and package preservation
|
||
|
||
Normal progression is three atomic package generations after a built case:
|
||
|
||
1. Enter INTERPRETING_CASE and preserve context.json with a case-interpreting checkpoint.
|
||
2. CaseLLMRunner records a gateway round using the existing failover/retry policy.
|
||
3. Publish accepted case_interpretation.json and case_interpretation.md together with a
|
||
case-interpreted completion checkpoint and manifest pointers.
|
||
|
||
The completed phase stays INTERPRETING_CASE. The future Hypothesis Builder will explicitly
|
||
start GENERATING_HYPOTHESES. No later phase is executed in this milestone.
|
||
|
||
Phase artifacts live under `analysis/interpretations/NNNN/`: context.json, accepted JSON,
|
||
deterministic Markdown and immutable errors/<attempt-hash>.json reports. This phase-specific
|
||
layout allows later phases to append their artifacts without changing PackageWriter's
|
||
existing rule that analysis/revisions/NNNN namespaces are frozen once present. All artifact
|
||
files remain immutable and carry their analysis revision. Existing original/ data, old
|
||
analysis and unknown files remain byte-identical. package_id stays stable; only package_generation
|
||
advances within a revision.
|
||
|
||
Manifest pointers are interpretation_context, case_interpretation and
|
||
case_interpretation_summary. The JSON result is authoritative; the Markdown is a pure
|
||
escaped rendering with no additional interpretation. Both case and context SHA-256
|
||
bindings identify the exact canonical JSON serialization produced by CaseBuilder.
|
||
|
||
The default stable operation uses attempt_id="default". Retries share that ID, request
|
||
and configuration fingerprints. A completed phase is validated and reused independently
|
||
of a later provider/config change. A confirmed gateway result is also reused if a crash
|
||
occurred before phase publication; publication/index crashes are recovered through
|
||
checkpoint replay. Only an unconfirmed remote call may need repeating. Cancellation
|
||
releases operation locks and keeps confirmed preparation for resumption.
|
||
|
||
A deliberately changed provider configuration or a corrected response after rejection
|
||
can use a new attempt_id. Rejections stay in the same analysis revision without replacing
|
||
old records. An exhausted BLOCKED operation needs explicit lifecycle resumption and a
|
||
new attempt_id, as described in llm-gateway-v1.md. Underlying case changes still require
|
||
a new analysis revision rather than another attempt against a stale canonical case.
|
||
|
||
## Verification scope
|
||
|
||
Tests cover structured/reference rejection, missing expectations, provenance separation,
|
||
bounded context, immutable output, escaped summaries, retry/defer/block behavior,
|
||
portable replay, cancellation, concurrency and crash recovery. A real local HTTP server
|
||
exercises the bundled adapter with the domain schema and prompt. The optional BUG-20
|
||
regression now includes interpretation publication and byte-for-byte preservation of
|
||
the supplied packages using a fixture provider.
|
||
|
||
These are offline integration checks, not proof that a model can explain or solve BUG-20.
|
||
No real LLM endpoint, cloud credentials, live research or known fix is supplied in these
|
||
tests. That semantic end-to-end evaluation remains a later milestone.
|