5.8 KiB
Final hypothesis evaluation and diagnosis v1
DiagnosisBuilder implements architecture phases 4A and 4B. The completed
investigation is evaluated as a whole by one bounded, provider-neutral model
operation. It produces a traceable diagnosis or controlled human-review output.
It does not produce a Change Request, patch, SQL, build or lab execution.
docs/architecture.md remains authoritative and unchanged.
from viano_bug_analysis.analysis import DiagnosisBuilder, DiagnosisConfig
builder = DiagnosisBuilder(store, gateway,
DiagnosisConfig.load("config/diagnosis.local.toml"), router=router)
result = await builder.build(case_id, package_id)
Input requires a completed current-revision investigation, unchanged canonical raw case inputs and fully terminal budget accounts with settled actions. Missing prior stages are rejected; they are never silently executed. The context includes canonical case, accepted interpretation, full hypothesis end states/questions, budget snapshot and current-revision stored evidence with contents/provenance. No investigation action or credit adjustment is performed in this phase.
Model decision and deterministic gates
diagnosis-output-v1 has status, selected hypothesis, root cause (summary,
technical description and evidence IDs), qualitative confidence, exactly one
final evaluation per hypothesis, countercheck adequacy/reason/evidence, remaining
uncertainty, final reason and recommended human-review actions.
A confirmed result requires supported/strongly_supported confidence, a supported selected hypothesis that was not factually rejected, nonempty stored root-cause findings also cited by its evaluation, and at least one alternative with a reasoned countercheck citing actual shared findings. Evidence provenance does not restrict which hypotheses may use it. See Decision v2 for explicit scope limits, contradictions and retry feedback. Unknown/duplicate/missing hypothesis IDs and fabricated or empty support references are rejected. Loop confidence and its countercheck marker alone never authorize a diagnosis. Partial/empty snapshots are not proof of absence. The model must distinguish intended external behavior from local implementation, and human assumptions from observations.
These checks validate provenance and structural support; they cannot prove that an LLM correctly inferred causality or that a stated countercheck is semantically sufficient. Actual model reasoning must be evaluated against known cases later. The offline tests use simulated semantic conclusions, not a seeded known fix.
Inconclusive results require null selected hypothesis/root cause and actionable human-review recommendations. They retain every hypothesis end state and open question. A deterministic Markdown summary adds no independent conclusions.
Persistence, stopping and lifecycle
Frozen context is persisted under analysis/diagnosis/NNNN/context.json, bound
by SHA256 and an explicit start checkpoint. Model calls use the existing durable
LLM runner, ordered provider failover and configured retries, without sleeping a
worker. max_invalid_responses defaults to 3 and is frozen per revision; omitted
config on resume adopts the stored value. Explicit policy changes are rejected.
Each build dispatch performs at most one eligible provider round. Invalid model
answers return REJECTED and append an audit error; another dispatch starts the
next bounded attempt. No partial diagnosis or budget change is published.
After the configured number of rejected final answers, a deterministic
inconclusive report records that final validation failed, without alleging a
technical root cause. Rejected raw replies remain in the package.
Transport/provider configuration failures return BLOCKED, preserving all
budgets and history; they are not evidence of an inconclusive investigation.
After lifecycle resume, an explicit stable retry_id can reopen an exhausted or
configuration-failed transport operation. It cannot bypass ordinary backoff or
reset invalid-response counts. Corrected configuration gets a distinct durable
operation. Review actions are recommendations, not dispatched external tests.
Successful publication atomically appends diagnosis_result.json and
diagnosis_result.md. The JSON wraps identity/context/model-call bindings and the
strict diagnosis evaluation payload. An inconclusive result also appends
failure_report.json, moves to FAILED_TO_ANALYZE, and sets routing to review.
A confirmed diagnosis stays BUILDING_DIAGNOSIS: OUTPUT_READY and outbox success
must wait for the future Change Request Builder and output validation.
Manifest pointers are diagnosis, diagnosis_summary, diagnosis_context and
failure_report. No coding request is generated.
Replays verify the context, checkpoint hash, identity, exact saved model request, model output, schema, deterministic summary and manifest pointers. A crash after model publication resumes the same accepted call without a second model request. Missing or inconsistent pointers never silently rebuild completed output. CaseAssembler excludes only checkpoint-declared diagnosis artifacts from raw inputs. Unknown files remain preserved and participate in freshness checking. All original input bytes remain intact; package generation advances normally, package identity and analysis revision remain stable.
DiagnosisLimits has adjustable safety defaults of 2 MiB context and 4 MiB JSON
artifacts/summaries; these are not package-format boundaries. Align these limits
with the gateway request and package file limits. Oversized contexts are rejected
explicitly, never silently truncated. Config example: config/diagnosis.example.toml.
The next milestone is Change Request Builder with deterministic bugfix policy and final output validation. A scheduler/daemon and web feedback UI remain separate.