96 lines
5.8 KiB
Markdown
96 lines
5.8 KiB
Markdown
# Final hypothesis evaluation and diagnosis v1
|
|
|
|
`DiagnosisBuilder` implements architecture phases 4A and 4B. The completed
|
|
investigation is evaluated as a whole by one bounded, provider-neutral model
|
|
operation. It produces a traceable diagnosis or controlled human-review output.
|
|
It does not produce a Change Request, patch, SQL, build or lab execution.
|
|
`docs/architecture.md` remains authoritative and unchanged.
|
|
|
|
```python
|
|
from viano_bug_analysis.analysis import DiagnosisBuilder, DiagnosisConfig
|
|
builder = DiagnosisBuilder(store, gateway,
|
|
DiagnosisConfig.load("config/diagnosis.local.toml"), router=router)
|
|
result = await builder.build(case_id, package_id)
|
|
```
|
|
|
|
Input requires a completed current-revision investigation, unchanged canonical
|
|
raw case inputs and fully terminal budget accounts with settled actions. Missing
|
|
prior stages are rejected; they are never silently executed. The context includes
|
|
canonical case, accepted interpretation, full hypothesis end states/questions,
|
|
budget snapshot and current-revision stored evidence with contents/provenance.
|
|
No investigation action or credit adjustment is performed in this phase.
|
|
|
|
## Model decision and deterministic gates
|
|
|
|
`diagnosis-output-v1` has status, selected hypothesis, root cause (summary,
|
|
technical description and evidence IDs), qualitative confidence, exactly one
|
|
final evaluation per hypothesis, countercheck adequacy/reason/evidence, remaining
|
|
uncertainty, final reason and recommended human-review actions.
|
|
|
|
A confirmed result requires supported/strongly_supported confidence, a supported
|
|
selected hypothesis that was not factually rejected, nonempty stored root-cause
|
|
findings also cited by its evaluation, and at least one alternative with a
|
|
reasoned countercheck citing actual shared findings. Evidence provenance does
|
|
not restrict which hypotheses may use it. See [Decision v2](diagnosis-decision-v2.md)
|
|
for explicit scope limits, contradictions and retry feedback. Unknown/duplicate/missing hypothesis IDs and fabricated
|
|
or empty support references are rejected. Loop confidence and its countercheck
|
|
marker alone never authorize a diagnosis. Partial/empty snapshots are not proof
|
|
of absence. The model must distinguish intended external behavior from local
|
|
implementation, and human assumptions from observations.
|
|
|
|
These checks validate provenance and structural support; they cannot prove that
|
|
an LLM correctly inferred causality or that a stated countercheck is semantically
|
|
sufficient. Actual model reasoning must be evaluated against known cases later.
|
|
The offline tests use simulated semantic conclusions, not a seeded known fix.
|
|
|
|
Inconclusive results require null selected hypothesis/root cause and actionable
|
|
human-review recommendations. They retain every hypothesis end state and open
|
|
question. A deterministic Markdown summary adds no independent conclusions.
|
|
|
|
## Persistence, stopping and lifecycle
|
|
|
|
Frozen context is persisted under `analysis/diagnosis/NNNN/context.json`, bound
|
|
by SHA256 and an explicit start checkpoint. Model calls use the existing durable
|
|
LLM runner, ordered provider failover and configured retries, without sleeping a
|
|
worker. `max_invalid_responses` defaults to 3 and is frozen per revision; omitted
|
|
config on resume adopts the stored value. Explicit policy changes are rejected.
|
|
Each `build` dispatch performs at most one eligible provider round. Invalid model
|
|
answers return `REJECTED` and append an audit error; another dispatch starts the
|
|
next bounded attempt. No partial diagnosis or budget change is published.
|
|
After the configured number of rejected final answers, a deterministic
|
|
inconclusive report records that final validation failed, without alleging a
|
|
technical root cause. Rejected raw replies remain in the package.
|
|
|
|
Transport/provider configuration failures return `BLOCKED`, preserving all
|
|
budgets and history; they are not evidence of an inconclusive investigation.
|
|
After lifecycle resume, an explicit stable `retry_id` can reopen an exhausted or
|
|
configuration-failed transport operation. It cannot bypass ordinary backoff or
|
|
reset invalid-response counts. Corrected configuration gets a distinct durable
|
|
operation. Review actions are recommendations, not dispatched external tests.
|
|
|
|
Successful publication atomically appends `diagnosis_result.json` and
|
|
`diagnosis_result.md`. The JSON wraps identity/context/model-call bindings and the
|
|
strict `diagnosis` evaluation payload. An inconclusive result also appends
|
|
`failure_report.json`, moves to `FAILED_TO_ANALYZE`, and sets routing to `review`.
|
|
A confirmed diagnosis stays `BUILDING_DIAGNOSIS`: `OUTPUT_READY` and outbox success
|
|
must wait for the future Change Request Builder and output validation.
|
|
Manifest pointers are `diagnosis`, `diagnosis_summary`, `diagnosis_context` and
|
|
`failure_report`. No coding request is generated.
|
|
|
|
Replays verify the context, checkpoint hash, identity, exact saved model request,
|
|
model output, schema, deterministic summary and manifest pointers. A crash after
|
|
model publication resumes the same accepted call without a second model request.
|
|
Missing or inconsistent pointers never silently rebuild completed output.
|
|
CaseAssembler excludes only checkpoint-declared diagnosis artifacts from raw
|
|
inputs. Unknown files remain preserved and participate in freshness checking.
|
|
All original input bytes remain intact; package generation advances normally,
|
|
package identity and analysis revision remain stable.
|
|
|
|
`DiagnosisLimits` has adjustable safety defaults of 2 MiB context and 4 MiB JSON
|
|
artifacts/summaries; these are not package-format boundaries. Align these limits
|
|
with the gateway request and package file limits. Oversized contexts are rejected
|
|
explicitly, never silently truncated. Config example: `config/diagnosis.example.toml`.
|
|
|
|
The next milestone is Change Request Builder with deterministic bugfix policy and
|
|
final output validation. A scheduler/daemon and web feedback UI remain separate.
|