Allow shared evidence and bounded causal diagnosis with retry feedback

This commit is contained in:
Nico committed 2026-10-15 18:05:38 +02:00
1 parent 5082354bc3
commit c1f44e4e02
21 files changed
+727 -41

No files matched your search

+4 -3
View File
@@ -29,9 +29,10 @@ uncertainty, final reason and recommended human-review actions.
A confirmed result requires supported/strongly_supported confidence, a supported
selected hypothesis that was not factually rejected, nonempty stored root-cause
findings also cited by its evaluation, and at least one alternative with actual
investigation actions and a reasoned countercheck citing evidence originating
from that alternative. Unknown/duplicate/missing hypothesis IDs and fabricated
findings also cited by its evaluation, and at least one alternative with a
reasoned countercheck citing actual shared findings. Evidence provenance does
not restrict which hypotheses may use it. See [Decision v2](diagnosis-decision-v2.md)
for explicit scope limits, contradictions and retry feedback. Unknown/duplicate/missing hypothesis IDs and fabricated
or empty support references are rejected. Loop confidence and its countercheck
marker alone never authorize a diagnosis. Partial/empty snapshots are not proof
of absence. The model must distinguish intended external behavior from local
+43
View File
@@ -0,0 +1,43 @@
# Evidence-grounded diagnosis decision v2
The decision is whether an adequately supported cause justifies a bounded,
verifiable change, not whether every conceivable cause has been excluded.
There remains at least one targeted alternative countercheck; unlimited
investigation of theoretical or newly proposed hypotheses is not required.
Evidence IDs are usable across hypotheses, including root-cause support and
counterchecks. The record's hypothesis ID is provenance only. Technical checks
still require actual referenced findings; a schema-only capture is insufficient
for causal support or a meaningful countercheck. The evaluator must explain
which prerequisite/prediction a finding tests and respect its coverage. Absence
within a complete filtered snapshot is not absence of all possible runtime paths.
New diagnosis contexts freeze evaluation_protocol=2. diagnosis_v2 and
diagnosis_early_v2 require decision_basis: causal_explanation, scope_limits,
sufficient_for_change and contradictions. Each material contradiction cites
known evidence and records its resolution and reason. A confirmed output cannot
claim sufficient support while marking a material contradiction unresolved.
Unfinished alternatives require explicit remaining_uncertainty; another supported
hypothesis is not automatically a contradiction or an obligation to research it.
The deterministic validator checks structure, identities, references, nonempty
findings, consistency and declared unresolved contradictions. Semantic relevance,
whether findings are actually contradictory, adequacy of a countercheck, and
whether prose overstates evidence remain LLM judgments. The structure does not
prove that a model has detected every contradiction or told the truth. Prompts
explicitly reject unjustified universal exclusion and require scoped claims.
Cause confidence and implementation readiness remain separate. Change Request
and output checks are unchanged; missing implementation data must not be invented.
This does not add dynamic hypothesis creation or change investigation budgets.
Every new diagnosis retry includes prior failure codes, call references and
correction guidance from the same review namespace. It never overwrites the
frozen base context. The precise request is reproducible after a restart and
when validating accepted output. A pending historical call retains its old
prompt/hash; after a historical rejection the next fresh attempt uses v2 with
feedback. Completed historical diagnoses remain replayable.
The current live case is not rewritten or auto-approved by this change. Its
stored response can be structurally tested offline under shared-evidence rules;
that is not a new semantic review of its reasoning. No extra summary LLM is used.
+9 -6
View File
@@ -8,9 +8,12 @@ The service performs no patch, SQL write or client automation.
New investigation prompts (`investigation_v2`, `investigation_project_v3`) offer
`next.kind = request_diagnosis`, with `reason` and `countercheck_refs`.
Current prompts are `investigation_v3` / `investigation_project_v4`; older
persisted prompt versions remain reproducible.
The selected hypothesis must be `strongly_supported`, cite concrete nonempty
investigation evidence of its own, and cite an actual result from an alternative.
An alternative already rated supported/strongly_supported prevents a request.
investigation evidence, and cite findings that meaningfully test an alternative.
Evidence is shared: collecting hypothesis IDs never restrict its use. Another
supported alternative does not by itself establish a material contradiction.
The controller validates IDs and structure; whether the evidence establishes
causality and whether a negative result has sufficient coverage remain explicit
responsibilities of the independent final evaluator.
@@ -28,16 +31,16 @@ and its initial turn for each hypothesis remain in use outside preparation.
## Independent decision and continuation
`diagnosis_early_v1` evaluates every hypothesis, all evidence and remaining
`diagnosis_early_v2` evaluates every hypothesis, all evidence and remaining
uncertainty independently. Acceptance requires the requested hypothesis,
`strongly_supported` confidence, direct evidence for it, an adequate investigated
alternative countercheck and no other evaluation marked supported. Ordinary
`strongly_supported` confidence, direct evidence for it, an adequate evidence-grounded
alternative countercheck, and no explicitly unresolved material contradiction. Ordinary
final diagnosis validation remains mandatory. An accepted review creates the
normal diagnosis artifact; the existing separate Change Request and output
validators still decide whether a coding task can be issued. Missing coordinates
or other implementation prerequisites must remain explicit missing information.
An inconclusive review, insufficient confidence, competing support or exhausted
An inconclusive review, insufficient confidence, unresolved material counterevidence or exhausted
invalid-response retries declines the request. The case returns to
INVESTIGATING_HYPOTHESES in the same analysis revision. The immutable review reason
is included in subsequent investigation context. No failure report or coding