5.9 KiB
Causal confidence and local checkpoint performance
The stable-case-cache Bug 17 trial reached 20 dispatches without a fatal error, but kept selecting a "missing/mismatched quest accept hook" hypothesis while finding a matching hook, the relevant quest ID, dispatch and AI creation. Novel information about working implementation was being confused with evidence of the hypothesized defect. This is a reasoning-quality issue, independent of cache hits. No new paid model run was made for this change.
Explicit causal review, without another LLM call
New project investigation contexts use investigation_project_v12. Each existing
model answer now includes a short update.causal_review:
effect: supports / contradicts / mixed / neutral, for the exact selected mechanism;support,against: up to eight known stored evidence IDs on each side;reason: at most 600 characters explaining the causal relation.
The review covers accumulated evidence, not only the most recent query. A high
assessment.semantic_score means informative evidence, not positive causal
support. Locating a normal function/trigger/ID must not support its absence.
Untraced links are uncertainty, not proof of a defect. A complete scoped empty
query can support missing-data hypotheses; schema evidence can establish a schema
defect but does not establish runtime database contents. No Vcore/WoW-specific
entity, ID, implementation pattern or natural-language truth classifier is
hardcoded into the controller.
The controller validates references, their presence in update.evidence_refs,
and the declared direction's required support/against lists. Unknown references
or invalid reviews reject the entire decision before tools, assessment settlement
or spending. Once valid, deterministic calibration applies:
| Declared cumulative effect | Effective controller confidence |
|---|---|
| contradicts | weakened |
| neutral or mixed, with requested supported/strongly_supported | plausible |
| otherwise | requested confidence |
Calibration is not another invalid-answer/retry loop: a contradictory high confidence claim is capped in the accepted controller update. The original model answer remains immutable. Requested confidence, effective confidence and the review are retained in assessment notes. Existing confidence scheduling then prefers stronger alternatives after bootstrap; it does not exhaust the weakened hypothesis's budget just to finish it. Useful counterevidence can still receive ordinary information rewards, with all budget invariants unchanged.
Early diagnosis uses the effective confidence, so contradiction/mixed/neutral cannot bypass its strong-evidence gate. Independent final diagnosis remains in place. The model still has to classify the meaning of evidence correctly: this contract makes that judgment explicit and auditable, not infallible. Quality needs verification in a later real model run. Older prompt contracts, saved answers and replay schemas retain their old behaviour.
Stable aliases and the full-history cache partition continue in v12. No historic research payload is filtered, and no additional model call, summary or paid diagnostic review is introduced. The new short output review has some token cost; it is bounded and intended to avoid much more expensive misdirected iterations.
Local performance bottleneck
The measured trial had 148 durable ZIP generations: 4,427,901,136 bytes in total.
The self-contained current g148 package was 36,556,503 bytes. latest() hashed
every historical ZIP on every lookup and revalidated all of them after restart.
Budget snapshots and checkpoint operations invoked this repeatedly, producing
growing local costs unrelated to provider inference.
Normal CheckpointStore.latest() now selects the newest managed-name candidate
and fully validates that package, including manifest identity/generation,
all archive contents, configured safety limits and the complete embedded
checkpoint history. Names select only a candidate; they do not establish case
identity or authorize contents. Symlinks and malformed managed generation names
remain rejected. Numeric selection supports generations beyond eight digits.
Fresh full-byte hashing still protects cached validation; size/mtime is never
trusted. A corrupt or misidentified head is an error, never a rollback.
Old ZIP snapshots remain untouched. Their embedded history/content is already
carried by the head, as required for a portable case. Normal lookup no longer
audits the physical health of redundant older ZIP files. Corruption confined to
an obsolete snapshot does not block a valid current case. Explicit
latest(case_id, package_id, audit_history=True) retains the former full-snapshot
audit and will detect it. This separates current-case validation from backup
auditing; it does not delete history or relax current package validation.
checkpoint_lookup is also timed through existing operational telemetry, so
future trials can distinguish lookup time from other nested operations. Nested
operation durations must not be summed as independent wall time.
Offline checks and limits of the measurement
A cold BudgetManager.snapshot on the archived g148 case, with cProfile enabled,
changed from 38.836 seconds / 99,054,413 function calls to
0.653 seconds / 1,805,962 calls after this change. Both measurements include
the same read/replay operation and fresh CheckpointStore instances. OS caches,
profiling and machine load affect timings; this is a local diagnostic comparison,
not a prediction of end-to-end trial speed or API costs.
Regression tests cover calibrated confidence and scheduling, high-value counterevidence/rewards, empty-query evidence, atomic invalid-reference rejection, old contracts, same-size/same-mtime tampering, corrupt heads, forged newer names, head-only reads, and explicit full-history audits. The original trial package and its existing investigation state are not rewritten or resumed by this fix.