146 lines
8.7 KiB
Markdown
146 lines
8.7 KiB
Markdown
# Hypothesis Builder v1
|
|
|
|
This implements architecture section 15 and initial budget allocation from section 16.
|
|
It produces untested causal candidates from a confirmed case and Case Interpretation.
|
|
It implements no investigation queries, EvidenceStore, BudgetManager accounting,
|
|
diagnosis, patch generation or Change Requests. Architecture.md remains unchanged.
|
|
|
|
## API and configuration
|
|
|
|
```python
|
|
from viano_bug_analysis.analysis import HypothesisBuilder, HypothesisConfig
|
|
|
|
policy = HypothesisConfig.load("config/analysis.local.toml")
|
|
builder = HypothesisBuilder(store, gateway, policy)
|
|
result = await builder.build(case_id, package_id)
|
|
```
|
|
|
|
Copy analysis.example.toml to the ignored analysis.local.toml. Defaults are
|
|
min_hypotheses=3, max_hypotheses=5 and initial_budget=100. Counts and budget must be
|
|
positive integers, and minimum cannot exceed maximum. Five is a configurable policy
|
|
default, not an immutable package-format restriction.
|
|
|
|
The interpreter must have confirmed the current analysis revision. The builder reuses
|
|
and verifies that interpretation against its original case, raw inputs, sources,
|
|
summary and model outcome before generating anything. It does not silently start the
|
|
earlier LLM phase. Missing observations prevent unsupported hypothesis generation.
|
|
Package filenames play no functional role.
|
|
|
|
## Prompt and source contracts
|
|
|
|
The versioned hypotheses_v1.txt prompt is independent of model/provider. It receives
|
|
the interpretation's canonical case catalogue, the accepted interpretation statements
|
|
as I aliases with JSON pointers, the frozen generation policy and prior generations'
|
|
initial proposals. Historical proposals are labelled as untested initial proposals;
|
|
future investigation results will require their own contracts. No raw source/database
|
|
payloads are loaded into the prompt, and input text is untrusted data.
|
|
|
|
Every candidate provides title, mechanism, rationale, distinction, case source_ids,
|
|
interpretation_ids, expected_evidence, falsifying_evidence and open_questions. All
|
|
references must exist in this exact context. The first candidate must cite human
|
|
observation/report/comment data and the first known divergence, otherwise the failure
|
|
stage, otherwise reported behavior. Alternatives may rely more on explicit technical
|
|
facts, but cannot be grounded solely in unconfirmed assumptions, external expectations
|
|
or unread implementation references.
|
|
|
|
The controller rejects normalized duplicate titles/mechanisms, including differences
|
|
only in Unicode presentation, case, punctuation or whitespace. This cannot prove
|
|
semantic diversity or factual quality. Model adherence, plausible alternatives and
|
|
useful falsification criteria must be evaluated later with actual models and known
|
|
bugs. A nonblank distinction is not proof that mechanisms really differ.
|
|
|
|
No IDs, priorities, statuses, confidence values, budgets, patch instructions or tool
|
|
requests are accepted from the model. Each record is controller-labelled H1, H2, ...,
|
|
positioned in returned order, status=new and confidence=unassessed. H1 is marked
|
|
tester_primary; the rest are alternatives. Percent confidence is not part of the
|
|
contract. Hypotheses are possible causes, never a confirmed root cause.
|
|
|
|
## Budget and revision-wide cap
|
|
|
|
Policy is saved before the first gateway request and pinned for that analysis revision.
|
|
Changing counts or initial budget during a pending/rejected attempt is rejected rather
|
|
than resetting limits. A completed generation is reused regardless of later runtime
|
|
config or attempt_id; another call cannot create a second batch or refill allocations.
|
|
New policy values take effect with a new analysis revision.
|
|
|
|
Every accepted hypothesis receives budget_allocation.initial=initial_budget in abstract
|
|
integer investigation_units. It is independent of tokens, money, priority and model
|
|
confidence. This is an immutable creation allocation, not a live remaining balance;
|
|
budget_accounting is explicitly not_started. Future BudgetManager accounts must be
|
|
keyed by case/package identity, analysis revision and H ID and consume each allocation
|
|
once, using their persisted accounting state on resumption.
|
|
|
|
The immutable registry records max_hypotheses, created_count, remaining_slots, next_id
|
|
and every creation's ID, origin artifact/hash/JSON pointer and allocated budget. Count
|
|
is based on creations, not active hypotheses. Rejected, abandoned or budget-exhausted
|
|
statuses cannot remove a creation or restore capacity. Only one initial batch is
|
|
published per revision; a future hypothesis.create operation must extend this durable
|
|
creation history and enforce the same saved cap. That operation is not implemented here.
|
|
|
|
Invalid model responses are rejected atomically: no H IDs or budget allocations are
|
|
created, and no partial registry is published. Raw proposals may remain in gateway audit
|
|
artifacts, but are not accepted diagnostic hypotheses. Correcting an invalid answer uses
|
|
an explicit new attempt_id and the same frozen policy.
|
|
|
|
New analysis revisions can allocate fresh budgets, with IDs scoped as (analysis_revision,
|
|
H ID). Old declarations, allocations and history remain unchanged in the case package.
|
|
No repeated call within the same revision can impersonate reopening or silently discard
|
|
an existing registry by removing manifest pointers.
|
|
Declared current-revision hypotheses or registries without supported creation history
|
|
are rejected rather than silently forgotten and replaced with a fresh allocation.
|
|
|
|
BudgetManager will later enforce positive action costs, reward < cost, no reward for
|
|
duplicates, low-information termination and exhaustion. This milestone does not run any
|
|
autonomous research loop, so it does not yet debit/refund budgets or claim to enforce
|
|
investigation iteration limits. The loop must not be added before those accounting rules.
|
|
|
|
## Durable phase and failure handling
|
|
|
|
The normal path is three atomic generations after interpretation:
|
|
|
|
1. Enter GENERATING_HYPOTHESES and store policy.json and context.json.
|
|
2. CaseLLMRunner records a gateway round with configured ordered failover and retries.
|
|
3. Publish hypotheses.json, registry.json and deterministic escaped hypotheses.md together,
|
|
setting manifest pointers hypotheses, hypothesis_registry and hypotheses_summary.
|
|
|
|
hypothesis_context and hypothesis_policy also identify the saved inputs. Artifacts live
|
|
under analysis/hypotheses/NNNN/; errors/<attempt-hash>.json retains rejected attempts.
|
|
Accepted JSON, registry and summary are checked together. All original and unknown files
|
|
are preserved byte-for-byte. package_generation increases; package_id and analysis_revision
|
|
stay stable within the phase.
|
|
|
|
The completed phase remains GENERATING_HYPOTHESES. The future investigation controller
|
|
will explicitly start INVESTIGATING_HYPOTHESES. No BudgetManager or investigation actions
|
|
are invoked here.
|
|
|
|
Results are COMPLETED, RETRY_PENDING, DEFERRED, BLOCKED or ERROR. Unavailability uses the
|
|
existing configurable ten-round policy, persisted due times and explicit block after
|
|
exhaustion; it never sleeps inside a dispatch. Invalid answers are distinct from unreachable
|
|
infrastructure. Failures are logged with GENERATING_HYPOTHESES as the actual phase.
|
|
|
|
Checkpoints bind policy/context/case/interpretation/model inputs and outputs by hash and
|
|
reference. Confirmed model responses survive a crash before final publication, and final
|
|
publication/index crashes replay without new hypotheses or allocations. Nonblocking
|
|
case locks prevent duplicate dispatches; generation checks reject concurrently stale
|
|
results. A remote call that never reached a durable checkpoint may repeat after cancellation
|
|
or a crash, as with the existing gateway. A successful batch cannot be regenerated by
|
|
another attempt ID, pointer deletion or terminal status markers.
|
|
|
|
HypothesisLimits default to 1 MiB context, 32 prior generation documents and 2 MiB each
|
|
for output and summary. These are configurable operational defaults, independent of
|
|
format contracts. Underlying interpreter, gateway and ZIP limits still apply. History
|
|
is not silently truncated when a configured limit is reached.
|
|
|
|
## Tests and remaining scope
|
|
|
|
Offline tests cover budget equality, model-field rejection, count caps, pinned policies,
|
|
duplicate/reference rejection, registry consistency, retry/blocking, portable replay,
|
|
crash recovery, concurrent changes, reopening and preserved old allocations. A local
|
|
HTTP server tests the domain schema over the actual adapter. The optional wowdiag/wowsrc
|
|
pair regression now reaches hypothesis generation using a fixture provider and verifies
|
|
full preservation of the prior package and both original archives.
|
|
|
|
No real model, paid endpoint or known fix is used. These tests prove contracts and package
|
|
lifecycle, not that generated candidates explain BUG-20. Investigation Tools, EvidenceStore,
|
|
budget accounting and the investigation loop are the subsequent planned milestones.
|