VianoBugAnalysis/docs/investigation-loop-v1.md

LLM investigation loop v1

The Investigator connects the existing Hypothesis Builder, LLMGateway, snapshot Tool Router, EvidenceStore, BudgetManager and asynchronous test coordinator. docs/architecture.md remains unchanged and authoritative. This phase performs investigation, not final diagnosis or Change Request generation. It never writes project source, SQL, patches, branches, builds or lab/client instructions for direct execution.

API and dispatch

from viano_bug_analysis.analysis import Investigator, InvestigationConfig

loop = Investigator(
    store, gateway,
    InvestigationConfig.load("config/investigation.local.toml"),
    router=router,  # Supply the existing Vcore schema adapter where needed.
)
result = await loop.run(case_id, package_id)

step processes at most one model decision and its confirmed local execution. run repeats eligible steps up to max_steps_per_dispatch, then returns YIELDED. Callers can explicitly bound a dispatch through max_steps. PROGRESSED, REJECTED, YIELDED, RETRY_PENDING, DEFERRED, BLOCKED and COMPLETED describe scheduling outcomes. Unsafe input, stale state, exceeded operational limits or changed policies raise a controlled InvestigationError; duplicate concurrent dispatch raises the existing nonblocking LockBusy. The server worker/scheduler remains a separate integration: no continuous daemon, sleeping retry loop or UI is introduced.

Input must have an accepted current-revision interpretation and hypothesis batch. The Investigator verifies the existing budget origins and canonical case freshness; it does not silently build missing earlier phases or reopen changed raw inputs. Existing pending actions without a controller-owned plan/last evidence binding are rejected explicitly. Existing initialized budgets are reused, not refilled. When no budget config is explicitly supplied, the stored effective budget policy is adopted automatically for subsequent accounting. An explicit changed config still follows BudgetManager's policy-change rejection rules.

Deterministic cycle

1. Give each hypothesis one initial turn, then prefer higher confidence with numeric-ID ties. A reviewed early completion request may defer alternatives; see Early diagnosis review.
2. Build a bounded context from canonical case, accepted interpretation, current hypothesis states, questions, notes, budget policy/balances, current-revision stored evidence and the previous result. First iteration has no prior result.
3. Persist that context and its hash before the model call. investigation_v1 is retained for historical requests. New prompts extend the output contract with reviewable early diagnosis and assessment-only deferral.
4. Execute one eligible LLM round through the existing persisted failover/retry gateway. No global worker lock spans the external call.
5. Validate the whole reply: schema, references, confidence, question updates, assessment binding, tool scope/arguments and projected affordability/status.
6. Persist the accepted decision as an execution plan before applying effects.
7. Settle the previous evidence's bounded semantic assessment, if present; then reserve the next action and perform exactly one allowed query or test handoff.
8. Persist raw reply and normalized evidence, then account its deterministic novelty through record_evidence. Semantic assessment occurs in a later model iteration, not on the result's publication step.
9. Publish the updated hypothesis state and scheduling position.

Validation simulates settlement and reservation with the same deterministic budget rules used by execution. A valid assessment with an invalid next action is wholly rejected: it receives neither a refund nor partial question/confidence updates. Local preflight reads bounded snapshots but does not dispatch query handlers or perform external work. Runtime failures after a valid plan are recoverable from that plan; they are distinct from technically invalid model replies.

Model/controller contract

Each reply has exactly assessment, update and next:

• assessment: null on first iteration; otherwise integer semantic score 0–100, concrete reason and valid evidence IDs including the previous result.
• update: qualitative confidence (unassessed, weak, plausible, supported, strongly_supported, weakened), known evidence IDs, existing open questions to resolve, new nonduplicate questions and an audit note.
• next: one existing tool operation/arguments, an external human/automated test descriptor, or finish/reject with a reason.

Evidence citations must exist in the supplied current-revision context. Supported confidence requires nonempty stored findings. Resolved questions require evidence and cannot be unknown or already resolved. A strong-support update combined with rejection is inconsistent and rejected. These checks establish technical grounding, not scientific proof that the model's causal assessment is correct.

The controller assigns action/request IDs, deadlines, checkpoint IDs, question IDs, costs, refunds and balances. They are not model-controlled output fields. The first loop controller works over the accepted initial hypotheses. Dynamic hypothesis.create, cross-hypothesis mutation and arbitrary controller commands are not exposed; creation limits and immutable birth records remain unchanged. Adding replacement hypotheses requires a dedicated creation/budget/registry contract rather than granting a model extra allocations through this schema.

All source, database, human feedback and prior model notes are untrusted data. Symbol occurrences and partial/empty queries retain their existing scope warnings. No conclusions about current live server behavior are inferred from old snapshots. The durable context includes current-revision evidence. The opt-in project-research prompt uses a compact view documented in project-research-v1.md. If the configured byte bound is exceeded, dispatch rejects explicitly; operators must align context, gateway request and artifact limits.

Budget and stopping

Every investigation query/test uses a positive configured cost. Refunds remain strictly below cost; deterministic novelty and semantic utility must both justify them. Empty/duplicate findings earn no novelty reward, overlaps receive the existing item-based cap, and low-information/exhausted accounts are terminal. Model token usage and monetary metrics remain separate from investigation units.

The new additive BudgetManager complete operation marks budget eligibility as investigation_finished, preserves the balance and does not label a supported hypothesis as factually rejected. Pending actions must first be assessed/closed. The original hypothesis birth artifact is immutable; current qualitative states live in the controller's versioned snapshots.

Logical defaults, frozen per analysis revision:

Config	Default	Meaning
max_iterations_per_hypothesis	50	Maximum research-producing model iterations; one final assessment/closure is permitted
max_invalid_responses	3	Consecutive rejected replies before blocking the case for technical correction
max_questions_per_hypothesis	30	Bound on accumulated question records
test_deadline_seconds	21600	Controller-owned external test deadline
max_steps_per_dispatch	8	Fair scheduler yield boundary

An invalid reply changes only retry/audit bookkeeping. A following prompt includes its safe rejection code. Repeated invalid replies cause BLOCKED with INVALID_LLM_RESPONSES, requiring explicit operator resume after correction. The hypothesis stays active; pending results, reservations and remaining credit are preserved. No budget completion or zero-score settlement is fabricated. The same initial hypotheses cannot be regenerated to reset their limits.

The iteration cap prevents free model-only research loops even when credit is large. The final extra iteration can assess a pending result but may not start another action. Questions, earlier notes and evidence remain in immutable history.

Once all accounts/dispositions are terminal, the case enters BUILDING_DIAGNOSIS with final investigation state and deterministic summary.md. It does not create a diagnosis, select a root cause or route a Coding-Agent task. The summary reports balances, confidence, dispositions, evidence and outstanding questions. Its countercheck marker records actions on alternatives; it is not semantic proof that an alternative was meaningfully disproved. The later evaluator must still consider uncertainty, evidence quality and missing counterchecks.

Provider failures and external tests

Unreachable providers use existing ordered failover, configured delay and attempt count (default 10). Pending retries return immediately with retry_not_before. Exhaustion returns BLOCKED, preserving budgets, pending evidence and the frozen context. Known credential, HTTP configuration, privacy/adapter and unexpected provider failures also block the case rather than abandon its hypotheses.

After the external problem is fixed and the due blocked case is explicitly resumed through the lifecycle, a caller can supply a stable retry_id to step or run. This opens a new transport operation for an exhausted/configuration- failed call; it does not reset investigation iterations, credit or evidence. It cannot override an ordinary provider backoff or reset model-invalid counters. Saved successful execution plans are completed without another model call, even when transport configuration changes after their publication.

Human/automated test descriptors require explicitly configured action costs. The controller freezes a deadline, reserves credit, stores the exact request and delegates to AsyncTestCoordinator. The case returns BLOCKED/AWAITING_TEST_FEEDBACK without parking a worker. No time-based charges are applied. Authenticated, confirmed feedback resumes the same revision; the loop imports it as evidence and later evaluates it. Web delivery/authentication, bot steering, claims/cancellation and deadlines' notifications remain external future adapters. See async-tests-v1.md.

Persistence and recovery

Context and state files are appended under analysis/investigation/NNNN/. manifest.artifacts.investigation_state and investigation_summary are pointers. Each state checkpoint has monotonically increasing sequence, parent ref and hash binding; context hashes, identity and selected hypothesis are checked on replay. Accepted plans are additionally bound to the original successful LLM call, operation, saved request, prompt/schema/context and exact output document.

Reservation, result publication, evidence accounting, assessment and final state publication are separately idempotent. After a crash, an accepted plan resumes those steps; existing evidence is reused, not queried again or assigned new IDs. Repeated completion repairs/validates stored output without new provider calls. Pointer deletion or changed policy cannot silently restart the investigation. CaseAssembler excludes only checkpoint-declared investigation artifacts, so expected controller work does not make the accepted case stale.

InvestigationLimits are independent operational defaults: 1 MiB context, 4 MiB JSON artifact and 10,000 state checkpoints. They are adjustable safety bounds, not permanent format limits. Compiled static schema validators are reused. CheckpointStore caches validated generations only behind a newly computed full archive-byte hash and unchanged limits. PackageReader additionally takes a bounded immutable ZIP snapshot and reuses detached validation metadata for identical snapshot hashes/limits, with a bounded entry/metadata cache. No timestamp-based trust, extraction, skipped changed-file validation or raw archive retention in the cache is introduced. Static bundled schema changes require process restart.

Tests exercise round-robin investigations, real read-only queries, semantic assessment/novelty, bounded failures, limits, reference validation, questions, provider retries/config errors, actual local HTTP transport, portable resume, crashes after evidence publication, external test confirmation and byte-preserving output. The BUG-20 integration uses actual wowdiag/wowsrc data and simulated model decisions; it tests orchestration, not real model reasoning or discovery of the known fix. No paid/cloud model is called by these tests.

Question resolution is idempotent: reconfirming an already resolved question does not invalidate an otherwise valid decision and needs no new evidence. Resolving any still-open question requires evidence references. Unknown question IDs invalidate the entire decision before assessments, tools or budget changes are applied. Mixing old confirmations with new resolutions follows the same rules; existing question history is preserved.