149 lines
8.2 KiB
Markdown
149 lines
8.2 KiB
Markdown
# EvidenceStore v1
|
|
|
|
This milestone persists validated tool replies and confirmed test feedback as
|
|
portable evidence. It adds no model calls, autonomous investigation, diagnosis,
|
|
patch generation, external research service, web UI or bot steering. The
|
|
authoritative architecture is unchanged.
|
|
|
|
## Controller API
|
|
|
|
```python
|
|
from viano_bug_analysis.evidence import EvidenceStore
|
|
|
|
# Reserve each requested action through BudgetManager before the query.
|
|
reply = router.query(budget.snapshot(case_id, package_id).package, request)
|
|
evidence = EvidenceStore(store, router=router)
|
|
saved = evidence.store_tool_results(
|
|
case_id, package_id, "iteration-0001", request, reply,
|
|
)
|
|
budget.record_evidence(case_id, package_id, "H1", "query-0001", saved.record_refs[0])
|
|
# A future validated controller assessment supplies the bounded semantic score.
|
|
budget.settle(case_id, package_id, "H1", "query-0001",
|
|
semantic_score=50, reason="Assessment of the stored evidence")
|
|
```
|
|
|
|
The caller's router must use the same trusted descriptor adapter and operational
|
|
limits as the original query. Before publication, EvidenceStore deterministically
|
|
recomputes the whole batch against the current package and compares the complete
|
|
reply. Invented snippets, rows, counts, provenance, changed identities or stale
|
|
package generations reject the entire batch. Each result must match a still
|
|
reserved budget action and operation. The full batch is published atomically;
|
|
no partial evidence records or ghost IDs survive rejection.
|
|
|
|
`store_feedback(case_id, package_id, operation_id, feedback_ref)` imports an
|
|
already checkpoint-confirmed async test reply, bound to the original hypothesis,
|
|
action and pending reservation. Draft/unconfirmed feedback is not evidence.
|
|
Feedback and attachment hashes are verified; no new test request is submitted.
|
|
|
|
`snapshot(case_id, package_id)` validates and returns all record references in
|
|
creation order without requiring an active current budget revision. History is
|
|
therefore readable while a reopened case has not yet rebuilt its hypotheses.
|
|
An empty store has `index_ref = None` and no records. Mutations require initialized
|
|
budget accounting and `GENERATING_HYPOTHESES` or `INVESTIGATING_HYPOTHESES`.
|
|
Blocked snapshots preserve their budget and evidence; new records wait for the
|
|
existing controlled resume path.
|
|
|
|
## Records, content and immutable indices
|
|
|
|
The controller assigns `E000001`, `E000002`, … across the entire case lifecycle.
|
|
IDs do not reset at an analysis revision or after service restart/transport.
|
|
The model cannot supply IDs, novelty scores, duplicate markers or budget values.
|
|
|
|
```text
|
|
analysis/evidence/0001/<hashed-operation-id>/request.json
|
|
analysis/evidence/0001/<hashed-operation-id>/result.json
|
|
analysis/evidence/0001/<hashed-operation-id>/E000001.json
|
|
analysis/evidence/0001/<hashed-operation-id>/index.json
|
|
analysis/evidence/content/<content-sha256>.json
|
|
```
|
|
|
|
`evidence-record-v1` binds case/package identity and revision, hypothesis/action,
|
|
operation, content ref/hash, first identical record (`duplicate_of`), novel atom
|
|
hashes and deterministic novelty score. Its origin links the unchanged request
|
|
and reply with hashes and original reply position, or the confirmed feedback ref
|
|
and hash. Source bytes, line numbers, row JSON pointers, coverage and truncation
|
|
markers remain in the stored normalized content and original reply.
|
|
|
|
`evidence-content-v1` separates actual content from query IDs, provider metadata,
|
|
pagination bookkeeping and source package generation. Source occurrences retain
|
|
location/hash/text; database rows retain location/hash/selected row values; schema
|
|
descriptors retain columns and provenance. Full contents are reused by hash.
|
|
Identical content obtained again gets a new occurrence record with a reference
|
|
to the first record; neither original record nor provenance is overwritten.
|
|
|
|
`evidence-index-v1` is an immutable cumulative index of record IDs/refs/hashes,
|
|
with a parent ref/hash and next-ID counter. `manifest.artifacts.evidence_index`
|
|
points to the latest index. Checkpoints bind that index hash and the new record
|
|
refs. Deleting a pointer or changing counts, contents, origin hashes, duplicate
|
|
markers or novelty metadata fails replay; it cannot reset IDs or restore novelty.
|
|
All original and unknown package payloads survive unchanged. New publications
|
|
increase `package_generation`, leaving `analysis_revision` unchanged.
|
|
|
|
## Novelty and budget integration
|
|
|
|
Each located content item is an atom. Its identity includes content kind, scope,
|
|
coverage and the original item's data/provenance. Atoms are tracked across all
|
|
stored evidence, hypotheses and analysis revisions. Sorting and deduplication
|
|
are deterministic; changing request/action IDs or grouping an existing excerpt
|
|
into a new result is not new evidence.
|
|
|
|
```text
|
|
novelty_score = floor(100 * unseen_item_count / returned_item_count)
|
|
```
|
|
|
|
Empty findings and confirmed `unable_to_test` replies have score zero. A partly
|
|
overlapping excerpt can only claim its new items. For example, one new line in
|
|
a two-line excerpt caps novelty at 50; rediscovering both lines caps it at zero.
|
|
This detects exact located-content reuse, not semantic equivalence or correctness.
|
|
Different projections, genuinely different source snapshots and richer excerpts
|
|
can contain new atoms; judging their practical relevance remains the semantic
|
|
assessment's job. Coverage and truncation remain explicit, so an empty partial
|
|
snapshot is not a claim of server-wide absence.
|
|
|
|
`BudgetManager.record_evidence` verifies record/index/checkpoint bindings, loads
|
|
the canonical content and passes only newly observed atoms to accounting. Its
|
|
optional `novelty_cap` result field is a backward-compatible addition to the
|
|
budget v1 request/event schemas. The deterministic score cannot exceed either
|
|
the evidence cap or the configured new-evidence score. Empty novel payloads earn
|
|
no novelty reward even when a controller supplies semantic score 100. Existing
|
|
direct `record_result` callers remain supported unchanged.
|
|
|
|
Evidence publication alone does not debit, refund, settle, refill or terminate
|
|
any account. Result accounting and later semantic settlement remain separate
|
|
idempotent steps. A crash between those steps leaves the reservation recoverable,
|
|
not free to allocate again. Rewards still satisfy `reward < cost` and retain the
|
|
configured low-information stopping rules. No hypothesis confidence or root
|
|
cause is inferred by the store.
|
|
|
|
## Robustness and limits
|
|
|
|
The store uses a per-case evidence lock, then the existing short publication
|
|
locks and generation comparison. No lock waits on an external human or model.
|
|
Read-only recomputation verifies local snapshots while publication is protected.
|
|
Repeated identical `operation_id` payloads reuse the existing records; changed
|
|
payloads reject. IDs are scoped by revision/operation for operation replay but
|
|
the evidence ID sequence remains case-wide. Completed publication with a failed
|
|
local index update is repaired on replay without another record or budget debit.
|
|
|
|
Replay validates schemas, index lineage, hashes, original request/reply bindings,
|
|
source/feedback attachment hashes and recomputes novelty/duplicate metadata.
|
|
Hashes detect inconsistency, not an adversary rewriting and rehashing an entire
|
|
case history; they are not signatures.
|
|
|
|
`EvidenceLimits` defaults to 4 MiB per JSON artifact, 10,000 records and 64 MiB
|
|
across distinct retained evidence artifacts. Limits reject explicitly and are
|
|
configurable operational defaults, not permanent v1 format caps. Package, case,
|
|
router and budget limits remain independently applicable. Cumulative indices
|
|
and full history replay trade simplicity for overhead; optimization is later work.
|
|
|
|
CaseAssembler excludes only checkpoint-declared evidence files and their content
|
|
refs from raw input identity. Thus storing investigation evidence does not
|
|
invalidate the accepted case; undeclared files still count as new raw input.
|
|
|
|
Tests cover preservation, provenance, duplicate and overlapping excerpts, scoped
|
|
empty results, forged/stale replies, missing reservations, idempotence, limits,
|
|
pointer deletion, reopen/transport, publication crashes and confirmed test
|
|
feedback. The real BUG-20 regression additionally stores its source/schema/row
|
|
results and accounts them through the existing budget engine, using simulated
|
|
semantic assessments and no known fix.
|