Consolidate versioned source evidence across investigation results

docs/investigation-consolidation-v6.md

@@ -0,0 +1,75 @@
+# Cross-result investigation consolidation
+
+Only new project-backed investigation requests use `investigation_project_v6`.
+Other LLM phases, tool result persistence, budgets and lifecycle are unchanged.
+Frozen v4/v5 contexts retain their prompts, views and request hashes. The new
+algorithm operates after the v5 view, before request freezing and hashing.
+
+## File identity and shared code
+
+A FILE entry identifies the exact combination of observed file metadata
+(including path and content hash/blob) and pinned capture origin. Equal paths
+with different hashes, revisions or origins remain distinct. Captures without
+an origin or content identity are not assumed to describe the same snapshot.
+The existing shared_metadata table continues to carry repeated provenance;
+FILE entries reuse these exact provenance references rather than inventing one.
+
+Source captures keep their evidence ID, query, coverage, truncation, paging and
+requested/returned bounds. Their items reference shared_sources through
+`shared_source_lines_v1`. Each reference contains FILE ID and ordered slices
+`[start,count]` of positions in the shared line table, not source line numbers.
+The table retains actual line numbers, text and truncation flags. Equal records
+are transmitted once across captures, including overlapping source intervals.
+Absent line numbers remain gaps. Repeated rows and original ordering reconstruct
+exactly. Different records at the same line are both retained and explicitly
+listed under conflicting_lines, even if the purported file hash is identical.
+No observation wins silently.
+
+Read-query paths use path_ref only when exactly one FILE identity matches the
+path and origin. FILE metadata contains the path. The controller can resolve
+known FILE aliases in source.read paths using only the active context, followed
+by the existing path, repository and read-only validation. Unknown aliases grant
+no access and remain subject to ordinary tool rejection. Existing P aliases keep
+their original allocation and meaning.
+
+## Equal results and different queries
+
+Large identical result objects may share a RESULT entry. Queries and origins stay
+attached to each evidence record: equal returned DB rows do not imply equal
+filters, snapshots or coverage. Only exact typed JSON equality is used. False,
+zero, null and absent fields are not interchangeable. No older evidence or
+counterevidence is filtered out, and no extra summarization/model call is added.
+
+The final candidate is used only if its canonical model text is smaller than the
+v5 view; otherwise the v5 encoding is retained within the v6 prompt. This avoids
+expanding a request merely to introduce reference tables. It is a deterministic
+representation choice, not a monetary limit or investigation stopping rule.
+
+`restore_view` reconstructs the previous working view, including original item
+order, duplicate records, query paths, actual values and every evidence ID.
+Offline benchmarks compare canonical JSON after reconstruction, not only hashes
+or field counts. The system instructions were shortened while retaining the
+existing task, evidence, budget, tool and early-review rules and describing all
+new and fallback encodings. Output schemas and controller gates are unchanged.
+
+## Measurements and limits
+
+`scripts/measure_investigation_context.py PACKAGE` now defaults to v6, with
+`--target-version investigation_project_v5` available for comparison. It reports
+user text and total message text (including system instructions), component
+sizes and exact reconstruction. No provider request is made. Stored contexts
+include rejected/unused attempts; these are not billed-call totals.
+
+On the local archives:
+
+| Trial | Stored contexts | v5 user text | v6 user text | v5 all messages | v6 all messages |
+|---|---:|---:|---:|---:|---:|
+| Bug 17 | 14 | 1,017,342 | 1,007,291 | 1,125,982 | 1,087,343 |
+| Bug 20 | 9 | 549,750 | 549,750 | 619,590 | 601,212 |
+
+Additional full-message reductions are 3.43% and 2.97%, respectively. Bug 17's
+captured source intervals were mostly adjacent rather than overlapping; Bug 20
+had little source content to consolidate. The earlier v5 reduction must not be
+counted again as new savings. Repeated/overlapping source workloads benefit more,
+as tested separately. Actual provider tokens, cost and reasoning quality still
+need evaluation in a later live run. Remaining unique code/history is still sent.