VianoBugAnalysis/docs/phase-json-consolidation.md

Shared JSON consolidation across analysis phases

New requests use case_interpretation_resolved_v7, hypotheses_v3, diagnosis_v4 / diagnosis_early_v4, and change_request_v5. Entity resolution and investigation request encodings are unchanged in this milestone. The archived canonical context and the output schema are unchanged, apart from additive protocol markers/version enums. docs/architecture.md is unchanged.

Lossless transport view

analysis/json_consolidation.py works on the already established phase-specific model view:

• Repeated long values/objects are stored once in shared and addressed by a local $ref.
• Frequently repeated object field names are stored once in shapes; $record:[shape,values] retains every original field/value, including evidence IDs and provenance.
• Uniform object arrays may use $table with common attributes, columns and ordered rows.
• Uniform source-line records may use $lines with common file metadata and contiguous blocks of plain text. Gaps, duplicate line numbers and original order remain unchanged.
• The smallest candidate is chosen; unused shared definitions are removed. If the full encoding would be larger, ordinary JSON is retained. Reserved-key collisions fall back to ordinary data; an input resembling the transport envelope is escaped as a literal.

restore() reconstructs the exact input JSON value. Missing and null fields, boolean and integer values, database column names, qualified negative results, filtering, paging and snapshots are preserved. There is no semantic summarization, extra LLM request, evidence selection or deletion. The decoder is an offline validation aid, not a facility for executing model-provided data. Model output remains ordinary JSON under the original schema.

The new system prompts explain the encoding. The model must cite original source/evidence identifiers, not structural reference aliases. Original data remains untrusted. This structurally more compact presentation still needs live quality validation; exact programmatic restoration is not proof of unchanged model comprehension.

Frozen requests and recovery

Interpretation stores an optional model_encoding marker in genuinely new modern contexts and reads that marker back when rebuilding a persisted context. Existing v6 contexts remain v6. Hypothesis construction takes its prompt version from the saved context on resume. Diagnosis and handoff reconstruct the saved request version during provider retry/crash recovery; only a new request uses the new codec. Older prompts are unchanged. Correction feedback recognizes the new interpretation version and retains its identity/citation guidance.

Offline measurement

scripts/measure_phase_consolidation.py PACKAGE.zip reads saved requests, projects only their input through the codec and verifies exact restoration. No LLM, credentials, network access or package mutation occurs. Counts are characters, not billed tokens. All-message counts include the additional system encoding instructions, plus existing feedback messages. Gateway output schemas are unchanged. Minor new protocol-marker bytes are not simulated.

Using the successful Bug 20 full-run archive:

Phase	Saved requests	Before: all message characters	After	Reduction
Interpretation	2	172,681	155,961	9.7%
Hypotheses	1	84,401	75,623	10.4%
Early diagnosis	1	72,373	62,270	14.0%
Coding/human handoff	1	37,582	31,908	15.1%

Bug 17 v7 trial: interpretation 78,920 -> 71,186 characters (9.8%); hypotheses 79,048 -> 71,000 (10.2%). No diagnosis/handoff exists in that interrupted archive.

These are additional reductions relative to previously compacted phase views, not comparisons with original raw case packages. Actual token savings and analytical quality require a subsequent paid live run.