Add safe Vcore wowdiag and wowsrc imports with original-data provenance
This commit is contained in:
Nico committed
1 parent
4543e7cd31
commit
741c1ca4a5
25 files changed
+1469
-9
No files matched your search
@@ -0,0 +1,129 @@
|
||||
# Vcore adapters v1
|
||||
|
||||
The authoritative architecture is unchanged. All WoW/Vcore mapping is in
|
||||
`profiles/vcore/`; supplied tool code and package text are data, never executed.
|
||||
No LLM, diagnosis, interpretation, research, SQL or investigation runs here.
|
||||
|
||||
## Source contracts
|
||||
|
||||
Supports wowdiag bug format 2 (tested 0.10.6) and optional wowsrc bug format 7
|
||||
(tested 0.7). Manifests supply identity/version; filenames and wrapper names do
|
||||
not. A pair must agree on bug ID, core revision, input wowdiag version/format and
|
||||
the entire embedded wowdiag manifest. This checks metadata consistency, not
|
||||
authenticity. Missing core revision is reported. Other commands/formats and
|
||||
malformed known records are rejected; unknown fields/files remain preserved.
|
||||
|
||||
## Archive safety
|
||||
|
||||
ArchiveReader takes an immutable bounded compressed snapshot and checks the
|
||||
complete gzip stream including CRC/trailer, then reads TAR through a temporary
|
||||
spool without member extraction. Source manifests are at the root or one wrapper
|
||||
level, exactly one per archive. Ordinary members outside the wrapper are retained.
|
||||
Absolute/parent/dot/empty paths, backslashes, colons, controls, duplicate or
|
||||
case-colliding names, file/directory conflicts, links, sparse files, devices and
|
||||
special entries are rejected. Member reads verify length. PAX/GNU metadata is
|
||||
bounded before allocation through a TarInfo subclass tested on Python 3.11.
|
||||
|
||||
ArchiveLimits are configurable operational defaults, not format restrictions:
|
||||
|
||||
| Limit | Default |
|
||||
|---|---:|
|
||||
| Compressed archive | 128 MiB |
|
||||
| Expanded TAR, including padding/headers | 256 MiB |
|
||||
| Ordinary member | 64 MiB |
|
||||
| Visible entries | 10,000 |
|
||||
| Parsed source JSON | 4 MiB |
|
||||
| Extended metadata header | 64 KiB |
|
||||
| Expansion ratio | 200:1 |
|
||||
| Combined import payload | 128 MiB |
|
||||
|
||||
Normalized/materialized bytes are held in memory within the import budget;
|
||||
bounded snapshots and spool buffers add overhead. PackageLimits and
|
||||
CaseBuilderLimits apply independently. Inputs must be complete trusted local
|
||||
files. Original TAR bytes preserve empty directories and unknown metadata.
|
||||
|
||||
## Mapping and provenance
|
||||
|
||||
wowdiag descriptions become supplied observations, expected_behavior stays
|
||||
supplied expected text, reproduction_notes become comments, and aggregate bug
|
||||
title/description stay a separate record. Reproductions preserve observations,
|
||||
the whole steps string as one verbatim element, result and explicit location/focus
|
||||
fields. Tester_note data_json text retains the original field pointer and decoded
|
||||
JSON pointer. Server events, snapshots and nested contexts stay raw references.
|
||||
Explicit typed target/quest/focus values become entities; prose and numeric hits
|
||||
never generate entities or hypotheses. Duplicate IDs/unresolved links are rejected;
|
||||
a reproduction can belong through its report when direct bug_id is null.
|
||||
|
||||
Missing expected behavior/steps remain missing. Completeness is not invented.
|
||||
Timezone-less times remain unchanged, with SOURCE_TIME_UNQUALIFIED; no offset is
|
||||
borrowed from generated_at. Source authors are preserved. Database/source/runtime
|
||||
and copied attachment files receive generic roles; unknown files are unclassified.
|
||||
|
||||
wowsrc context remains opaque source/build/custom/change/environment and schema
|
||||
references. A copied development file requires a safe existing package_path and
|
||||
matching size; copied=false retains its reason as SOURCE_NOT_COPIED. Absolute
|
||||
server file paths are metadata, never local I/O targets. Findings, rankings and
|
||||
location-script status remain source statements without causal interpretation.
|
||||
|
||||
Generic v1 contracts have additive optional origin metadata: raw source_ref,
|
||||
JSON Pointer, archive_ref/member_name, decoded pointer and unchanged timestamp.
|
||||
CaseBuilder propagates these into human/fact/entity/reproduction provenance and
|
||||
checks referenced files exist. Summaries show original source/time. Generic report
|
||||
data_quality_issues preserve source gaps. These extensions are project-independent;
|
||||
older v1 documents remain valid. Normalized report-document counts include aggregate
|
||||
and quality records, so differ from original database report counts.
|
||||
|
||||
## Import API and layout
|
||||
|
||||
```python
|
||||
from viano_bug_analysis.profiles.vcore import VcoreImporter, AdapterError
|
||||
try:
|
||||
VcoreImporter().create(
|
||||
'diagnostic.tar.gz', 'incoming.vbcase.zip', wowsrc='context.tar.gz',
|
||||
package_id='caller-owned-stable-id', case_id='caller-owned-case-id',
|
||||
)
|
||||
except AdapterError as error:
|
||||
print(error.code, error.path, str(error))
|
||||
```
|
||||
|
||||
Initial identity is supplied explicitly, never derived from filenames or silently
|
||||
equated with source bug ID. Initial state is NEW, generation/revision zero and
|
||||
project wow/vcore. Neutral PackageFactory creates a validated ZIP atomically with
|
||||
no clobber from a caller-supplied manifest/bytes; it has no project semantics.
|
||||
|
||||
Each source snapshot is stored at original/vcore/imports/<pair-hash>/<tool>/ with
|
||||
archive.tar.gz and members/<original-member-path>. Both compressed originals and
|
||||
every ordinary member survive byte for byte. Generic views, input index and
|
||||
versioned import.json receipt are at case/inputs/vcore/<pair-hash>/. Receipt records
|
||||
source versions/hashes, bug ID and issues. Manifest pointers select case_inputs
|
||||
and vcore_import. The pair hash uses archive SHA-256 values; renaming does not
|
||||
change it, while repacking is a distinct source snapshot.
|
||||
|
||||
prepare returns an ImportPlan without writing. augment(source, destination,
|
||||
wowdiag=..., wowsrc=...) uses PackageWriter, preserves old contents and identity,
|
||||
increments generation and updates manifest indices. Other profiles/bugs, content
|
||||
collisions and identical repeated imports are rejected (IMPORT_EXISTS: use the
|
||||
existing package). No automatic reopening occurs; new evidence after completion
|
||||
requires the explicit lifecycle transition to a new analysis revision.
|
||||
Existing explicit input roles and index extensions are merged rather than lost.
|
||||
When the same diagnostic archive is supplemented with wowsrc, unchanged normalized
|
||||
source rows are reused by archive hash, original member/pointer and equivalent
|
||||
content. This avoids duplicating human observations; all original artifacts remain.
|
||||
|
||||
## Regression tests
|
||||
|
||||
Portable sanitized synthetic fixtures cover mappings, preservation, malformed
|
||||
archives, links/path/header/size limits, pair consistency, missing copied files,
|
||||
times and CaseBuilder roundtrips. No original binaries or credentials are committed.
|
||||
The complete supplied BUG-20 pair has an opt-in test:
|
||||
|
||||
```powershell
|
||||
$env:VBA_WOWDIAG_SAMPLE = 'path/to/diagnostic.tar.gz'
|
||||
$env:VBA_WOWSRC_SAMPLE = 'path/to/source-context.tar.gz'
|
||||
.\.venv\Scripts\python.exe -m pytest tests/profiles/vcore/test_real_package.py -q
|
||||
```
|
||||
|
||||
It checks every raw member byte and unchanged original hashes, three observations,
|
||||
missing expected behavior, 103 database/93 source references, valid output and
|
||||
idempotent repeated construction. Without paths it is skipped; portable tests
|
||||
remain runnable offline.
|
||||
Reference in new issue
Block a user