Add safe Vcore wowdiag and wowsrc imports with original-data provenance
This commit is contained in:
1 parent
4543e7cd31
commit
741c1ca4a5
25 files changed
+1469
-9
No files matched your search
@@ -8,6 +8,8 @@ Job state, checkpoints and inbox admission are documented in
|
||||
[docs/lifecycle-v1.md](docs/lifecycle-v1.md).
|
||||
Generic case construction and its input contracts are documented in
|
||||
[docs/case-builder-v1.md](docs/case-builder-v1.md).
|
||||
Vcore imports for wowdiag/wowsrc are documented in
|
||||
[docs/vcore-adapters-v1.md](docs/vcore-adapters-v1.md).
|
||||
|
||||
Requires Python 3.11 or newer.
|
||||
|
||||
|
||||
@@ -126,6 +126,12 @@ validation itself streams. No artifact interpretation is performed.
|
||||
|
||||
## Writer API and publication contract
|
||||
|
||||
`PackageFactory.create(destination, manifest, artifacts)` creates a generation-zero
|
||||
package from caller-supplied metadata and byte artifacts. It adds integrity records,
|
||||
applies PackageLimits, validates the staged ZIP, flushes/fsyncs and publishes
|
||||
atomically without replacing an existing target. It is project-independent;
|
||||
subsequent enrichment still uses PackageWriter.
|
||||
|
||||
```python
|
||||
from viano_bug_analysis.package import PackageWriter, PackageWriteError
|
||||
|
||||
|
||||
@@ -0,0 +1,129 @@
|
||||
# Vcore adapters v1
|
||||
|
||||
The authoritative architecture is unchanged. All WoW/Vcore mapping is in
|
||||
`profiles/vcore/`; supplied tool code and package text are data, never executed.
|
||||
No LLM, diagnosis, interpretation, research, SQL or investigation runs here.
|
||||
|
||||
## Source contracts
|
||||
|
||||
Supports wowdiag bug format 2 (tested 0.10.6) and optional wowsrc bug format 7
|
||||
(tested 0.7). Manifests supply identity/version; filenames and wrapper names do
|
||||
not. A pair must agree on bug ID, core revision, input wowdiag version/format and
|
||||
the entire embedded wowdiag manifest. This checks metadata consistency, not
|
||||
authenticity. Missing core revision is reported. Other commands/formats and
|
||||
malformed known records are rejected; unknown fields/files remain preserved.
|
||||
|
||||
## Archive safety
|
||||
|
||||
ArchiveReader takes an immutable bounded compressed snapshot and checks the
|
||||
complete gzip stream including CRC/trailer, then reads TAR through a temporary
|
||||
spool without member extraction. Source manifests are at the root or one wrapper
|
||||
level, exactly one per archive. Ordinary members outside the wrapper are retained.
|
||||
Absolute/parent/dot/empty paths, backslashes, colons, controls, duplicate or
|
||||
case-colliding names, file/directory conflicts, links, sparse files, devices and
|
||||
special entries are rejected. Member reads verify length. PAX/GNU metadata is
|
||||
bounded before allocation through a TarInfo subclass tested on Python 3.11.
|
||||
|
||||
ArchiveLimits are configurable operational defaults, not format restrictions:
|
||||
|
||||
| Limit | Default |
|
||||
|---|---:|
|
||||
| Compressed archive | 128 MiB |
|
||||
| Expanded TAR, including padding/headers | 256 MiB |
|
||||
| Ordinary member | 64 MiB |
|
||||
| Visible entries | 10,000 |
|
||||
| Parsed source JSON | 4 MiB |
|
||||
| Extended metadata header | 64 KiB |
|
||||
| Expansion ratio | 200:1 |
|
||||
| Combined import payload | 128 MiB |
|
||||
|
||||
Normalized/materialized bytes are held in memory within the import budget;
|
||||
bounded snapshots and spool buffers add overhead. PackageLimits and
|
||||
CaseBuilderLimits apply independently. Inputs must be complete trusted local
|
||||
files. Original TAR bytes preserve empty directories and unknown metadata.
|
||||
|
||||
## Mapping and provenance
|
||||
|
||||
wowdiag descriptions become supplied observations, expected_behavior stays
|
||||
supplied expected text, reproduction_notes become comments, and aggregate bug
|
||||
title/description stay a separate record. Reproductions preserve observations,
|
||||
the whole steps string as one verbatim element, result and explicit location/focus
|
||||
fields. Tester_note data_json text retains the original field pointer and decoded
|
||||
JSON pointer. Server events, snapshots and nested contexts stay raw references.
|
||||
Explicit typed target/quest/focus values become entities; prose and numeric hits
|
||||
never generate entities or hypotheses. Duplicate IDs/unresolved links are rejected;
|
||||
a reproduction can belong through its report when direct bug_id is null.
|
||||
|
||||
Missing expected behavior/steps remain missing. Completeness is not invented.
|
||||
Timezone-less times remain unchanged, with SOURCE_TIME_UNQUALIFIED; no offset is
|
||||
borrowed from generated_at. Source authors are preserved. Database/source/runtime
|
||||
and copied attachment files receive generic roles; unknown files are unclassified.
|
||||
|
||||
wowsrc context remains opaque source/build/custom/change/environment and schema
|
||||
references. A copied development file requires a safe existing package_path and
|
||||
matching size; copied=false retains its reason as SOURCE_NOT_COPIED. Absolute
|
||||
server file paths are metadata, never local I/O targets. Findings, rankings and
|
||||
location-script status remain source statements without causal interpretation.
|
||||
|
||||
Generic v1 contracts have additive optional origin metadata: raw source_ref,
|
||||
JSON Pointer, archive_ref/member_name, decoded pointer and unchanged timestamp.
|
||||
CaseBuilder propagates these into human/fact/entity/reproduction provenance and
|
||||
checks referenced files exist. Summaries show original source/time. Generic report
|
||||
data_quality_issues preserve source gaps. These extensions are project-independent;
|
||||
older v1 documents remain valid. Normalized report-document counts include aggregate
|
||||
and quality records, so differ from original database report counts.
|
||||
|
||||
## Import API and layout
|
||||
|
||||
```python
|
||||
from viano_bug_analysis.profiles.vcore import VcoreImporter, AdapterError
|
||||
try:
|
||||
VcoreImporter().create(
|
||||
'diagnostic.tar.gz', 'incoming.vbcase.zip', wowsrc='context.tar.gz',
|
||||
package_id='caller-owned-stable-id', case_id='caller-owned-case-id',
|
||||
)
|
||||
except AdapterError as error:
|
||||
print(error.code, error.path, str(error))
|
||||
```
|
||||
|
||||
Initial identity is supplied explicitly, never derived from filenames or silently
|
||||
equated with source bug ID. Initial state is NEW, generation/revision zero and
|
||||
project wow/vcore. Neutral PackageFactory creates a validated ZIP atomically with
|
||||
no clobber from a caller-supplied manifest/bytes; it has no project semantics.
|
||||
|
||||
Each source snapshot is stored at original/vcore/imports/<pair-hash>/<tool>/ with
|
||||
archive.tar.gz and members/<original-member-path>. Both compressed originals and
|
||||
every ordinary member survive byte for byte. Generic views, input index and
|
||||
versioned import.json receipt are at case/inputs/vcore/<pair-hash>/. Receipt records
|
||||
source versions/hashes, bug ID and issues. Manifest pointers select case_inputs
|
||||
and vcore_import. The pair hash uses archive SHA-256 values; renaming does not
|
||||
change it, while repacking is a distinct source snapshot.
|
||||
|
||||
prepare returns an ImportPlan without writing. augment(source, destination,
|
||||
wowdiag=..., wowsrc=...) uses PackageWriter, preserves old contents and identity,
|
||||
increments generation and updates manifest indices. Other profiles/bugs, content
|
||||
collisions and identical repeated imports are rejected (IMPORT_EXISTS: use the
|
||||
existing package). No automatic reopening occurs; new evidence after completion
|
||||
requires the explicit lifecycle transition to a new analysis revision.
|
||||
Existing explicit input roles and index extensions are merged rather than lost.
|
||||
When the same diagnostic archive is supplemented with wowsrc, unchanged normalized
|
||||
source rows are reused by archive hash, original member/pointer and equivalent
|
||||
content. This avoids duplicating human observations; all original artifacts remain.
|
||||
|
||||
## Regression tests
|
||||
|
||||
Portable sanitized synthetic fixtures cover mappings, preservation, malformed
|
||||
archives, links/path/header/size limits, pair consistency, missing copied files,
|
||||
times and CaseBuilder roundtrips. No original binaries or credentials are committed.
|
||||
The complete supplied BUG-20 pair has an opt-in test:
|
||||
|
||||
```powershell
|
||||
$env:VBA_WOWDIAG_SAMPLE = 'path/to/diagnostic.tar.gz'
|
||||
$env:VBA_WOWSRC_SAMPLE = 'path/to/source-context.tar.gz'
|
||||
.\.venv\Scripts\python.exe -m pytest tests/profiles/vcore/test_real_package.py -q
|
||||
```
|
||||
|
||||
It checks every raw member byte and unchanged original hashes, three observations,
|
||||
missing expected behavior, 103 database/93 source references, valid output and
|
||||
idempotent repeated construction. Without paths it is skipped; portable tests
|
||||
remain runnable offline.
|
||||
@@ -150,6 +150,13 @@ class CaseAssembler:
|
||||
add(path, role)
|
||||
reports = [(path, load(path, "report")) for path, role in roles if role == "report"]
|
||||
reproductions = [(path, load(path, "reproduction")) for path, role in roles if role == "reproduction"]
|
||||
for _, document in reports + reproductions:
|
||||
nodes = [document] + [node for field in ("observations", "expected_behavior", "comments", "tester_hypotheses", "facts", "entities", "data_quality_issues") for node in document.get(field, [])]
|
||||
for node in nodes:
|
||||
if "origin" in node:
|
||||
for field in ("source_ref", "archive_ref"):
|
||||
if node["origin"].get(field) is not None:
|
||||
require_file(reader, node["origin"][field])
|
||||
for _, document in reproductions:
|
||||
for path in document.get("attachments", []):
|
||||
add(path, "attachment")
|
||||
@@ -174,11 +181,15 @@ class CaseAssembler:
|
||||
identity = (entity["type"], entity_id)
|
||||
record = entities.setdefault(identity, {"type": identity[0], "id": identity[1], "roles": [], "provenance": []})
|
||||
record["roles"] = sorted(set(record["roles"]) | set(entity.get("roles", [])))
|
||||
record["provenance"].append(provenance(path, f"/entities/{index}", "report" if document["format"] == "viano-bug-report" else "reproduction_record"))
|
||||
record["provenance"].append(provenance(path, f"/entities/{index}", "report" if document["format"] == "viano-bug-report" else "reproduction_record", entity))
|
||||
issues = []
|
||||
|
||||
def issue(code, message, refs=None):
|
||||
issues.append({"code": code, "message": message, "refs": refs or []})
|
||||
for path, report in reports:
|
||||
for index, item in enumerate(report.get("data_quality_issues", [])):
|
||||
source = item.get("origin", {"source_ref": path, "json_pointer": f"/data_quality_issues/{index}"})
|
||||
issue(item["code"], item["message"], [{"source_ref": source["source_ref"], "json_pointer": source["json_pointer"]}])
|
||||
|
||||
if not index_ref:
|
||||
issue("MISSING_INPUT_INDEX", "No explicit case input index was supplied")
|
||||
@@ -215,7 +226,8 @@ class CaseAssembler:
|
||||
"format": "viano-case", "format_version": 1, "builder_version": "1",
|
||||
"case": {"case_id": manifest["case_id"], "package_id": manifest["package_id"], "analysis_revision": int(manifest["analysis_revision"]),
|
||||
"source_generation": int(manifest["package_generation"]), "project": copy.deepcopy(manifest["project"])},
|
||||
"bug": {"reports": [{"source_ref": path, "title": document.get("title"), "summary": document.get("summary")} for path, document in reports]},
|
||||
"bug": {"reports": [{"source_ref": path, "title": document.get("title"), "summary": document.get("summary"),
|
||||
**({"origin": copy.deepcopy(document["origin"])} if "origin" in document else {})} for path, document in reports]},
|
||||
"human_observations": {"observations": observations, "comments": comments, "tester_hypotheses": hypotheses},
|
||||
"expected_behavior": expected,
|
||||
"reproduction": {"records": reproduction_records},
|
||||
|
||||
@@ -3,8 +3,11 @@ import copy
|
||||
|
||||
def provenance(path: str, pointer: str, kind: str, statement: dict | None = None) -> dict:
|
||||
value = statement or {}
|
||||
return {"kind": kind, "source_ref": path, "json_pointer": pointer,
|
||||
"author": value.get("author"), "observed_at": value.get("observed_at")}
|
||||
result = {"kind": kind, "source_ref": path, "json_pointer": pointer,
|
||||
"author": value.get("author"), "observed_at": value.get("observed_at")}
|
||||
if "origin" in value:
|
||||
result["origin"] = copy.deepcopy(value["origin"])
|
||||
return result
|
||||
|
||||
|
||||
def collect_statements(documents: list[tuple[str, dict]], field: str, prefix: str) -> list[dict]:
|
||||
|
||||
@@ -8,12 +8,15 @@ from .report_collector import provenance
|
||||
def collect_reproductions(documents: list[tuple[str, dict]], inline_limit: int) -> tuple[list[dict], list[dict]]:
|
||||
records, facts = [], []
|
||||
for path, document in documents:
|
||||
records.append({"id": f"R{len(records) + 1:04d}", "source_ref": path,
|
||||
record = {"id": f"R{len(records) + 1:04d}", "source_ref": path,
|
||||
"steps": list(document.get("steps", [])), "events_ref": document.get("events_ref"),
|
||||
"attachments": list(document.get("attachments", []))})
|
||||
"attachments": list(document.get("attachments", []))}
|
||||
if "origin" in document:
|
||||
record["origin"] = copy.deepcopy(document["origin"])
|
||||
records.append(record)
|
||||
for index, fact in enumerate(document.get("facts", [])):
|
||||
record = {"id": f"F{len(facts) + 1:04d}", "name": fact["name"],
|
||||
"provenance": provenance(path, f"/facts/{index}", "reproduction_record")}
|
||||
"provenance": provenance(path, f"/facts/{index}", "reproduction_record", fact)}
|
||||
if len(json_bytes(fact["value"])) > inline_limit:
|
||||
record["value_ref"] = {"source_ref": path, "json_pointer": f"/facts/{index}/value"}
|
||||
else:
|
||||
|
||||
@@ -11,6 +11,17 @@ def _literal(value) -> str:
|
||||
return text.replace("\r", "\\r").replace("\n", "<br>")
|
||||
|
||||
|
||||
def _origin_lines(origin: dict | None) -> list[str]:
|
||||
if origin is None:
|
||||
return []
|
||||
result = [f" Original source: {_literal(origin['source_ref'])} {_literal(origin['json_pointer'])}"]
|
||||
if "decoded_json_pointer" in origin:
|
||||
result.append(f" Decoded JSON pointer: {_literal(origin['decoded_json_pointer'])}")
|
||||
if "source_timestamp" in origin:
|
||||
result.append(f" Source timestamp (unchanged): {_literal(origin['source_timestamp'])}")
|
||||
return result
|
||||
|
||||
|
||||
def render_summary(case: dict) -> str:
|
||||
"""Pure presentation of the canonical case, with no additional interpretation."""
|
||||
validate(case, "case")
|
||||
@@ -22,6 +33,7 @@ def render_summary(case: dict) -> str:
|
||||
lines += [f"- Source: {_literal(report['source_ref'])}",
|
||||
f" Title: {_literal(report['title']) if report['title'] is not None else '(not supplied)'}",
|
||||
f" Summary: {_literal(report['summary']) if report['summary'] is not None else '(not supplied)'}"]
|
||||
lines.extend(_origin_lines(report.get("origin")))
|
||||
for label, records in (("Direct human observations", case["human_observations"]["observations"]),
|
||||
("Tester comments", case["human_observations"]["comments"]),
|
||||
("Tester hypotheses (supplied)", case["human_observations"]["tester_hypotheses"]),
|
||||
@@ -33,11 +45,13 @@ def render_summary(case: dict) -> str:
|
||||
f" Source: {_literal(source['source_ref'])} {_literal(source['json_pointer'])}",
|
||||
f" Author: {_literal(source['author']) if source['author'] is not None else '(not supplied)'}",
|
||||
f" Observed at: {_literal(source['observed_at']) if source['observed_at'] is not None else '(not supplied)'}"]
|
||||
lines.extend(_origin_lines(source.get("origin")))
|
||||
if "claim" in record:
|
||||
lines.append(f" Claim: {_literal(record['claim']['key'])} = {_literal(json.dumps(record['claim']['value'], sort_keys=True, ensure_ascii=False))}")
|
||||
lines += ["", f"Expected-behavior completeness: {case['expected_behavior']['completeness']}", "", "## Reproductions", ""]
|
||||
for record in case["reproduction"]["records"]:
|
||||
lines.append(f"- {_literal(record['id'])}: {_literal(record['source_ref'])}")
|
||||
lines.extend(_origin_lines(record.get("origin")))
|
||||
for step in record["steps"]:
|
||||
lines.append(f" Step: {_literal(step)}")
|
||||
if record["events_ref"]:
|
||||
@@ -49,6 +63,7 @@ def render_summary(case: dict) -> str:
|
||||
lines.append(f"- {_literal(entity['type'])} / {_literal(entity['id'])}; roles: {_literal(', '.join(entity['roles']))}")
|
||||
for source in entity["provenance"]:
|
||||
lines.append(f" Source: {_literal(source['source_ref'])} {_literal(source['json_pointer'])}")
|
||||
lines.extend(_origin_lines(source.get("origin")))
|
||||
lines += ["", "## Facts supplied by reproduction records", ""]
|
||||
for fact in case["runtime_facts"]["facts"]:
|
||||
source = fact["provenance"]
|
||||
@@ -59,6 +74,7 @@ def render_summary(case: dict) -> str:
|
||||
value = _literal(json.dumps(fact['value'], sort_keys=True, ensure_ascii=False))
|
||||
lines.append(f"- {_literal(fact['id'])}: {_literal(fact['name'])} = {value}")
|
||||
lines.append(f" Source: {_literal(source['source_ref'])} {_literal(source['json_pointer'])}")
|
||||
lines.extend(_origin_lines(source.get("origin")))
|
||||
for label, refs in (("Runtime data", case["runtime_facts"]["raw_refs"]), ("Database context", case["database_context"]["refs"]),
|
||||
("Source context", case["source_context"]["refs"]), ("Supplied research", case["research"]["refs"]),
|
||||
("Prior cases", case["inputs"]["prior_cases"]), ("Unclassified inputs", case["inputs"]["unclassified"])):
|
||||
|
||||
@@ -3,6 +3,7 @@ from .limits import PackageLimits
|
||||
from .reader import PackageReader
|
||||
from .validator import PackageValidator
|
||||
from .writer import PackageWriter
|
||||
from .factory import PackageFactory
|
||||
|
||||
__all__ = ["PackageReader", "PackageValidator", "PackageLimits", "PackageReadError",
|
||||
"ValidationIssue", "ValidationResult", "PackageWriter", "PackageWriteError"]
|
||||
"ValidationIssue", "ValidationResult", "PackageWriter", "PackageWriteError", "PackageFactory"]
|
||||
@@ -0,0 +1,66 @@
|
||||
import copy
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import tempfile
|
||||
import warnings
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
from .errors import PackageWriteError
|
||||
from .manifest import safe_path
|
||||
from .validator import ARCHIVE_ERRORS, PackageValidator
|
||||
|
||||
|
||||
class PackageFactory:
|
||||
"""Create an initial v1 package from caller-owned identity and supplied bytes."""
|
||||
|
||||
def __init__(self, limits=None):
|
||||
self.validator = PackageValidator(limits)
|
||||
|
||||
def create(self, destination, manifest: dict, artifacts: dict[str, bytes]):
|
||||
temporary = None
|
||||
published = False
|
||||
target = Path(destination)
|
||||
try:
|
||||
if target.exists() or target.is_symlink():
|
||||
raise PackageWriteError(self.validator._failure("TARGET_EXISTS", "Initial package target already exists"))
|
||||
document = copy.deepcopy(manifest)
|
||||
if document.get("package_generation") != 0:
|
||||
raise PackageWriteError(self.validator._failure("INITIAL_GENERATION", "Initial generation must be zero"))
|
||||
for path, data in artifacts.items():
|
||||
if not safe_path(path) or path == "manifest.json" or not isinstance(data, bytes):
|
||||
raise PackageWriteError(self.validator._failure("ARTIFACT_INVALID", "Unsafe path or non-byte artifact", path))
|
||||
if len(data) > self.validator.limits.max_file_bytes:
|
||||
raise PackageWriteError(self.validator._failure("FILE_LIMIT", "Initial artifact exceeds configured limit", path))
|
||||
document.setdefault("integrity", {}).setdefault(path, {}).update(size=len(data), sha256=hashlib.sha256(data).hexdigest())
|
||||
data = json.dumps(document, sort_keys=True, ensure_ascii=False, allow_nan=False).encode("utf-8")
|
||||
if len(data) > min(self.validator.limits.max_manifest_bytes, self.validator.limits.max_file_bytes):
|
||||
raise PackageWriteError(self.validator._failure("FILE_LIMIT", "Initial manifest exceeds configured limit"))
|
||||
if len(artifacts) + 1 > self.validator.limits.max_entries or sum(map(len, artifacts.values())) + len(data) > min(self.validator.limits.max_total_bytes, self.validator.limits.max_archive_bytes):
|
||||
raise PackageWriteError(self.validator._failure("PACKAGE_LIMIT", "Initial package exceeds configured limits"))
|
||||
with tempfile.NamedTemporaryFile(dir=target.parent, prefix=".vbcase-", suffix=".tmp", delete=False) as stream:
|
||||
temporary = Path(stream.name)
|
||||
with zipfile.ZipFile(stream, "w", compression=zipfile.ZIP_STORED) as archive:
|
||||
for path, payload in artifacts.items():
|
||||
archive.writestr(path, payload)
|
||||
archive.writestr("manifest.json", data)
|
||||
stream.flush()
|
||||
os.fsync(stream.fileno())
|
||||
result = self.validator.validate(temporary)
|
||||
if not result.valid:
|
||||
raise PackageWriteError(result)
|
||||
os.link(temporary, target)
|
||||
published = True
|
||||
return result
|
||||
except ARCHIVE_ERRORS + (TypeError, KeyError, RecursionError, AttributeError) as error:
|
||||
raise PackageWriteError(self.validator._failure("CREATE_ERROR", str(error))) from error
|
||||
finally:
|
||||
if temporary is not None:
|
||||
try:
|
||||
temporary.unlink(missing_ok=True)
|
||||
except OSError as error:
|
||||
if published:
|
||||
warnings.warn(f"Package created but staging cleanup failed: {error}", ResourceWarning)
|
||||
else:
|
||||
raise PackageWriteError(self.validator._failure("CLEANUP_ERROR", str(error))) from error
|
||||
@@ -0,0 +1 @@
|
||||
"""Project-specific adapters, separate from deterministic package/case foundations."""
|
||||
@@ -0,0 +1,7 @@
|
||||
from .archive_reader import ArchiveLimits, ArchiveReader
|
||||
from .errors import AdapterError
|
||||
from .importer import VcoreImporter, ImportPlan
|
||||
from .wowdiag_adapter import WowdiagAdapter
|
||||
from .wowsrc_adapter import WowsrcAdapter
|
||||
|
||||
__all__ = ["ArchiveLimits", "ArchiveReader", "AdapterError", "VcoreImporter", "ImportPlan", "WowdiagAdapter", "WowsrcAdapter"]
|
||||
@@ -0,0 +1,160 @@
|
||||
import gzip
|
||||
import io
|
||||
import os
|
||||
import tarfile
|
||||
import tempfile
|
||||
import zlib
|
||||
from dataclasses import dataclass
|
||||
|
||||
from viano_bug_analysis.package.manifest import parse_manifest, safe_path
|
||||
from viano_bug_analysis.package.validator import CHUNK_SIZE
|
||||
|
||||
from .errors import AdapterError
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ArchiveLimits:
|
||||
max_archive_bytes: int = 128 * 1024**2
|
||||
max_tar_bytes: int = 256 * 1024**2
|
||||
max_file_bytes: int = 64 * 1024**2
|
||||
max_entries: int = 10_000
|
||||
max_json_bytes: int = 4 * 1024**2
|
||||
max_header_bytes: int = 64 * 1024
|
||||
max_ratio: int = 200
|
||||
max_import_bytes: int = 128 * 1024**2
|
||||
|
||||
def __post_init__(self):
|
||||
if any(isinstance(value, bool) or not isinstance(value, int) or value <= 0 for value in vars(self).values()):
|
||||
raise ValueError("Archive limits must be positive integers")
|
||||
|
||||
|
||||
class ArchiveReader:
|
||||
"""Validate gzip fully and read TAR members without filesystem extraction."""
|
||||
|
||||
def __init__(self, path, limits: ArchiveLimits | None = None):
|
||||
self.path = path
|
||||
self.limits = limits or ArchiveLimits()
|
||||
self._original = self._spool = self._tar = None
|
||||
self._raw = None
|
||||
self.members = {}
|
||||
|
||||
def __enter__(self):
|
||||
try:
|
||||
self._original = open(self.path, "rb")
|
||||
archive_size = os.fstat(self._original.fileno()).st_size
|
||||
if archive_size > self.limits.max_archive_bytes:
|
||||
raise AdapterError("ARCHIVE_LIMIT", "Compressed source exceeds configured limit")
|
||||
self._raw = self._original.read(self.limits.max_archive_bytes + 1)
|
||||
if len(self._raw) > self.limits.max_archive_bytes:
|
||||
raise AdapterError("ARCHIVE_LIMIT", "Source archive changed or exceeds configured limit")
|
||||
archive_size = len(self._raw)
|
||||
self._spool = tempfile.SpooledTemporaryFile(max_size=4 * 1024**2)
|
||||
size = 0
|
||||
with gzip.GzipFile(fileobj=io.BytesIO(self._raw)) as compressed:
|
||||
while chunk := compressed.read(CHUNK_SIZE):
|
||||
size += len(chunk)
|
||||
if size > self.limits.max_tar_bytes or size > max(archive_size, 1) * self.limits.max_ratio:
|
||||
raise AdapterError("EXPANSION_LIMIT", "TAR expansion exceeds configured policy")
|
||||
self._spool.write(chunk)
|
||||
self._spool.seek(0)
|
||||
header_limit = self.limits.max_header_bytes
|
||||
|
||||
class BoundedTarInfo(tarfile.TarInfo):
|
||||
def _proc_member(self, archive):
|
||||
metadata_types = (tarfile.XHDTYPE, tarfile.XGLTYPE, tarfile.SOLARIS_XHDTYPE, tarfile.GNUTYPE_LONGNAME, tarfile.GNUTYPE_LONGLINK)
|
||||
if self.type not in metadata_types and (self.type == tarfile.GNUTYPE_SPARSE or not (self.isfile() or self.isdir())):
|
||||
raise AdapterError("SPECIAL_MEMBER", "Special TAR member rejected before extended processing", self.name)
|
||||
return super()._proc_member(archive)
|
||||
|
||||
def _proc_pax(self, *args, **kwargs):
|
||||
if self.size < 0 or self.size > header_limit:
|
||||
raise AdapterError("HEADER_LIMIT", "PAX metadata exceeds configured limit")
|
||||
return super()._proc_pax(*args, **kwargs)
|
||||
|
||||
def _proc_gnulong(self, *args, **kwargs):
|
||||
if self.size < 0 or self.size > header_limit:
|
||||
raise AdapterError("HEADER_LIMIT", "GNU metadata exceeds configured limit")
|
||||
return super()._proc_gnulong(*args, **kwargs)
|
||||
|
||||
self._tar = tarfile.open(fileobj=self._spool, mode="r:", tarinfo=BoundedTarInfo)
|
||||
names = {}
|
||||
for member in self._tar:
|
||||
if len(names) >= self.limits.max_entries:
|
||||
raise AdapterError("ENTRY_LIMIT", "Too many TAR entries")
|
||||
name = member.name
|
||||
if not safe_path(name, directory=member.isdir()):
|
||||
raise AdapterError("UNSAFE_PATH", "Unsafe TAR member path", name)
|
||||
if not (member.isfile() or member.isdir()) or member.issparse():
|
||||
raise AdapterError("SPECIAL_MEMBER", "Links, sparse files and special TAR entries are unsupported", name)
|
||||
if member.size < 0 or member.size > self.limits.max_file_bytes or (member.isdir() and member.size):
|
||||
raise AdapterError("FILE_LIMIT", "TAR member exceeds configured limit", name)
|
||||
normalized = name.rstrip("/").casefold()
|
||||
if normalized in names:
|
||||
raise AdapterError("MEMBER_COLLISION", "Duplicate or case-colliding TAR entry", name)
|
||||
names[normalized] = member.isdir()
|
||||
if member.isfile():
|
||||
self.members[name] = member
|
||||
for name in names:
|
||||
parts = name.split("/")
|
||||
if any(names.get("/".join(parts[:index])) is False for index in range(1, len(parts))):
|
||||
raise AdapterError("MEMBER_COLLISION", "A TAR file is also used as a directory", name)
|
||||
candidates = [name for name in self.members if name.split("/")[-1] == "manifest.json" and len(name.split("/")) <= 2]
|
||||
if len(candidates) != 1:
|
||||
raise AdapterError("MANIFEST_AMBIGUOUS", "Expected exactly one package-level source manifest")
|
||||
self.manifest_member = candidates[0]
|
||||
self.root = self.manifest_member.removesuffix("manifest.json")
|
||||
self.manifest = self.read_json("manifest.json")
|
||||
if not isinstance(self.manifest, dict):
|
||||
raise AdapterError("SOURCE_MANIFEST", "Source manifest must be an object")
|
||||
return self
|
||||
except AdapterError:
|
||||
self.close()
|
||||
raise
|
||||
except (OSError, EOFError, tarfile.TarError, zlib.error, ValueError, RecursionError) as error:
|
||||
self.close()
|
||||
raise AdapterError("ARCHIVE_INVALID", str(error)) from error
|
||||
|
||||
def member_name(self, relative: str) -> str:
|
||||
if not safe_path(relative) or self.root + relative not in self.members:
|
||||
raise AdapterError("SOURCE_REFERENCE", "Referenced source member is absent or unsafe", relative)
|
||||
return self.root + relative
|
||||
|
||||
def read_member(self, name: str, limit: int | None = None) -> bytes:
|
||||
if self._tar is None:
|
||||
raise RuntimeError("Use ArchiveReader in a with block")
|
||||
if name not in self.members:
|
||||
raise AdapterError("SOURCE_REFERENCE", "TAR member not found", name)
|
||||
limit = min(limit or self.limits.max_file_bytes, self.limits.max_file_bytes)
|
||||
if self.members[name].size > limit:
|
||||
raise AdapterError("FILE_LIMIT", "Source member exceeds configured read limit", name)
|
||||
try:
|
||||
with self._tar.extractfile(self.members[name]) as stream:
|
||||
data = stream.read(limit + 1)
|
||||
if len(data) != self.members[name].size or len(data) > limit:
|
||||
raise AdapterError("MEMBER_TRUNCATED", "TAR member length differs from header", name)
|
||||
return data
|
||||
except (OSError, EOFError, tarfile.TarError, ValueError) as error:
|
||||
raise AdapterError("ARCHIVE_INVALID", str(error), name) from error
|
||||
|
||||
def read_json(self, relative: str):
|
||||
name = self.member_name(relative)
|
||||
try:
|
||||
return parse_manifest(self.read_member(name, self.limits.max_json_bytes))
|
||||
except (ValueError, RecursionError) as error:
|
||||
raise AdapterError("SOURCE_JSON", str(error), name) from error
|
||||
|
||||
def original_bytes(self) -> bytes:
|
||||
if self._raw is None:
|
||||
raise RuntimeError("ArchiveReader is closed")
|
||||
return self._raw
|
||||
|
||||
def close(self):
|
||||
for stream in (self._tar, self._spool, self._original):
|
||||
if stream is not None:
|
||||
stream.close()
|
||||
self._tar = self._spool = self._original = None
|
||||
self._raw = None
|
||||
self.members = {}
|
||||
|
||||
def __exit__(self, *args):
|
||||
self.close()
|
||||
@@ -0,0 +1,105 @@
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
from viano_bug_analysis.lifecycle.storage import json_bytes
|
||||
from viano_bug_analysis.package.manifest import FORMAT_CHECKER
|
||||
|
||||
from .errors import AdapterError
|
||||
|
||||
|
||||
@dataclass
|
||||
class AdaptedInputs:
|
||||
additions: dict[str, bytes] = field(default_factory=dict)
|
||||
reports: list[str] = field(default_factory=list)
|
||||
reproductions: list[str] = field(default_factory=list)
|
||||
attachments: list[str] = field(default_factory=list)
|
||||
context: dict = field(default_factory=lambda: {key: [] for key in ("runtime", "database", "source", "research")})
|
||||
issues: list[dict] = field(default_factory=list)
|
||||
|
||||
|
||||
def check_manifest(reader, tool: str, version: int):
|
||||
manifest = reader.manifest
|
||||
if manifest.get("tool") != tool or type(manifest.get("format_version")) is not int or manifest["format_version"] != version or manifest.get("command") != "bug":
|
||||
raise AdapterError("SOURCE_FORMAT", f"Expected {tool} bug package format {version}")
|
||||
if type(manifest.get("bug_id")) is not int or manifest["bug_id"] < 0 or not isinstance(manifest.get("tool_version"), str) or not manifest["tool_version"].strip():
|
||||
raise AdapterError("SOURCE_MANIFEST", "Source requires bug_id and tool_version")
|
||||
if not isinstance(manifest.get("warnings", []), list) or any(not isinstance(value, str) for value in manifest.get("warnings", [])):
|
||||
raise AdapterError("SOURCE_MANIFEST", "Source warnings must be a list of strings")
|
||||
if manifest.get("core_revision") is not None and not isinstance(manifest["core_revision"], str):
|
||||
raise AdapterError("SOURCE_MANIFEST", "core_revision must be text or null")
|
||||
|
||||
|
||||
def payload_ref(reader, prefix: str, relative: str) -> str:
|
||||
return prefix + "/members/" + reader.member_name(relative)
|
||||
|
||||
|
||||
def origin(reader, prefix, relative, pointer, timestamp=None, decoded=None):
|
||||
result = {"source_ref": payload_ref(reader, prefix, relative), "json_pointer": pointer,
|
||||
"archive_ref": prefix + "/archive.tar.gz", "member_name": reader.member_name(relative)}
|
||||
if timestamp is not None:
|
||||
result["source_timestamp"] = timestamp
|
||||
if decoded is not None:
|
||||
result["decoded_json_pointer"] = decoded
|
||||
return result
|
||||
|
||||
|
||||
def materialize(reader, prefix, role_for) -> AdaptedInputs:
|
||||
result = AdaptedInputs()
|
||||
original = reader.original_bytes()
|
||||
result.additions[prefix + "/archive.tar.gz"] = original
|
||||
total = len(original)
|
||||
for name in sorted(reader.members):
|
||||
if total + reader.members[name].size > reader.limits.max_import_bytes:
|
||||
raise AdapterError("IMPORT_LIMIT", "Materialized source exceeds configured memory/import budget")
|
||||
data = reader.read_member(name)
|
||||
total += len(data)
|
||||
if total > reader.limits.max_import_bytes:
|
||||
raise AdapterError("IMPORT_LIMIT", "Materialized source exceeds configured memory/import budget")
|
||||
path = prefix + "/members/" + name
|
||||
result.additions[path] = data
|
||||
logical = name.removeprefix(reader.root) if name.startswith(reader.root) else None
|
||||
role = role_for(logical) if logical is not None else None
|
||||
if role == "attachment":
|
||||
result.attachments.append(path)
|
||||
elif role is not None:
|
||||
result.context[role].append(path)
|
||||
return result
|
||||
|
||||
|
||||
def rows(reader, relative):
|
||||
value = reader.read_json(relative)
|
||||
if not isinstance(value, list) or any(not isinstance(row, dict) for row in value):
|
||||
raise AdapterError("SOURCE_ROWS", "Expected a JSON array of objects", relative)
|
||||
return value
|
||||
|
||||
|
||||
def statement(text, reader, prefix, relative, pointer, author=None, timestamp=None, decoded=None, issues=None):
|
||||
if text is None or text == "":
|
||||
return None
|
||||
if not isinstance(text, str):
|
||||
raise AdapterError("SOURCE_FIELD", "Expected textual observation", relative + pointer)
|
||||
if not text.strip():
|
||||
return None
|
||||
result = {"text": text, "origin": origin(reader, prefix, relative, pointer, timestamp, decoded)}
|
||||
if author is not None:
|
||||
if not isinstance(author, str):
|
||||
raise AdapterError("SOURCE_FIELD", "Expected textual author")
|
||||
if author.strip():
|
||||
result["author"] = author
|
||||
if timestamp is not None:
|
||||
if not isinstance(timestamp, str):
|
||||
raise AdapterError("SOURCE_FIELD", "Expected timestamp text")
|
||||
if FORMAT_CHECKER.conforms(timestamp, "date-time"):
|
||||
result["observed_at"] = timestamp
|
||||
elif issues is not None:
|
||||
issues.append({"code": "SOURCE_TIME_UNQUALIFIED", "message": "Source time has no usable timezone; retained without conversion", "origin": result["origin"]})
|
||||
return result
|
||||
|
||||
|
||||
def write_document(result, path, document, contract):
|
||||
from viano_bug_analysis.case.contracts import validate
|
||||
from viano_bug_analysis.case.errors import CaseBuildError
|
||||
try:
|
||||
validate(document, contract, path)
|
||||
result.additions[path] = json_bytes(document)
|
||||
except (CaseBuildError, ValueError, TypeError) as error:
|
||||
raise AdapterError("NORMALIZATION_ERROR", str(error), path) from error
|
||||
@@ -0,0 +1,5 @@
|
||||
class AdapterError(Exception):
|
||||
def __init__(self, code: str, message: str, path: str | None = None):
|
||||
self.code = code
|
||||
self.path = path
|
||||
super().__init__(message)
|
||||
@@ -0,0 +1,202 @@
|
||||
import hashlib
|
||||
from contextlib import ExitStack
|
||||
from dataclasses import dataclass, replace
|
||||
from pathlib import Path
|
||||
|
||||
from viano_bug_analysis.case.contracts import validate
|
||||
from viano_bug_analysis.case.errors import CaseBuildError
|
||||
from viano_bug_analysis.lifecycle.job import timestamp
|
||||
from viano_bug_analysis.lifecycle.storage import json_bytes
|
||||
from viano_bug_analysis.package import PackageReader, PackageReadError, PackageWriter, PackageWriteError
|
||||
from viano_bug_analysis.package.factory import PackageFactory
|
||||
from viano_bug_analysis.package.manifest import parse_manifest
|
||||
|
||||
from .archive_reader import ArchiveLimits, ArchiveReader
|
||||
from .common import check_manifest, write_document
|
||||
from .errors import AdapterError
|
||||
from .wowdiag_adapter import WowdiagAdapter
|
||||
from .wowsrc_adapter import WowsrcAdapter
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ImportPlan:
|
||||
additions: dict[str, bytes]
|
||||
index_ref: str
|
||||
receipt_ref: str
|
||||
bug_id: int
|
||||
reproductions_count: int
|
||||
|
||||
|
||||
class VcoreImporter:
|
||||
def __init__(self, archive_limits: ArchiveLimits | None = None, package_limits=None):
|
||||
self.archive_limits = archive_limits or ArchiveLimits()
|
||||
self.package_limits = package_limits
|
||||
|
||||
def _package_json(self, reader, path):
|
||||
with reader._archive.open(path) as stream:
|
||||
data = reader.validator._read_bounded(stream, min(self.archive_limits.max_json_bytes, reader.validator.limits.max_file_bytes))
|
||||
return parse_manifest(data)
|
||||
|
||||
def _merge_inputs(self, reader, plan):
|
||||
old_ref = reader.manifest["artifacts"].get("case_inputs")
|
||||
if not old_ref:
|
||||
return dict(plan.additions)
|
||||
old = self._package_json(reader, old_ref)
|
||||
new = parse_manifest(plan.additions[plan.index_ref])
|
||||
validate(old, "case-inputs")
|
||||
if any(len(old.get(field, [])) != len(set(old.get(field, []))) for field in ("reports", "reproductions", "attachments")):
|
||||
raise AdapterError("SOURCE_INDEX", "Existing input index repeats source documents")
|
||||
old_documents = set(old.get("reports", []) + old.get("reproductions", []) + [old_ref])
|
||||
if any(path not in reader.files for path in old_documents):
|
||||
raise AdapterError("SOURCE_REFERENCE", "Existing input index has unresolved documents")
|
||||
if sum(reader._archive.getinfo(path).file_size for path in old_documents) + sum(map(len, plan.additions.values())) > self.archive_limits.max_import_bytes:
|
||||
raise AdapterError("IMPORT_LIMIT", "Existing structured views plus new import exceed configured budget")
|
||||
# Reuse the same normalized source row when supplementing a diag-only
|
||||
# import with wowsrc. No unrelated existing input roles are dropped.
|
||||
def canonical(value):
|
||||
if isinstance(value, dict):
|
||||
result = {}
|
||||
for key, child in value.items():
|
||||
if key == "value":
|
||||
result[key] = child # arbitrary fact/claim JSON is not metadata
|
||||
elif key == "origin" and isinstance(child, dict):
|
||||
result[key] = {k: v for k, v in child.items() if k not in {"source_ref", "archive_ref"}}
|
||||
elif key == "events_ref" and isinstance(child, str):
|
||||
result[key] = child.split("/members/", 1)[-1]
|
||||
else:
|
||||
result[key] = canonical(child)
|
||||
return result
|
||||
if isinstance(value, list):
|
||||
return [canonical(child) for child in value]
|
||||
return value
|
||||
|
||||
def signature(document, raw_hash):
|
||||
source = document.get("origin")
|
||||
if not isinstance(source, dict) or not raw_hash:
|
||||
return None
|
||||
return raw_hash, source.get("member_name"), source.get("json_pointer"), json_bytes(canonical(document))
|
||||
|
||||
for field, contract in (("reports", "report"), ("reproductions", "reproduction")):
|
||||
known = {}
|
||||
for path in old.get(field, []):
|
||||
document = self._package_json(reader, path)
|
||||
validate(document, contract)
|
||||
archive = document.get("origin", {}).get("archive_ref")
|
||||
hash_ = reader.manifest["integrity"].get(archive, {}).get("sha256")
|
||||
token = signature(document, hash_)
|
||||
if token is not None:
|
||||
known[token] = path
|
||||
selected = []
|
||||
for path in new.get(field, []):
|
||||
document = parse_manifest(plan.additions[path])
|
||||
archive = document.get("origin", {}).get("archive_ref")
|
||||
hash_ = hashlib.sha256(plan.additions[archive]).hexdigest() if archive in plan.additions else None
|
||||
selected.append(known.get(signature(document, hash_), path))
|
||||
new[field] = list(dict.fromkeys(old.get(field, []) + selected))
|
||||
new["attachments"] = list(dict.fromkeys(old.get("attachments", []) + new.get("attachments", [])))
|
||||
context = dict(old.get("context", {}))
|
||||
for role, paths in new.get("context", {}).items():
|
||||
context[role] = list(dict.fromkeys(context.get(role, []) + paths))
|
||||
new["context"] = context
|
||||
merged = old | new
|
||||
validate(merged, "case-inputs")
|
||||
return dict(plan.additions) | {plan.index_ref: json_bytes(merged)}
|
||||
|
||||
def prepare(self, wowdiag, wowsrc=None) -> ImportPlan:
|
||||
try:
|
||||
if sum(Path(path).stat().st_size for path in (wowdiag, wowsrc) if path is not None) > self.archive_limits.max_import_bytes:
|
||||
raise AdapterError("IMPORT_LIMIT", "Combined original archives exceed configured budget")
|
||||
with ExitStack() as stack:
|
||||
diag = stack.enter_context(ArchiveReader(wowdiag, self.archive_limits))
|
||||
check_manifest(diag, "wowdiag", 2)
|
||||
src = stack.enter_context(ArchiveReader(wowsrc, self.archive_limits)) if wowsrc is not None else None
|
||||
if src:
|
||||
check_manifest(src, "wowsrc", 7)
|
||||
embedded = src.read_json("input/wowdiag_manifest.json")
|
||||
if src.manifest["bug_id"] != diag.manifest["bug_id"] or src.manifest.get("core_revision") != diag.manifest.get("core_revision") or json_bytes(embedded) != json_bytes(diag.manifest) or src.manifest.get("input_wowdiag_format") != 2 or src.manifest.get("input_wowdiag_version") != diag.manifest["tool_version"]:
|
||||
raise AdapterError("SOURCE_PAIR_MISMATCH", "wowsrc does not describe the supplied wowdiag package")
|
||||
hashes = [hashlib.sha256(reader.original_bytes()).hexdigest() for reader in (diag, src) if reader is not None]
|
||||
import_id = hashlib.sha256(json_bytes(hashes)).hexdigest()
|
||||
raw_prefix = f"original/vcore/imports/{import_id}"
|
||||
normalized_prefix = f"case/inputs/vcore/{import_id}"
|
||||
inputs = WowdiagAdapter().adapt(diag, raw_prefix + "/wowdiag", normalized_prefix + "/wowdiag")
|
||||
if not diag.manifest.get("core_revision"):
|
||||
inputs.issues.append({"code": "CORE_REVISION_MISSING", "message": "No core revision was supplied; matching revisions cannot be verified"})
|
||||
if src:
|
||||
remaining = self.archive_limits.max_import_bytes - sum(map(len, inputs.additions.values()))
|
||||
if remaining <= 0:
|
||||
raise AdapterError("IMPORT_LIMIT", "No remaining source import budget")
|
||||
src.limits = replace(src.limits, max_import_bytes=remaining)
|
||||
sources = WowsrcAdapter().adapt(src, raw_prefix + "/wowsrc")
|
||||
inputs.additions.update(sources.additions)
|
||||
for role, refs in sources.context.items():
|
||||
inputs.context[role].extend(refs)
|
||||
inputs.issues.extend(sources.issues)
|
||||
if sum(map(len, inputs.additions.values())) > self.archive_limits.max_import_bytes:
|
||||
raise AdapterError("IMPORT_LIMIT", "Combined imported sources exceed configured budget")
|
||||
quality = {"format": "viano-bug-report", "format_version": 1,
|
||||
"data_quality_issues": [{"code": issue["code"], "message": issue["message"], **({"origin": issue["origin"]} if "origin" in issue else {})} for issue in inputs.issues]}
|
||||
quality_ref = normalized_prefix + "/source-quality.json"
|
||||
write_document(inputs, quality_ref, quality, "report")
|
||||
inputs.reports.append(quality_ref)
|
||||
receipt_ref = normalized_prefix + "/import.json"
|
||||
receipt = {"format": "viano-vcore-import", "format_version": 1, "adapter_version": "1", "bug_id": diag.manifest["bug_id"],
|
||||
"sources": [{"tool": reader.manifest["tool"], "tool_version": reader.manifest["tool_version"],
|
||||
"format_version": reader.manifest["format_version"], "sha256": hash_,
|
||||
"archive_ref": raw_prefix + "/" + reader.manifest["tool"] + "/archive.tar.gz"}
|
||||
for reader, hash_ in zip((diag, src), hashes)], "issues": inputs.issues}
|
||||
inputs.additions[receipt_ref] = json_bytes(receipt)
|
||||
inputs.context["runtime"].append(receipt_ref)
|
||||
index_ref = normalized_prefix + "/case-inputs.json"
|
||||
index = {"format": "viano-case-inputs", "format_version": 1, "reports": inputs.reports,
|
||||
"reproductions": inputs.reproductions, "context": inputs.context, "attachments": inputs.attachments}
|
||||
validate(index, "case-inputs")
|
||||
inputs.additions[index_ref] = json_bytes(index)
|
||||
if sum(map(len, inputs.additions.values())) > self.archive_limits.max_import_bytes:
|
||||
raise AdapterError("IMPORT_LIMIT", "Normalized import exceeds configured budget")
|
||||
return ImportPlan(inputs.additions, index_ref, receipt_ref, diag.manifest["bug_id"], len(inputs.reproductions))
|
||||
except (ValueError, TypeError, OSError, CaseBuildError) as error:
|
||||
raise AdapterError("IMPORT_ERROR", str(error)) from error
|
||||
|
||||
def create(self, wowdiag, destination, *, package_id: str, case_id: str, wowsrc=None):
|
||||
plan = self.prepare(wowdiag, wowsrc)
|
||||
created = timestamp()
|
||||
manifest = {"format": "viano-bug-case-package", "format_version": 1, "package_id": package_id, "package_generation": 0,
|
||||
"case_id": case_id, "project": {"type": "wow", "profile": "vcore"}, "created_at": created, "updated_at": created,
|
||||
"state": "NEW", "routing": {"destination": "inbox", "reason": None, "retry_count": 0, "retry_not_before": None},
|
||||
"analysis_revision": 0, "artifacts": {"case": None, "diagnosis": None, "change_request": None,
|
||||
"case_inputs": plan.index_ref, "vcore_import": plan.receipt_ref},
|
||||
"history": {"analysis_revisions": 0, "reproductions": plan.reproductions_count, "evidence": 0, "handoffs": 0, "feedback": 0, "events_ref": None}, "integrity": {}}
|
||||
try:
|
||||
return PackageFactory(self.package_limits).create(destination, manifest, plan.additions)
|
||||
except PackageWriteError as error:
|
||||
raise AdapterError("PACKAGE_ERROR", str(error)) from error
|
||||
|
||||
def augment(self, source, destination, *, wowdiag, wowsrc=None):
|
||||
plan = self.prepare(wowdiag, wowsrc)
|
||||
try:
|
||||
with PackageReader(source, self.package_limits) as reader:
|
||||
manifest = reader.manifest
|
||||
if manifest["project"]["type"] != "wow" or manifest["project"]["profile"] != "vcore":
|
||||
raise AdapterError("PROJECT_MISMATCH", "Vcore inputs require a wow/vcore case")
|
||||
old_receipt = manifest["artifacts"].get("vcore_import")
|
||||
if old_receipt:
|
||||
previous = self._package_json(reader, old_receipt)
|
||||
if not isinstance(previous, dict) or previous.get("format") != "viano-vcore-import" or type(previous.get("format_version")) is not int or previous["format_version"] != 1:
|
||||
raise AdapterError("IMPORT_RECEIPT", "Unsupported or malformed previous import receipt", old_receipt)
|
||||
if previous.get("bug_id") != plan.bug_id:
|
||||
raise AdapterError("BUG_ID_MISMATCH", "Cannot attach a different source bug to this case")
|
||||
additions = self._merge_inputs(reader, plan)
|
||||
for name, data in additions.items():
|
||||
if name in reader.files and reader.read_bytes(name) != data:
|
||||
raise AdapterError("IMPORT_COLLISION", "An existing import artifact has different content", name)
|
||||
if manifest["artifacts"].get("vcore_import") == plan.receipt_ref and manifest["artifacts"].get("case_inputs") == plan.index_ref:
|
||||
raise AdapterError("IMPORT_EXISTS", "Identical source archives are already selected; use the existing package")
|
||||
changes = {name: data for name, data in additions.items() if name not in reader.files}
|
||||
return PackageWriter(self.package_limits).write(source, destination, additions=changes,
|
||||
manifest_updates={"artifacts": {"case_inputs": plan.index_ref, "vcore_import": plan.receipt_ref},
|
||||
"history": {"reproductions": max(manifest["history"]["reproductions"], plan.reproductions_count)}})
|
||||
except (PackageReadError, PackageWriteError) as error:
|
||||
raise AdapterError("PACKAGE_ERROR", str(error)) from error
|
||||
except (OSError, ValueError, TypeError, KeyError, RecursionError, CaseBuildError) as error:
|
||||
raise AdapterError("IMPORT_ERROR", str(error)) from error
|
||||
@@ -0,0 +1,112 @@
|
||||
from viano_bug_analysis.package.manifest import parse_manifest
|
||||
|
||||
from .common import check_manifest, materialize, origin, payload_ref, rows, statement, write_document
|
||||
from .errors import AdapterError
|
||||
|
||||
|
||||
class WowdiagAdapter:
|
||||
def adapt(self, reader, prefix, normalized_prefix=None):
|
||||
check_manifest(reader, "wowdiag", 2)
|
||||
role_for = lambda path: "attachment" if path.startswith("attachments/") else "database" if path.startswith("database/") else "source" if path.startswith("source/") else "runtime" if path.startswith(("bug/", "environment/", "findings/")) or path in {"manifest.json", "findings.json"} else None
|
||||
result = materialize(reader, prefix, role_for)
|
||||
normalized_prefix = normalized_prefix or prefix + "/normalized"
|
||||
bug = reader.read_json("bug/bug.json")
|
||||
if not isinstance(bug, dict) or type(bug.get("bug_id")) is not int or bug["bug_id"] != reader.manifest["bug_id"]:
|
||||
raise AdapterError("BUG_ID_MISMATCH", "Bug record disagrees with source manifest")
|
||||
reports = rows(reader, "bug/reports.json")
|
||||
reproductions = rows(reader, "bug/reproductions.json")
|
||||
events = rows(reader, "bug/reproduction_events.json")
|
||||
if not reports:
|
||||
result.issues.append({"code": "SOURCE_REPORTS_MISSING", "message": "wowdiag supplied no individual bug reports"})
|
||||
for relative, records, id_field in (("bug/reports.json", reports, "report_id"), ("bug/reproductions.json", reproductions, "reproduction_id"), ("bug/reproduction_events.json", events, "event_id")):
|
||||
identities = set()
|
||||
for row in records:
|
||||
identity = row.get(id_field)
|
||||
if type(identity) is not int or identity < 0 or identity in identities:
|
||||
raise AdapterError("SOURCE_ID", "Missing, invalid or duplicate source row ID", relative)
|
||||
identities.add(identity)
|
||||
if row.get("bug_id") is not None and (type(row["bug_id"]) is not int or row["bug_id"] != reader.manifest["bug_id"]):
|
||||
raise AdapterError("BUG_ID_MISMATCH", "Source row belongs to another bug", relative)
|
||||
reproduction_ids = {row["reproduction_id"] for row in reproductions}
|
||||
report_ids = {row["report_id"] for row in reports}
|
||||
if any(event.get("reproduction_id") not in reproduction_ids for event in events) or any(row.get("report_id") is not None and row["report_id"] not in report_ids for row in reproductions):
|
||||
raise AdapterError("SOURCE_RELATION", "Source rows have unresolved reproduction/report links")
|
||||
if any(row.get("bug_id") is None and row.get("report_id") not in report_ids for row in reproductions):
|
||||
raise AdapterError("SOURCE_RELATION", "Reproduction needs a direct bug or linked report association")
|
||||
for index, row in enumerate(reports):
|
||||
normalized = {"format": "viano-bug-report", "format_version": 1,
|
||||
"observations": [], "expected_behavior": [], "comments": [], "tester_hypotheses": [],
|
||||
"origin": origin(reader, prefix, "bug/reports.json", f"/{index}"), "entities": []}
|
||||
title = row.get("title")
|
||||
if isinstance(title, str) and title.strip():
|
||||
normalized["title"] = title
|
||||
for field, destination in (("description", "observations"), ("expected_behavior", "expected_behavior")):
|
||||
value = statement(row.get(field), reader, prefix, "bug/reports.json", f"/{index}/{field}", row.get("character_name"), row.get("created_at"), issues=result.issues)
|
||||
if value:
|
||||
normalized[destination].append(value)
|
||||
# Free-form reproduction notes are supplied prose, not inferred steps.
|
||||
note = statement(row.get("reproduction_notes"), reader, prefix, "bug/reports.json", f"/{index}/reproduction_notes", row.get("character_name"), issues=result.issues)
|
||||
if note:
|
||||
normalized["comments"].append(note)
|
||||
self._entity(normalized, row.get("target_type"), row.get("target_entry"), "reported_target", origin(reader, prefix, "bug/reports.json", f"/{index}/target_entry"))
|
||||
self._entity(normalized, "quest", row.get("quest_id"), "explicit_quest", origin(reader, prefix, "bug/reports.json", f"/{index}/quest_id"))
|
||||
self._entity(normalized, reader.manifest.get("diagnostic_focus_type"), reader.manifest.get("diagnostic_focus_id"), "diagnostic_focus", origin(reader, prefix, "manifest.json", "/diagnostic_focus_id"))
|
||||
path = normalized_prefix + f"/report-{row['report_id']}.json"
|
||||
write_document(result, path, normalized, "report")
|
||||
result.reports.append(path)
|
||||
# Preserve aggregate title/description as an explicitly separate supplied record.
|
||||
aggregate = {"format": "viano-bug-report", "format_version": 1, "origin": origin(reader, prefix, "bug/bug.json", ""), "entities": []}
|
||||
for field, destination in (("title", "title"), ("description", "summary")):
|
||||
value = bug.get(field)
|
||||
if isinstance(value, str) and value.strip():
|
||||
aggregate[destination] = value
|
||||
self._entity(aggregate, bug.get("primary_entity_type"), bug.get("primary_entity_id"), "reported_primary_entity", origin(reader, prefix, "bug/bug.json", "/primary_entity_id"))
|
||||
self._entity(aggregate, reader.manifest.get("diagnostic_focus_type"), reader.manifest.get("diagnostic_focus_id"), "diagnostic_focus", origin(reader, prefix, "manifest.json", "/diagnostic_focus_id"))
|
||||
if reader.root + "bug/runtime_entities.json" in reader.members:
|
||||
for index, entity in enumerate(rows(reader, "bug/runtime_entities.json")):
|
||||
self._entity(aggregate, entity.get("type"), entity.get("id"), "runtime_observed", origin(reader, prefix, "bug/runtime_entities.json", f"/{index}"))
|
||||
path = normalized_prefix + "/bug-record.json"
|
||||
write_document(result, path, aggregate, "report")
|
||||
result.reports.append(path)
|
||||
for index, row in enumerate(reproductions):
|
||||
normalized = {"format": "viano-reproduction", "format_version": 1,
|
||||
"origin": origin(reader, prefix, "bug/reproductions.json", f"/{index}"),
|
||||
"observations": [], "facts": [], "entities": [],
|
||||
"events_ref": payload_ref(reader, prefix, "bug/reproduction_events.json")}
|
||||
steps = row.get("steps")
|
||||
if steps is not None and not isinstance(steps, str):
|
||||
raise AdapterError("SOURCE_FIELD", "Source steps must be text or null")
|
||||
normalized["steps"] = [steps] if steps and steps.strip() else []
|
||||
observed = statement(row.get("observation"), reader, prefix, "bug/reproductions.json", f"/{index}/observation", row.get("tester_character_name"), row.get("finished_at"), issues=result.issues)
|
||||
if observed:
|
||||
normalized["observations"].append(observed)
|
||||
for event_index, event in enumerate(events):
|
||||
if event["reproduction_id"] == row["reproduction_id"] and event.get("source") == "tester" and event.get("event_type") == "tester_note":
|
||||
raw = event.get("data_json")
|
||||
try:
|
||||
decoded = parse_manifest(raw.encode("utf-8")) if isinstance(raw, str) else raw
|
||||
except (ValueError, RecursionError) as error:
|
||||
raise AdapterError("SOURCE_JSON", str(error), "bug/reproduction_events.json") from error
|
||||
if not isinstance(decoded, dict):
|
||||
raise AdapterError("SOURCE_FIELD", "Tester note data must be a JSON object")
|
||||
note = statement(decoded.get("text"), reader, prefix, "bug/reproduction_events.json", f"/{event_index}/data_json", row.get("tester_character_name"), event.get("event_time"), decoded="/text", issues=result.issues)
|
||||
if note:
|
||||
normalized["observations"].append(note)
|
||||
for field in ("result", "map_id", "zone_id", "area_id", "position_x", "position_y", "position_z", "orientation", "quest_id", "diagnostic_focus_type", "diagnostic_focus_id"):
|
||||
if row.get(field) is not None:
|
||||
normalized["facts"].append({"name": field, "value": row[field], "origin": origin(reader, prefix, "bug/reproductions.json", f"/{index}/{field}")})
|
||||
self._entity(normalized, row.get("diagnostic_focus_type"), row.get("diagnostic_focus_id"), "diagnostic_focus", origin(reader, prefix, "bug/reproductions.json", f"/{index}/diagnostic_focus_id"))
|
||||
path = normalized_prefix + f"/reproduction-{row['reproduction_id']}.json"
|
||||
write_document(result, path, normalized, "reproduction")
|
||||
result.reproductions.append(path)
|
||||
for warning in reader.manifest.get("warnings", []):
|
||||
result.issues.append({"code": "SOURCE_WARNING", "message": str(warning), "tool": "wowdiag"})
|
||||
return result
|
||||
|
||||
@staticmethod
|
||||
def _entity(document, type_, id_, role, origin_):
|
||||
if id_ is None or type_ is None:
|
||||
return
|
||||
if not isinstance(type_, str) or not type_.strip() or type(id_) is not int or id_ < 0:
|
||||
raise AdapterError("SOURCE_ENTITY", "Invalid typed source entity")
|
||||
document["entities"].append({"type": type_, "id": id_, "roles": [role], "origin": origin_})
|
||||
@@ -0,0 +1,25 @@
|
||||
from .common import check_manifest, materialize, payload_ref, rows
|
||||
from .errors import AdapterError
|
||||
|
||||
|
||||
class WowsrcAdapter:
|
||||
def adapt(self, reader, prefix):
|
||||
check_manifest(reader, "wowsrc", 7)
|
||||
role_for = lambda path: "database" if path.startswith("database/") else "source" if path.startswith(("source/", "build/", "custom/", "changes/", "environment/", "input/", "handoff/")) or path == "manifest.json" else None
|
||||
result = materialize(reader, prefix, role_for)
|
||||
for field in ("primary_development_context", "supporting_development_context", "context_priority_policy", "source_ranking_policy", "database_schema_index", "database_relationship_map"):
|
||||
if reader.manifest.get(field) is not None:
|
||||
payload_ref(reader, prefix, reader.manifest[field])
|
||||
for record in rows(reader, "source/development_files_index.json"):
|
||||
if record.get("copied") is True:
|
||||
path = record.get("package_path")
|
||||
name = reader.member_name(path)
|
||||
if type(record.get("size")) is not int or record["size"] != reader.members[name].size:
|
||||
raise AdapterError("SOURCE_SIZE", "Copied source size differs from its index", path)
|
||||
elif record.get("copied") is False:
|
||||
result.issues.append({"code": "SOURCE_NOT_COPIED", "message": str(record.get("reason_not_copied", "not copied")), "source_file": record.get("file")})
|
||||
else:
|
||||
raise AdapterError("SOURCE_INDEX", "Development file index requires copied boolean")
|
||||
for warning in reader.manifest.get("warnings", []):
|
||||
result.issues.append({"code": "SOURCE_WARNING", "message": str(warning), "tool": "wowsrc"})
|
||||
return result
|
||||
@@ -104,6 +104,9 @@
|
||||
"type": "null"
|
||||
}
|
||||
]
|
||||
},
|
||||
"origin": {
|
||||
"$ref": "#/$defs/origin"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -221,6 +224,9 @@
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
}
|
||||
},
|
||||
"origin": {
|
||||
"$ref": "#/$defs/origin"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -587,6 +593,9 @@
|
||||
"null"
|
||||
],
|
||||
"format": "date-time"
|
||||
},
|
||||
"origin": {
|
||||
"$ref": "#/$defs/origin"
|
||||
}
|
||||
}
|
||||
},
|
||||
@@ -652,6 +661,37 @@
|
||||
"$ref": "#/$defs/claim"
|
||||
}
|
||||
}
|
||||
},
|
||||
"origin": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"source_ref",
|
||||
"json_pointer"
|
||||
],
|
||||
"properties": {
|
||||
"source_ref": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"json_pointer": {
|
||||
"type": "string",
|
||||
"pattern": "^(|/.*)$"
|
||||
},
|
||||
"archive_ref": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"member_name": {
|
||||
"type": "string"
|
||||
},
|
||||
"decoded_json_pointer": {
|
||||
"type": "string",
|
||||
"pattern": "^(|/.*)$"
|
||||
},
|
||||
"source_timestamp": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -57,6 +57,32 @@
|
||||
"items": {
|
||||
"$ref": "#/$defs/entity"
|
||||
}
|
||||
},
|
||||
"origin": {
|
||||
"$ref": "#/$defs/origin"
|
||||
},
|
||||
"data_quality_issues": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"code",
|
||||
"message"
|
||||
],
|
||||
"properties": {
|
||||
"code": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"message": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"origin": {
|
||||
"$ref": "#/$defs/origin"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
@@ -108,6 +134,9 @@
|
||||
},
|
||||
"claim": {
|
||||
"$ref": "#/$defs/claim"
|
||||
},
|
||||
"origin": {
|
||||
"$ref": "#/$defs/origin"
|
||||
}
|
||||
}
|
||||
},
|
||||
@@ -142,6 +171,40 @@
|
||||
"minLength": 1,
|
||||
"pattern": "\\S"
|
||||
}
|
||||
},
|
||||
"origin": {
|
||||
"$ref": "#/$defs/origin"
|
||||
}
|
||||
}
|
||||
},
|
||||
"origin": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"source_ref",
|
||||
"json_pointer"
|
||||
],
|
||||
"properties": {
|
||||
"source_ref": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"json_pointer": {
|
||||
"type": "string",
|
||||
"pattern": "^(|/.*)$"
|
||||
},
|
||||
"archive_ref": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"member_name": {
|
||||
"type": "string"
|
||||
},
|
||||
"decoded_json_pointer": {
|
||||
"type": "string",
|
||||
"pattern": "^(|/.*)$"
|
||||
},
|
||||
"source_timestamp": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -58,7 +58,10 @@
|
||||
"minLength": 1,
|
||||
"pattern": "\\S"
|
||||
},
|
||||
"value": {}
|
||||
"value": {},
|
||||
"origin": {
|
||||
"$ref": "#/$defs/origin"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
@@ -67,6 +70,9 @@
|
||||
"items": {
|
||||
"$ref": "#/$defs/entity"
|
||||
}
|
||||
},
|
||||
"origin": {
|
||||
"$ref": "#/$defs/origin"
|
||||
}
|
||||
},
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
@@ -118,6 +124,9 @@
|
||||
},
|
||||
"claim": {
|
||||
"$ref": "#/$defs/claim"
|
||||
},
|
||||
"origin": {
|
||||
"$ref": "#/$defs/origin"
|
||||
}
|
||||
}
|
||||
},
|
||||
@@ -152,6 +161,40 @@
|
||||
"minLength": 1,
|
||||
"pattern": "\\S"
|
||||
}
|
||||
},
|
||||
"origin": {
|
||||
"$ref": "#/$defs/origin"
|
||||
}
|
||||
}
|
||||
},
|
||||
"origin": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"source_ref",
|
||||
"json_pointer"
|
||||
],
|
||||
"properties": {
|
||||
"source_ref": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"json_pointer": {
|
||||
"type": "string",
|
||||
"pattern": "^(|/.*)$"
|
||||
},
|
||||
"archive_ref": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"member_name": {
|
||||
"type": "string"
|
||||
},
|
||||
"decoded_json_pointer": {
|
||||
"type": "string",
|
||||
"pattern": "^(|/.*)$"
|
||||
},
|
||||
"source_timestamp": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
import io
|
||||
import json
|
||||
import tarfile
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def make_tar(tmp_path):
|
||||
counter = 0
|
||||
def create(entries, *, root="arbitrary-wrapper", extra=None):
|
||||
nonlocal counter
|
||||
counter += 1
|
||||
path = tmp_path / f"source-{counter}.data"
|
||||
with tarfile.open(path, "w:gz") as archive:
|
||||
for name, payload in entries.items():
|
||||
data = payload if isinstance(payload, bytes) else json.dumps(payload).encode()
|
||||
info = tarfile.TarInfo(root + "/" + name if root else name)
|
||||
info.size = len(data)
|
||||
archive.addfile(info, io.BytesIO(data))
|
||||
for info, data in extra or []:
|
||||
archive.addfile(info, io.BytesIO(data) if data is not None else None)
|
||||
return path
|
||||
return create
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def source_pair(make_tar):
|
||||
diag_manifest = {"tool": "wowdiag", "format_version": 2, "tool_version": "0.10.6", "command": "bug", "bug_id": 20,
|
||||
"core_revision": "verified-source-revision", "diagnostic_focus_type": "quest", "diagnostic_focus_id": 11117, "warnings": []}
|
||||
diag_entries = {
|
||||
"manifest.json": diag_manifest,
|
||||
"bug/bug.json": {"bug_id": 20, "title": "Wolpertinger spawnen nicht", "description": "Supplied aggregate description"},
|
||||
"bug/reports.json": [{"report_id": 32, "bug_id": 20, "description": "Wolpertinger spawnen nicht", "expected_behavior": None,
|
||||
"character_name": "Tester", "created_at": "2026-10-08T13:02:21", "target_type": "creature", "target_entry": 23486, "quest_id": 11117}],
|
||||
"bug/reproductions.json": [{"reproduction_id": 41, "report_id": 32, "bug_id": 20, "result": "reproduced", "steps": None,
|
||||
"observation": "No spawns here; the other location worked", "tester_character_name": "Tester",
|
||||
"finished_at": "2026-10-08T13:04:02", "diagnostic_focus_type": "quest", "diagnostic_focus_id": 11117,
|
||||
"map_id": 0, "zone_id": 1, "area_id": 1}],
|
||||
"bug/reproduction_events.json": [{"event_id": 527, "reproduction_id": 41, "source": "tester", "event_type": "tester_note",
|
||||
"event_time": "2026-10-08T13:02:55", "data_json": json.dumps({"text": "Still no spawns"})}],
|
||||
"database/quest_template.json": [{"ID": 11117}], "source/secondary.json": [{"numeric_hit": 11117}],
|
||||
"unknown.bin": b"\x00\xff", "environment/core.json": {"commit": "verified-source-revision"},
|
||||
}
|
||||
src_manifest = {"tool": "wowsrc", "format_version": 7, "tool_version": "0.7", "command": "bug", "bug_id": 20,
|
||||
"core_revision": "verified-source-revision", "input_wowdiag_format": 2, "input_wowdiag_version": "0.10.6", "warnings": []}
|
||||
src_entries = {"manifest.json": src_manifest, "input/wowdiag_manifest.json": diag_manifest,
|
||||
"source/development_files_index.json": [{"file": "/unavailable/server/source.cpp", "package_path": "source/development_files/source.cpp", "copied": True, "size": 4}],
|
||||
"source/development_files/source.cpp": b"code", "database/schema.json": {"columns": []},
|
||||
"source/primary_implementation_units.json": [], "source/source_ranking_policy.json": {"numeric_only_fallback_promotion": False},
|
||||
"unknown.extension": b"retained"}
|
||||
return make_tar(diag_entries), make_tar(src_entries), diag_entries, src_entries
|
||||
@@ -0,0 +1,110 @@
|
||||
import gzip
|
||||
import tarfile
|
||||
from dataclasses import replace
|
||||
|
||||
import pytest
|
||||
|
||||
from viano_bug_analysis.profiles.vcore import AdapterError, ArchiveLimits, ArchiveReader
|
||||
|
||||
|
||||
@pytest.mark.parametrize("name", ["../escape", "/absolute", "C:/escape", "a\\b", "a//b", "a/./b", "a:stream"])
|
||||
def test_unsafe_paths(make_tar, name):
|
||||
info = tarfile.TarInfo(name)
|
||||
info.size = 1
|
||||
path = make_tar({"manifest.json": {}}, extra=[(info, b"x")])
|
||||
with pytest.raises(AdapterError) as error:
|
||||
with ArchiveReader(path):
|
||||
pass
|
||||
assert error.value.code == "UNSAFE_PATH"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("type_", [tarfile.SYMTYPE, tarfile.LNKTYPE, tarfile.FIFOTYPE, tarfile.CHRTYPE, tarfile.BLKTYPE, tarfile.GNUTYPE_SPARSE])
|
||||
def test_special_members_rejected(make_tar, type_):
|
||||
info = tarfile.TarInfo("wrapper/link")
|
||||
info.type = type_
|
||||
info.linkname = "../escape"
|
||||
path = make_tar({"manifest.json": {}}, extra=[(info, None)])
|
||||
with pytest.raises(AdapterError):
|
||||
with ArchiveReader(path):
|
||||
pass
|
||||
|
||||
|
||||
@pytest.mark.parametrize("entries", [{"a": b"x", "A": b"y"}, {"a": b"x", "a/b": b"y"}])
|
||||
def test_member_collisions(make_tar, entries):
|
||||
path = make_tar({"manifest.json": {}} | entries)
|
||||
with pytest.raises(AdapterError) as error:
|
||||
with ArchiveReader(path):
|
||||
pass
|
||||
assert error.value.code == "MEMBER_COLLISION"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("damage", ["gzip", "truncated", "crc", "tar"])
|
||||
def test_corrupt_source_rejected(make_tar, damage):
|
||||
path = make_tar({"manifest.json": {}})
|
||||
raw = bytearray(path.read_bytes())
|
||||
if damage == "gzip":
|
||||
raw = b"not gzip"
|
||||
elif damage == "truncated":
|
||||
raw = raw[:-4]
|
||||
elif damage == "crc":
|
||||
raw[-8] ^= 1
|
||||
else:
|
||||
unpacked = bytearray(gzip.decompress(raw))
|
||||
unpacked[0] ^= 1
|
||||
raw = gzip.compress(unpacked)
|
||||
path.write_bytes(raw)
|
||||
with pytest.raises(AdapterError):
|
||||
with ArchiveReader(path):
|
||||
pass
|
||||
|
||||
|
||||
@pytest.mark.parametrize("field,value", [("max_archive_bytes", 1), ("max_tar_bytes", 512), ("max_file_bytes", 1), ("max_entries", 1), ("max_json_bytes", 1), ("max_ratio", 1)])
|
||||
def test_configurable_limits(make_tar, field, value):
|
||||
path = make_tar({"manifest.json": {}, "data": b"abc"})
|
||||
with pytest.raises(AdapterError):
|
||||
with ArchiveReader(path, replace(ArchiveLimits(), **{field: value})):
|
||||
pass
|
||||
|
||||
|
||||
def test_pax_metadata_bound(make_tar):
|
||||
info = tarfile.TarInfo("extra")
|
||||
info.pax_headers = {"comment": "x" * 2000}
|
||||
info.size = 1
|
||||
path = make_tar({"manifest.json": {}}, extra=[(info, b"x")])
|
||||
with pytest.raises(AdapterError) as error:
|
||||
with ArchiveReader(path, ArchiveLimits(max_header_bytes=100)):
|
||||
pass
|
||||
assert error.value.code == "HEADER_LIMIT"
|
||||
|
||||
|
||||
def test_ambiguous_manifest(make_tar):
|
||||
path = make_tar({"one/manifest.json": {}, "two/manifest.json": {}}, root="")
|
||||
with pytest.raises(AdapterError) as error:
|
||||
with ArchiveReader(path):
|
||||
pass
|
||||
assert error.value.code == "MANIFEST_AMBIGUOUS"
|
||||
|
||||
|
||||
def test_snapshot_is_immutable_and_nothing_is_extracted(make_tar, tmp_path):
|
||||
path = make_tar({"manifest.json": {}, "unknown": b"data"})
|
||||
raw = path.read_bytes()
|
||||
with ArchiveReader(path) as reader:
|
||||
path.write_bytes(b"changed external file")
|
||||
assert reader.original_bytes() == raw
|
||||
assert reader.read_member(reader.member_name("unknown")) == b"data"
|
||||
with pytest.raises(AdapterError):
|
||||
reader.member_name("../escape")
|
||||
assert not (tmp_path / "arbitrary-wrapper").exists()
|
||||
|
||||
|
||||
def test_sparse_rejected_before_sparse_map_processing(make_tar, monkeypatch):
|
||||
info = tarfile.TarInfo("sparse")
|
||||
info.type = tarfile.GNUTYPE_SPARSE
|
||||
path = make_tar({"manifest.json": {}}, extra=[(info, None)])
|
||||
def never_parse(*args):
|
||||
pytest.fail("Sparse map processing should not execute")
|
||||
monkeypatch.setattr(tarfile.TarInfo, "_proc_sparse", never_parse)
|
||||
with pytest.raises(AdapterError) as error:
|
||||
with ArchiveReader(path):
|
||||
pass
|
||||
assert error.value.code == "SPECIAL_MEMBER"
|
||||
@@ -0,0 +1,213 @@
|
||||
import json
|
||||
import shutil
|
||||
|
||||
import pytest
|
||||
|
||||
from viano_bug_analysis.case import CaseBuilder
|
||||
from viano_bug_analysis.lifecycle import CheckpointStore, Workspace
|
||||
from viano_bug_analysis.package import PackageReader, PackageValidator
|
||||
from viano_bug_analysis.profiles.vcore import AdapterError, VcoreImporter
|
||||
from viano_bug_analysis.profiles.vcore import ArchiveLimits
|
||||
|
||||
|
||||
def test_pair_roundtrip_and_provenance(source_pair, tmp_path):
|
||||
diag, src, diag_entries, src_entries = source_pair
|
||||
destination = tmp_path / "renamed.package"
|
||||
importer = VcoreImporter()
|
||||
result = importer.create(diag, destination, wowsrc=src, package_id="stable", case_id="opaque-case")
|
||||
assert result.valid and result.manifest["package_generation"] == 0
|
||||
assert result.manifest["history"]["reproductions"] == 1
|
||||
store = CheckpointStore(Workspace(tmp_path / "service"))
|
||||
store.intake(destination)
|
||||
built = CaseBuilder(store).build("opaque-case", "stable")
|
||||
assert PackageValidator().validate(built.package).valid
|
||||
with PackageReader(built.package) as reader:
|
||||
case = json.loads(reader.read_bytes(built.case_ref))
|
||||
observations = case["human_observations"]["observations"]
|
||||
assert [row["text"] for row in observations] == ["Wolpertinger spawnen nicht", "No spawns here; the other location worked", "Still no spawns"]
|
||||
assert case["expected_behavior"]["completeness"] == "missing"
|
||||
assert case["human_observations"]["tester_hypotheses"] == []
|
||||
note = observations[-1]["provenance"]["origin"]
|
||||
assert note["json_pointer"] == "/0/data_json" and note["decoded_json_pointer"] == "/text"
|
||||
assert note["source_timestamp"] == "2026-10-08T13:02:55"
|
||||
assert observations[-1]["provenance"]["observed_at"] is None
|
||||
raw_note = json.loads(reader.read_bytes(note["source_ref"]))[0]["data_json"]
|
||||
assert json.loads(raw_note)["text"] == observations[-1]["text"]
|
||||
assert reader.read_bytes(note["archive_ref"]) == diag.read_bytes()
|
||||
assert any(row["type"] == "quest" and row["id"] == "11117" and "diagnostic_focus" in row["roles"] for row in case["entities"])
|
||||
assert any(issue["code"] == "SOURCE_TIME_UNQUALIFIED" for issue in case["data_quality"]["issues"])
|
||||
assert any(ref["path"].endswith("source.cpp") for ref in case["source_context"]["refs"])
|
||||
assert any(ref["path"].endswith("unknown.extension") for ref in case["inputs"]["unclassified"])
|
||||
assert "Source timestamp (unchanged):" in reader.read_bytes(built.summary_ref).decode()
|
||||
# All materialized raw files and both compressed originals are preserved.
|
||||
for entries in (diag_entries, src_entries):
|
||||
for name, data in entries.items():
|
||||
paths = [path for path in reader.files if "/members/" in path and path.endswith("/" + name)]
|
||||
expected = data if isinstance(data, bytes) else json.dumps(data).encode()
|
||||
assert any(reader.read_bytes(path) == expected for path in paths)
|
||||
assert CaseBuilder(store).build("opaque-case", "stable").reused
|
||||
|
||||
|
||||
def test_filename_has_no_meaning(source_pair, tmp_path):
|
||||
diag, src, _, _ = source_pair
|
||||
renamed_diag, renamed_src = tmp_path / "unrelated.one", tmp_path / "unrelated.two"
|
||||
shutil.copyfile(diag, renamed_diag)
|
||||
shutil.copyfile(src, renamed_src)
|
||||
first = VcoreImporter().prepare(diag, src)
|
||||
again = VcoreImporter().prepare(renamed_diag, renamed_src)
|
||||
assert first == again
|
||||
|
||||
|
||||
@pytest.mark.parametrize("field,value", [("tool", "other"), ("format_version", 99), ("command", "quest"), ("bug_id", 21), ("core_revision", "different"), ("input_wowdiag_format", 3)])
|
||||
def test_mismatched_pair_rejected(source_pair, make_tar, tmp_path, field, value):
|
||||
diag, _, _, src_entries = source_pair
|
||||
src_entries["manifest.json"] = dict(src_entries["manifest.json"], **{field: value})
|
||||
src = make_tar(src_entries)
|
||||
destination = tmp_path / "out"
|
||||
with pytest.raises(AdapterError):
|
||||
VcoreImporter().create(diag, destination, wowsrc=src, package_id="p", case_id="c")
|
||||
assert not destination.exists()
|
||||
|
||||
|
||||
@pytest.mark.parametrize("path", ["../outside", "/external/file", "missing"])
|
||||
def test_copied_source_references_not_followed(source_pair, make_tar, path):
|
||||
diag, _, _, entries = source_pair
|
||||
entries["source/development_files_index.json"][0]["package_path"] = path
|
||||
with pytest.raises(AdapterError) as error:
|
||||
VcoreImporter().prepare(diag, make_tar(entries))
|
||||
assert error.value.code == "SOURCE_REFERENCE"
|
||||
|
||||
|
||||
def test_missing_copied_source_reported(source_pair, make_tar):
|
||||
diag, _, _, entries = source_pair
|
||||
entries["source/development_files_index.json"][0] = {"file": "/external/large.cpp", "copied": False, "reason_not_copied": "size_limit"}
|
||||
plan = VcoreImporter().prepare(diag, make_tar(entries))
|
||||
receipt = json.loads(plan.additions[plan.receipt_ref])
|
||||
assert any(issue["code"] == "SOURCE_NOT_COPIED" for issue in receipt["issues"])
|
||||
|
||||
|
||||
def test_wrong_copied_size_rejected(source_pair, make_tar):
|
||||
diag, _, _, entries = source_pair
|
||||
entries["source/development_files_index.json"][0]["size"] = 123
|
||||
with pytest.raises(AdapterError) as error:
|
||||
VcoreImporter().prepare(diag, make_tar(entries))
|
||||
assert error.value.code == "SOURCE_SIZE"
|
||||
|
||||
|
||||
def test_augment_preserves_old_content_and_rejects_duplicate(source_pair, tmp_path):
|
||||
diag, src, _, _ = source_pair
|
||||
first, second = tmp_path / "first", tmp_path / "second"
|
||||
importer = VcoreImporter()
|
||||
importer.create(diag, first, package_id="stable", case_id="opaque")
|
||||
result = importer.augment(first, second, wowdiag=diag, wowsrc=src)
|
||||
assert result.manifest["package_generation"] == 1
|
||||
with PackageReader(first) as before, PackageReader(second) as after:
|
||||
for name in before.files:
|
||||
if name != "manifest.json":
|
||||
assert before.read_bytes(name) == after.read_bytes(name)
|
||||
with pytest.raises(AdapterError) as error:
|
||||
importer.augment(second, tmp_path / "third", wowdiag=diag, wowsrc=src)
|
||||
assert error.value.code == "IMPORT_EXISTS"
|
||||
|
||||
|
||||
def test_case_identity_is_caller_owned(source_pair, tmp_path):
|
||||
diag, _, _, _ = source_pair
|
||||
result = VcoreImporter().create(diag, tmp_path / "BUG-999.zip", package_id="caller-package", case_id="caller-case")
|
||||
assert result.manifest["package_id"] == "caller-package"
|
||||
assert result.manifest["case_id"] == "caller-case"
|
||||
|
||||
|
||||
def test_combined_import_budget(source_pair):
|
||||
diag, src, _, _ = source_pair
|
||||
with pytest.raises(AdapterError) as error:
|
||||
VcoreImporter(ArchiveLimits(max_import_bytes=100)).prepare(diag, src)
|
||||
assert error.value.code == "IMPORT_LIMIT"
|
||||
|
||||
|
||||
def test_unresolved_row_relations_rejected(source_pair, make_tar):
|
||||
_, src, entries, _ = source_pair
|
||||
entries["bug/reproduction_events.json"][0]["reproduction_id"] = 999
|
||||
with pytest.raises(AdapterError) as error:
|
||||
VcoreImporter().prepare(make_tar(entries))
|
||||
assert error.value.code == "SOURCE_RELATION"
|
||||
|
||||
|
||||
def test_duplicate_source_ids_rejected(source_pair, make_tar):
|
||||
_, _, entries, _ = source_pair
|
||||
entries["bug/reports.json"] *= 2
|
||||
with pytest.raises(AdapterError) as error:
|
||||
VcoreImporter().prepare(make_tar(entries))
|
||||
assert error.value.code == "SOURCE_ID"
|
||||
|
||||
|
||||
def test_explicit_expected_behavior_is_copied_not_completed(source_pair, make_tar):
|
||||
_, _, entries, _ = source_pair
|
||||
entries["bug/reports.json"][0]["expected_behavior"] = "Supplied expected behavior"
|
||||
plan = VcoreImporter().prepare(make_tar(entries))
|
||||
index = json.loads(plan.additions[plan.index_ref])
|
||||
report = json.loads(plan.additions[index["reports"][0]])
|
||||
assert report["expected_behavior"][0]["text"] == "Supplied expected behavior"
|
||||
assert "expected_behavior_complete" not in report
|
||||
|
||||
|
||||
def test_aware_source_time_preserved(source_pair, make_tar):
|
||||
_, _, entries, _ = source_pair
|
||||
entries["bug/reports.json"][0]["created_at"] = "2026-10-08T13:02:21+02:00"
|
||||
plan = VcoreImporter().prepare(make_tar(entries))
|
||||
report = json.loads(plan.additions[json.loads(plan.additions[plan.index_ref])["reports"][0]])
|
||||
assert report["observations"][0]["observed_at"] == "2026-10-08T13:02:21+02:00"
|
||||
|
||||
|
||||
def test_bad_existing_receipt_is_controlled(source_pair, manifest, tmp_path):
|
||||
from viano_bug_analysis.package import PackageFactory
|
||||
diag, _, _, _ = source_pair
|
||||
manifest["project"] = {"type": "wow", "profile": "vcore"}
|
||||
manifest["artifacts"]["vcore_import"] = "bad-receipt"
|
||||
original = tmp_path / "old"
|
||||
PackageFactory().create(original, manifest, {"bad-receipt": b"[]"})
|
||||
with pytest.raises(AdapterError) as error:
|
||||
VcoreImporter().augment(original, tmp_path / "out", wowdiag=diag)
|
||||
assert error.value.code == "IMPORT_RECEIPT"
|
||||
assert not (tmp_path / "out").exists()
|
||||
|
||||
|
||||
def test_copied_attachments_are_classified(source_pair, make_tar):
|
||||
_, _, entries, _ = source_pair
|
||||
entries["attachments/32/screenshot.bin"] = b"copied attachment"
|
||||
plan = VcoreImporter().prepare(make_tar(entries))
|
||||
index = json.loads(plan.additions[plan.index_ref])
|
||||
assert len(index["attachments"]) == 1
|
||||
assert plan.additions[index["attachments"][0]] == b"copied attachment"
|
||||
|
||||
|
||||
def test_report_link_allows_null_direct_bug_id(source_pair, make_tar):
|
||||
_, _, entries, _ = source_pair
|
||||
entries["bug/reproductions.json"][0]["bug_id"] = None
|
||||
assert VcoreImporter().prepare(make_tar(entries)).reproductions_count == 1
|
||||
|
||||
|
||||
def test_explicit_runtime_entities_are_normalized(source_pair, make_tar):
|
||||
_, _, entries, _ = source_pair
|
||||
entries["bug/runtime_entities.json"] = [{"type": "creature", "id": 123, "reasons": ["recorded_event"]}]
|
||||
plan = VcoreImporter().prepare(make_tar(entries))
|
||||
index = json.loads(plan.additions[plan.index_ref])
|
||||
records = [json.loads(plan.additions[path]) for path in index["reports"]]
|
||||
entities = [entity for record in records for entity in record.get("entities", [])]
|
||||
assert any(entity["id"] == 123 and entity["roles"] == ["runtime_observed"] and entity["origin"]["json_pointer"] == "/0" for entity in entities)
|
||||
|
||||
|
||||
def test_adding_wowsrc_does_not_duplicate_original_observations(source_pair, tmp_path):
|
||||
diag, src, _, _ = source_pair
|
||||
first, augmented = tmp_path / "first", tmp_path / "augmented"
|
||||
importer = VcoreImporter()
|
||||
importer.create(diag, first, package_id="p", case_id="c")
|
||||
importer.augment(first, augmented, wowdiag=diag, wowsrc=src)
|
||||
store = CheckpointStore(Workspace(tmp_path / "service"))
|
||||
store.intake(augmented)
|
||||
result = CaseBuilder(store).build("c", "p")
|
||||
with PackageReader(result.package) as reader:
|
||||
case = json.loads(reader.read_bytes(result.case_ref))
|
||||
assert len(case["human_observations"]["observations"]) == 3
|
||||
with pytest.raises(AdapterError) as error:
|
||||
importer.augment(augmented, tmp_path / "repeated", wowdiag=diag, wowsrc=src)
|
||||
assert error.value.code == "IMPORT_EXISTS"
|
||||
@@ -0,0 +1,40 @@
|
||||
"""Opt-in full-archive regression, without machine-specific paths or binary fixtures."""
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from viano_bug_analysis.case import CaseBuilder
|
||||
from viano_bug_analysis.lifecycle import CheckpointStore, Workspace
|
||||
from viano_bug_analysis.package import PackageReader, PackageValidator
|
||||
from viano_bug_analysis.profiles.vcore import ArchiveReader, VcoreImporter
|
||||
|
||||
|
||||
@pytest.mark.skipif(not os.environ.get("VBA_WOWDIAG_SAMPLE") or not os.environ.get("VBA_WOWSRC_SAMPLE"),
|
||||
reason="Set VBA_WOWDIAG_SAMPLE and VBA_WOWSRC_SAMPLE for the BUG-20 full archive regression")
|
||||
def test_full_bug20_package_pair(tmp_path):
|
||||
diag, src = Path(os.environ["VBA_WOWDIAG_SAMPLE"]), Path(os.environ["VBA_WOWSRC_SAMPLE"])
|
||||
original_hashes = [hashlib.sha256(path.read_bytes()).hexdigest() for path in (diag, src)]
|
||||
package = tmp_path / "anything.bin"
|
||||
VcoreImporter().create(diag, package, package_id="regression-package", case_id="regression-case", wowsrc=src)
|
||||
store = CheckpointStore(Workspace(tmp_path / "service"))
|
||||
store.intake(package)
|
||||
result = CaseBuilder(store).build("regression-case", "regression-package")
|
||||
assert PackageValidator().validate(result.package).valid
|
||||
with PackageReader(result.package) as reader:
|
||||
case = json.loads(reader.read_bytes(result.case_ref))
|
||||
assert len(case["human_observations"]["observations"]) == 3
|
||||
assert case["expected_behavior"]["completeness"] == "missing"
|
||||
assert case["human_observations"]["tester_hypotheses"] == []
|
||||
assert len(case["database_context"]["refs"]) == 103
|
||||
assert len(case["source_context"]["refs"]) == 93
|
||||
for input_path in (diag, src):
|
||||
with ArchiveReader(input_path) as source:
|
||||
for member in source.members:
|
||||
matching = [path for path in reader.files if path.endswith("/members/" + member)]
|
||||
assert len(matching) == 1
|
||||
assert reader.read_bytes(matching[0]) == source.read_member(member)
|
||||
assert CaseBuilder(store).build("regression-case", "regression-package").reused
|
||||
assert [hashlib.sha256(path.read_bytes()).hexdigest() for path in (diag, src)] == original_hashes
|
||||
@@ -0,0 +1,44 @@
|
||||
import copy
|
||||
import os
|
||||
|
||||
import pytest
|
||||
|
||||
from viano_bug_analysis.package import PackageFactory, PackageReader, PackageWriteError
|
||||
|
||||
|
||||
def test_generic_factory(manifest, tmp_path):
|
||||
original = copy.deepcopy(manifest)
|
||||
path = tmp_path / "first.package"
|
||||
assert PackageFactory().create(path, manifest, {"unknown": b"opaque"}).valid
|
||||
assert manifest == original
|
||||
with PackageReader(path) as reader:
|
||||
assert reader.manifest["package_generation"] == 0
|
||||
assert reader.read_bytes("unknown") == b"opaque"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("artifacts", [{"../escape": b"x"}, {"manifest.json": b"x"}, {"a": b"x", "A": b"y"}, {"a": b"x", "a/b": b"y"}, {"a": "not bytes"}])
|
||||
def test_invalid_creation_keeps_target_absent(manifest, tmp_path, artifacts):
|
||||
target = tmp_path / "out"
|
||||
with pytest.raises(PackageWriteError):
|
||||
PackageFactory().create(target, manifest, artifacts)
|
||||
assert not target.exists()
|
||||
assert not list(tmp_path.glob(".vbcase-*.tmp"))
|
||||
|
||||
|
||||
def test_factory_failure_preserves_existing_target(manifest, tmp_path):
|
||||
target = tmp_path / "out"
|
||||
target.write_bytes(b"previous")
|
||||
with pytest.raises(PackageWriteError):
|
||||
PackageFactory().create(target, manifest, {})
|
||||
assert target.read_bytes() == b"previous"
|
||||
|
||||
|
||||
def test_factory_fsync_failure(manifest, tmp_path, monkeypatch):
|
||||
target = tmp_path / "out"
|
||||
def fail(*args):
|
||||
raise OSError("disk error")
|
||||
monkeypatch.setattr(os, "fsync", fail)
|
||||
with pytest.raises(PackageWriteError):
|
||||
PackageFactory().create(target, manifest, {})
|
||||
assert not target.exists()
|
||||
assert not list(tmp_path.glob(".vbcase-*.tmp"))
|
||||
Reference in new issue
Block a user