feat: add case package validation foundation
This commit is contained in:
1 parent
2e590d5e12
commit
112487242a
18 files changed
+806
-1
No files matched your search
@@ -0,0 +1,7 @@
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
.pytest_cache/
|
||||
.venv/
|
||||
*.egg-info/
|
||||
build/
|
||||
dist/
|
||||
@@ -1,3 +1,17 @@
|
||||
# VianoBugAnalysis
|
||||
|
||||
AutomaticBuganalysis Programm, to auto detect bugs and instruct a coding agent
|
||||
Project-independent foundations for reading and validating long-lived ZIP case
|
||||
packages. The authoritative design is [docs/architecture.md](docs/architecture.md).
|
||||
The implemented v1 contract is documented in
|
||||
[docs/package-format-v1.md](docs/package-format-v1.md).
|
||||
|
||||
Requires Python 3.11 or newer.
|
||||
|
||||
```powershell
|
||||
python -m venv .venv
|
||||
.\.venv\Scripts\python.exe -m pip install -e ".[test]"
|
||||
.\.venv\Scripts\python.exe -m pytest
|
||||
```
|
||||
|
||||
This milestone includes no package writer, lifecycle worker, CaseBuilder, LLM,
|
||||
case interpretation, hypothesis generation or investigation tools.
|
||||
@@ -0,0 +1,125 @@
|
||||
# Case package format v1
|
||||
|
||||
The authoritative design is `architecture.md`. This document specifies only the
|
||||
project-independent package foundation. A package is a long-lived, self-contained
|
||||
case record in a ZIP archive. Its filename and extension carry no program meaning.
|
||||
Exactly one root `manifest.json` is the authoritative entry point.
|
||||
|
||||
## Identity, state and routing
|
||||
|
||||
`format` is `viano-bug-case-package`; `format_version` is integer `1`.
|
||||
`package_id` is an opaque nonblank string identifying the case package throughout
|
||||
its lifetime. Enriching and emitting it again preserves this identity and, in a
|
||||
future writer, increases `package_generation` (a nonnegative integer).
|
||||
`case_id` identifies the case. `analysis_revision` is a separate nonnegative
|
||||
integer, initially zero; it is independent of package generation.
|
||||
The reader does not derive or increment any of these values.
|
||||
|
||||
`project` requires generic `type` and `profile` strings and accepts `display_name`.
|
||||
`created_at` and `updated_at` are RFC-3339 timestamps with timezone; updated time
|
||||
must not precede creation time.
|
||||
Timestamp validation rejects leap seconds because Python's datetime model cannot
|
||||
represent them; lowercase `t`/`z` and fractional seconds are supported.
|
||||
|
||||
`state` uses the lifecycle states defined in the architecture: NEW, VALIDATED,
|
||||
BUILDING_CASE, INTERPRETING_CASE, GENERATING_HYPOTHESES,
|
||||
INVESTIGATING_HYPOTHESES, BUILDING_DIAGNOSIS, OUTPUT_READY, HANDED_OFF,
|
||||
BLOCKED, FAILED_TO_ANALYZE, REOPENED.
|
||||
|
||||
`routing` is separate and requires:
|
||||
|
||||
- `destination`: null or inbox, quarantine, cases, outbox, review; a logical route,
|
||||
never a filesystem path. Cache and runtime are service storage, not routes.
|
||||
- `reason`: null or nonblank explanatory string.
|
||||
- `retry_count`: nonnegative integer, initially zero.
|
||||
- `retry_not_before`: null or RFC-3339 timestamp.
|
||||
|
||||
No state/route combination triggers execution. This milestone checks the contract
|
||||
only, without implementing routing, retry, locking or lifecycle transitions.
|
||||
|
||||
## References and integrity
|
||||
|
||||
`artifacts` requires `case`, `diagnosis`, `change_request`, each null or an internal
|
||||
file path. Additional artifact pointers follow the same rule. `history` requires
|
||||
nonnegative counters `analysis_revisions`, `reproductions`, `evidence`, `handoffs`,
|
||||
`feedback`, and nullable `events_ref`. Counters are supplied summaries; the reader
|
||||
does not infer them from directory names or impose diagnostic semantics.
|
||||
|
||||
`integrity` maps internal file paths to `{ "sha256": "<64 lowercase hex digits>",
|
||||
"size": <uncompressed byte count> }`. All supplied records are checked. A fresh
|
||||
package may have an empty map. Producers should cover important immutable
|
||||
artifacts. The format foundation does not infer which project artifacts are
|
||||
important. Hashes detect changes, not authenticity. The manifest cannot reference
|
||||
or hash itself. Every non-null reference must resolve to an actual ZIP file.
|
||||
|
||||
The bundled `manifest-v1.schema.json` uses JSON Schema Draft 2020-12 and only local
|
||||
schema references; validation requires no network. Unknown extension fields are
|
||||
accepted and retained. Unsupported format versions, duplicate JSON keys,
|
||||
non-UTF-8 manifests and nonstandard numeric constants are rejected.
|
||||
|
||||
## Preservation
|
||||
|
||||
`original/` and completed `analysis/revisions/NNNN/` are immutable. New evidence,
|
||||
feedback and revisions are appended. Only explicitly designated index/pointer
|
||||
files (`manifest.json`, `analysis/current.json`) may later be updated. Unknown
|
||||
files are preserved unchanged. Validator and reader never write, extract, delete
|
||||
or rewrite anything. Historical append-only and output-superset comparisons,
|
||||
atomic generation and generation increments belong to a future writer; they
|
||||
cannot be proven from one standalone incoming archive.
|
||||
|
||||
## ZIP acceptance and configurable security policy
|
||||
|
||||
This implementation accepts stored and deflated ZIP entries, including ZIP64
|
||||
within configured limits. It rejects encryption, symlinks and other special
|
||||
filesystem entries, unsafe paths, duplicate entries, case-insensitive collisions,
|
||||
and file/directory conflicts. Paths must be relative POSIX paths without empty,
|
||||
dot or parent segments, backslashes, colons or control characters. Explicit empty
|
||||
directory entries are accepted. Paths are rejected, never silently repaired.
|
||||
The reader does not extract files; future extraction requires its own checks.
|
||||
|
||||
`PackageLimits` is a configurable operational security policy. These defaults
|
||||
are **not immutable limits of the v1 package format** and do not appear in the
|
||||
manifest schema:
|
||||
|
||||
| Limit | Default |
|
||||
|---|---:|
|
||||
| Archive bytes | 512 MiB |
|
||||
| Expanded total bytes | 2 GiB |
|
||||
| Expanded file bytes | 256 MiB |
|
||||
| Manifest bytes | 1 MiB |
|
||||
| ZIP entries (including directories) | 10,000 |
|
||||
| Per-entry compression ratio | 200:1 |
|
||||
|
||||
Declared sizes are checked before reading. Actual reads are bounded and total
|
||||
bytes are counted while streaming every file for CRC verification; listed SHA-256
|
||||
hashes are checked during the same pass. Manifest parsing is bounded separately.
|
||||
Malformed archive, I/O, JSON, schema and integrity failures produce structured
|
||||
validation issues. Programmer errors and process-level failures are not hidden.
|
||||
Input files must not be modified concurrently while open; the reader retains the
|
||||
same file handle and archive used for validation.
|
||||
|
||||
## Public API
|
||||
|
||||
```python
|
||||
from viano_bug_analysis.package import (
|
||||
PackageLimits, PackageValidator, PackageReader, PackageReadError,
|
||||
)
|
||||
|
||||
result = PackageValidator(PackageLimits()).validate("renamed-package.bin")
|
||||
if not result.valid:
|
||||
for issue in result.issues:
|
||||
print(issue.code, issue.path, issue.message)
|
||||
|
||||
try:
|
||||
with PackageReader("renamed-package.bin") as package:
|
||||
manifest = package.manifest # defensive copy, including extensions
|
||||
inventory = package.files # all file paths, including unknown files
|
||||
data = package.read_bytes("original/bug_report.json")
|
||||
except PackageReadError as error:
|
||||
issues = error.result.issues
|
||||
```
|
||||
|
||||
Read access outside the context is a programming error (`RuntimeError`). Invalid
|
||||
packages and missing member requests raise `PackageReadError` with structured
|
||||
details. `read_bytes` is bounded by policy but materializes a single file in memory;
|
||||
validation itself streams. No artifact interpretation is performed.
|
||||
@@ -0,0 +1,23 @@
|
||||
[build-system]
|
||||
requires = ["setuptools>=68"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "viano-bug-analysis"
|
||||
version = "0.1.0"
|
||||
description = "Project-independent case package validation and reading"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = ["jsonschema>=4.18,<5"]
|
||||
|
||||
[project.optional-dependencies]
|
||||
test = ["pytest>=8,<10"]
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
where = ["src"]
|
||||
|
||||
[tool.setuptools.package-data]
|
||||
"viano_bug_analysis.schemas" = ["*.json"]
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
pythonpath = ["src"]
|
||||
@@ -0,0 +1 @@
|
||||
"""VianoBugAnalysis: deterministic case package foundations."""
|
||||
@@ -0,0 +1,7 @@
|
||||
from .errors import PackageReadError, ValidationIssue, ValidationResult
|
||||
from .limits import PackageLimits
|
||||
from .reader import PackageReader
|
||||
from .validator import PackageValidator
|
||||
|
||||
__all__ = ["PackageReader", "PackageValidator", "PackageLimits", "PackageReadError",
|
||||
"ValidationIssue", "ValidationResult"]
|
||||
@@ -0,0 +1,26 @@
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ValidationIssue:
|
||||
code: str
|
||||
message: str
|
||||
path: str | None = None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ValidationResult:
|
||||
issues: tuple[ValidationIssue, ...] = ()
|
||||
manifest: dict[str, Any] | None = field(default=None, repr=False)
|
||||
files: tuple[str, ...] = ()
|
||||
|
||||
@property
|
||||
def valid(self) -> bool:
|
||||
return not self.issues
|
||||
|
||||
|
||||
class PackageReadError(Exception):
|
||||
def __init__(self, result: ValidationResult):
|
||||
self.result = result
|
||||
super().__init__("; ".join(f"{i.code}: {i.message}" for i in result.issues))
|
||||
@@ -0,0 +1,22 @@
|
||||
from dataclasses import dataclass, fields
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PackageLimits:
|
||||
"""Configurable security policy, not restrictions of the v1 format."""
|
||||
|
||||
max_archive_bytes: int = 512 * 1024**2
|
||||
max_total_bytes: int = 2 * 1024**3
|
||||
max_file_bytes: int = 256 * 1024**2
|
||||
max_manifest_bytes: int = 1024**2
|
||||
max_entries: int = 10_000
|
||||
max_compression_ratio: float = 200
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
import math
|
||||
for item in fields(self):
|
||||
value = getattr(self, item.name)
|
||||
if isinstance(value, bool) or not isinstance(value, (int, float)) or not math.isfinite(value) or value <= 0:
|
||||
raise ValueError(f"{item.name} must be positive and finite")
|
||||
if item.name != "max_compression_ratio" and not isinstance(value, int):
|
||||
raise ValueError(f"{item.name} must be an integer")
|
||||
@@ -0,0 +1,63 @@
|
||||
import json
|
||||
import re
|
||||
from datetime import datetime
|
||||
from importlib.resources import files
|
||||
from typing import Any
|
||||
|
||||
from jsonschema import Draft202012Validator, FormatChecker
|
||||
|
||||
from .errors import ValidationIssue
|
||||
|
||||
FORMAT_CHECKER = FormatChecker()
|
||||
|
||||
|
||||
@FORMAT_CHECKER.checks("date-time", raises=ValueError)
|
||||
def _timestamp(value: Any) -> bool:
|
||||
if not isinstance(value, str):
|
||||
return True
|
||||
if not re.fullmatch(r"\d{4}-\d{2}-\d{2}[Tt]\d{2}:\d{2}:\d{2}(?:\.\d+)?(?:[Zz]|[+-]\d{2}:\d{2})", value):
|
||||
return False
|
||||
if value[-1] not in "Zz" and (int(value[-5:-3]) > 23 or int(value[-2:]) > 59):
|
||||
return False
|
||||
datetime.fromisoformat(value.upper().replace("Z", "+00:00"))
|
||||
return True
|
||||
|
||||
|
||||
def safe_path(path: str, *, directory: bool = False) -> bool:
|
||||
if not isinstance(path, str) or not path or "\\" in path or ":" in path:
|
||||
return False
|
||||
if any(ord(c) < 32 or ord(c) == 127 for c in path):
|
||||
return False
|
||||
value = path[:-1] if directory and path.endswith("/") else path
|
||||
return all(part not in ("", ".", "..") for part in value.split("/"))
|
||||
|
||||
|
||||
def _unique_object(pairs: list[tuple[str, Any]]) -> dict[str, Any]:
|
||||
result = {}
|
||||
for key, value in pairs:
|
||||
if key in result:
|
||||
raise ValueError(f"Duplicate JSON key: {key}")
|
||||
result[key] = value
|
||||
return result
|
||||
|
||||
|
||||
def parse_manifest(data: bytes) -> dict[str, Any]:
|
||||
return json.loads(data.decode("utf-8"), object_pairs_hook=_unique_object,
|
||||
parse_constant=lambda value: (_ for _ in ()).throw(ValueError(f"Invalid JSON constant: {value}")))
|
||||
|
||||
|
||||
def manifest_issues(manifest: Any) -> list[ValidationIssue]:
|
||||
schema = json.loads(files("viano_bug_analysis.schemas").joinpath("manifest-v1.schema.json").read_text(encoding="utf-8"))
|
||||
validator = Draft202012Validator(schema, format_checker=FORMAT_CHECKER)
|
||||
issues = [ValidationIssue("MANIFEST_SCHEMA", error.message,
|
||||
"/".join(map(str, error.absolute_path)) or "manifest.json")
|
||||
for error in validator.iter_errors(manifest)]
|
||||
if issues:
|
||||
return issues
|
||||
if datetime.fromisoformat(manifest["updated_at"].upper().replace("Z", "+00:00")) < datetime.fromisoformat(manifest["created_at"].upper().replace("Z", "+00:00")):
|
||||
issues.append(ValidationIssue("MANIFEST_TIME", "updated_at precedes created_at", "manifest.json"))
|
||||
refs = list(manifest["artifacts"].values()) + [manifest["history"]["events_ref"]] + list(manifest["integrity"])
|
||||
for path in refs:
|
||||
if path is not None and (not safe_path(path) or path == "manifest.json"):
|
||||
issues.append(ValidationIssue("UNSAFE_REFERENCE", "Invalid artifact reference", path))
|
||||
return issues
|
||||
@@ -0,0 +1,71 @@
|
||||
import copy
|
||||
import os
|
||||
import zipfile
|
||||
|
||||
from .errors import PackageReadError
|
||||
from .limits import PackageLimits
|
||||
from .validator import ARCHIVE_ERRORS, PackageValidator
|
||||
|
||||
|
||||
class PackageReader:
|
||||
"""Read-only, validated access; no extraction or package rewriting."""
|
||||
|
||||
def __init__(self, path: str | os.PathLike[str], limits: PackageLimits | None = None):
|
||||
self.path = path
|
||||
self.validator = PackageValidator(limits)
|
||||
self._source = None
|
||||
self._archive = None
|
||||
self._result = None
|
||||
|
||||
def __enter__(self):
|
||||
if self._source is not None:
|
||||
raise RuntimeError("Reader is already open")
|
||||
try:
|
||||
self._source = open(self.path, "rb")
|
||||
if os.fstat(self._source.fileno()).st_size > self.validator.limits.max_archive_bytes:
|
||||
raise PackageReadError(self.validator._failure("ARCHIVE_LIMIT", "Archive exceeds configured size limit"))
|
||||
self._archive = zipfile.ZipFile(self._source)
|
||||
self._result = self.validator._validate_archive(self._archive)
|
||||
if not self._result.valid:
|
||||
raise PackageReadError(self._result)
|
||||
return self
|
||||
except PackageReadError:
|
||||
self.close()
|
||||
raise
|
||||
except ARCHIVE_ERRORS as error:
|
||||
self.close()
|
||||
raise PackageReadError(self.validator._failure("ARCHIVE_ERROR", str(error))) from error
|
||||
|
||||
def _require_open(self):
|
||||
if self._archive is None or self._result is None:
|
||||
raise RuntimeError("Use PackageReader inside a with block")
|
||||
|
||||
@property
|
||||
def manifest(self):
|
||||
self._require_open()
|
||||
return copy.deepcopy(self._result.manifest)
|
||||
|
||||
@property
|
||||
def files(self) -> tuple[str, ...]:
|
||||
self._require_open()
|
||||
return self._result.files
|
||||
|
||||
def read_bytes(self, path: str) -> bytes:
|
||||
self._require_open()
|
||||
if path not in self._result.files:
|
||||
raise PackageReadError(self.validator._failure("FILE_NOT_FOUND", "File is not in package", path))
|
||||
try:
|
||||
with self._archive.open(path) as stream:
|
||||
return self.validator._read_bounded(stream, self.validator.limits.max_file_bytes)
|
||||
except ARCHIVE_ERRORS as error:
|
||||
raise PackageReadError(self.validator._failure("READ_ERROR", str(error), path)) from error
|
||||
|
||||
def close(self):
|
||||
if self._archive is not None:
|
||||
self._archive.close()
|
||||
if self._source is not None:
|
||||
self._source.close()
|
||||
self._archive = self._source = self._result = None
|
||||
|
||||
def __exit__(self, *args):
|
||||
self.close()
|
||||
@@ -0,0 +1,107 @@
|
||||
import hashlib
|
||||
import os
|
||||
import stat
|
||||
import struct
|
||||
import zipfile
|
||||
import zlib
|
||||
from typing import BinaryIO
|
||||
|
||||
from .errors import ValidationIssue, ValidationResult
|
||||
from .limits import PackageLimits
|
||||
from .manifest import manifest_issues, parse_manifest, safe_path
|
||||
|
||||
CHUNK_SIZE = 64 * 1024
|
||||
ARCHIVE_ERRORS = (OSError, EOFError, ValueError, RuntimeError, NotImplementedError,
|
||||
zipfile.BadZipFile, zipfile.LargeZipFile, zlib.error, struct.error)
|
||||
|
||||
|
||||
class PackageValidator:
|
||||
def __init__(self, limits: PackageLimits | None = None):
|
||||
self.limits = limits or PackageLimits()
|
||||
|
||||
def validate(self, path: str | os.PathLike[str]) -> ValidationResult:
|
||||
try:
|
||||
with open(path, "rb") as source:
|
||||
if os.fstat(source.fileno()).st_size > self.limits.max_archive_bytes:
|
||||
return self._failure("ARCHIVE_LIMIT", "Archive exceeds configured size limit")
|
||||
with zipfile.ZipFile(source) as archive:
|
||||
return self._validate_archive(archive)
|
||||
except ARCHIVE_ERRORS as error:
|
||||
return self._failure("ARCHIVE_ERROR", str(error))
|
||||
|
||||
@staticmethod
|
||||
def _failure(code: str, message: str, path: str | None = None) -> ValidationResult:
|
||||
return ValidationResult((ValidationIssue(code, message, path),))
|
||||
|
||||
def _validate_archive(self, archive: zipfile.ZipFile) -> ValidationResult:
|
||||
limits = self.limits
|
||||
infos = archive.infolist()
|
||||
if len(infos) > limits.max_entries:
|
||||
return self._failure("ENTRY_LIMIT", "Too many ZIP entries")
|
||||
names: dict[str, bool] = {}
|
||||
total = 0
|
||||
for info in infos:
|
||||
name = info.orig_filename
|
||||
if not safe_path(name, directory=info.is_dir()):
|
||||
return self._failure("UNSAFE_PATH", "Unsafe ZIP path", name)
|
||||
key = name.rstrip("/").casefold()
|
||||
if key in names:
|
||||
return self._failure("ENTRY_COLLISION", "Duplicate or case-colliding ZIP entry", name)
|
||||
names[key] = info.is_dir()
|
||||
mode = info.external_attr >> 16
|
||||
if stat.S_IFMT(mode) not in (0, stat.S_IFREG, stat.S_IFDIR) or (stat.S_ISDIR(mode) and not info.is_dir()):
|
||||
return self._failure("SPECIAL_ENTRY", "Special filesystem entry is unsupported", name)
|
||||
if info.flag_bits & 1 or info.compress_type not in (zipfile.ZIP_STORED, zipfile.ZIP_DEFLATED):
|
||||
return self._failure("UNSUPPORTED_ENTRY", "Encryption or compression is unsupported", name)
|
||||
cap = min(limits.max_file_bytes, limits.max_manifest_bytes) if name == "manifest.json" else limits.max_file_bytes
|
||||
if info.file_size > cap or (info.is_dir() and info.file_size):
|
||||
return self._failure("FILE_LIMIT", "Entry exceeds configured size limit", name)
|
||||
if info.file_size / max(info.compress_size, 1) > limits.max_compression_ratio:
|
||||
return self._failure("RATIO_LIMIT", "Entry exceeds configured compression ratio", name)
|
||||
total += info.file_size
|
||||
if total > limits.max_total_bytes:
|
||||
return self._failure("TOTAL_LIMIT", "Package exceeds configured expanded size")
|
||||
for key in names:
|
||||
parts = key.split("/")
|
||||
if any(names.get("/".join(parts[:i])) is False for i in range(1, len(parts))):
|
||||
return self._failure("ENTRY_COLLISION", "File also used as directory", key)
|
||||
inventory = tuple(info.filename for info in infos if not info.is_dir())
|
||||
if "manifest.json" not in inventory:
|
||||
return self._failure("MANIFEST_MISSING", "Root manifest.json is required")
|
||||
try:
|
||||
with archive.open("manifest.json") as stream:
|
||||
data = self._read_bounded(stream, min(limits.max_manifest_bytes, limits.max_file_bytes))
|
||||
manifest = parse_manifest(data)
|
||||
issues = manifest_issues(manifest)
|
||||
if issues:
|
||||
return ValidationResult(tuple(issues))
|
||||
refs = list(manifest["artifacts"].values()) + [manifest["history"]["events_ref"]] + list(manifest["integrity"])
|
||||
for ref in refs:
|
||||
if ref is not None and ref not in inventory:
|
||||
return self._failure("REFERENCE_MISSING", "Referenced file is absent", ref)
|
||||
total_read = 0
|
||||
for name in inventory:
|
||||
digest = hashlib.sha256()
|
||||
size = 0
|
||||
with archive.open(name) as stream:
|
||||
while chunk := stream.read(CHUNK_SIZE):
|
||||
size += len(chunk)
|
||||
total_read += len(chunk)
|
||||
if size > limits.max_file_bytes or total_read > limits.max_total_bytes:
|
||||
return self._failure("STREAM_LIMIT", "Actual bytes exceed configured limit", name)
|
||||
digest.update(chunk)
|
||||
if size != archive.getinfo(name).file_size:
|
||||
return self._failure("SIZE_MISMATCH", "ZIP size does not match actual bytes", name)
|
||||
expected = manifest["integrity"].get(name)
|
||||
if expected and (expected["size"] != size or expected["sha256"] != digest.hexdigest()):
|
||||
return self._failure("INTEGRITY_MISMATCH", "Artifact hash or size does not match", name)
|
||||
return ValidationResult(manifest=manifest, files=inventory)
|
||||
except (UnicodeError, RecursionError) + ARCHIVE_ERRORS as error:
|
||||
return self._failure("PACKAGE_INVALID", str(error))
|
||||
|
||||
@staticmethod
|
||||
def _read_bounded(stream: BinaryIO, limit: int) -> bytes:
|
||||
data = stream.read(limit + 1)
|
||||
if len(data) > limit:
|
||||
raise ValueError("Read exceeds configured limit")
|
||||
return data
|
||||
@@ -0,0 +1 @@
|
||||
"""Bundled, offline JSON contracts."""
|
||||
@@ -0,0 +1,51 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"title": "Viano Bug Case Package manifest v1",
|
||||
"type": "object",
|
||||
"required": ["format", "format_version", "package_id", "package_generation", "case_id", "project", "created_at", "updated_at", "state", "routing", "analysis_revision", "artifacts", "history", "integrity"],
|
||||
"properties": {
|
||||
"format": {"const": "viano-bug-case-package"},
|
||||
"format_version": {"const": 1, "type": "integer"},
|
||||
"package_id": {"$ref": "#/$defs/id"},
|
||||
"package_generation": {"$ref": "#/$defs/count"},
|
||||
"case_id": {"$ref": "#/$defs/id"},
|
||||
"project": {
|
||||
"type": "object", "required": ["type", "profile"],
|
||||
"properties": {"type": {"$ref": "#/$defs/id"}, "profile": {"$ref": "#/$defs/id"}, "display_name": {"$ref": "#/$defs/id"}}
|
||||
},
|
||||
"created_at": {"type": "string", "format": "date-time"},
|
||||
"updated_at": {"type": "string", "format": "date-time"},
|
||||
"state": {"enum": ["NEW", "VALIDATED", "BUILDING_CASE", "INTERPRETING_CASE", "GENERATING_HYPOTHESES", "INVESTIGATING_HYPOTHESES", "BUILDING_DIAGNOSIS", "OUTPUT_READY", "HANDED_OFF", "BLOCKED", "FAILED_TO_ANALYZE", "REOPENED"]},
|
||||
"routing": {
|
||||
"type": "object", "required": ["destination", "reason", "retry_count", "retry_not_before"],
|
||||
"properties": {
|
||||
"destination": {"type": ["string", "null"], "enum": [null, "inbox", "quarantine", "cases", "outbox", "review"]},
|
||||
"reason": {"anyOf": [{"$ref": "#/$defs/id"}, {"type": "null"}]},
|
||||
"retry_count": {"$ref": "#/$defs/count"},
|
||||
"retry_not_before": {"type": ["string", "null"], "format": "date-time"}
|
||||
}
|
||||
},
|
||||
"analysis_revision": {"$ref": "#/$defs/count"},
|
||||
"artifacts": {
|
||||
"type": "object", "required": ["case", "diagnosis", "change_request"],
|
||||
"properties": {"case": {"$ref": "#/$defs/ref"}, "diagnosis": {"$ref": "#/$defs/ref"}, "change_request": {"$ref": "#/$defs/ref"}},
|
||||
"additionalProperties": {"$ref": "#/$defs/ref"}
|
||||
},
|
||||
"history": {
|
||||
"type": "object", "required": ["analysis_revisions", "reproductions", "evidence", "handoffs", "feedback", "events_ref"],
|
||||
"properties": {"analysis_revisions": {"$ref": "#/$defs/count"}, "reproductions": {"$ref": "#/$defs/count"}, "evidence": {"$ref": "#/$defs/count"}, "handoffs": {"$ref": "#/$defs/count"}, "feedback": {"$ref": "#/$defs/count"}, "events_ref": {"$ref": "#/$defs/ref"}}
|
||||
},
|
||||
"integrity": {
|
||||
"type": "object",
|
||||
"additionalProperties": {
|
||||
"type": "object", "required": ["sha256", "size"],
|
||||
"properties": {"sha256": {"type": "string", "pattern": "^[0-9a-f]{64}$"}, "size": {"$ref": "#/$defs/count"}}
|
||||
}
|
||||
}
|
||||
},
|
||||
"$defs": {
|
||||
"id": {"type": "string", "minLength": 1, "pattern": "\\S"},
|
||||
"count": {"type": "integer", "minimum": 0},
|
||||
"ref": {"type": ["string", "null"], "minLength": 1}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
import json
|
||||
import zipfile
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def manifest():
|
||||
return {
|
||||
"format": "viano-bug-case-package", "format_version": 1,
|
||||
"package_id": "stable-package", "package_generation": 0, "case_id": "case-A",
|
||||
"project": {"type": "generic", "profile": "test"},
|
||||
"created_at": "2026-01-01T00:00:00Z", "updated_at": "2026-01-01T00:00:00Z",
|
||||
"state": "NEW", "routing": {"destination": "inbox", "reason": None, "retry_count": 0, "retry_not_before": None},
|
||||
"analysis_revision": 0,
|
||||
"artifacts": {"case": None, "diagnosis": None, "change_request": None},
|
||||
"history": {"analysis_revisions": 0, "reproductions": 0, "evidence": 0, "handoffs": 0, "feedback": 0, "events_ref": None},
|
||||
"integrity": {}
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def make_package(tmp_path, manifest):
|
||||
def make(*, entries=None, document=None, raw=None, name="arbitrary.bin", compression=zipfile.ZIP_STORED):
|
||||
path = tmp_path / name
|
||||
with zipfile.ZipFile(path, "w", compression=compression) as archive:
|
||||
archive.writestr("manifest.json", raw if raw is not None else json.dumps(document if document is not None else manifest))
|
||||
for key, value in (entries or {}).items():
|
||||
archive.writestr(key, value)
|
||||
return path
|
||||
return make
|
||||
@@ -0,0 +1,51 @@
|
||||
|
||||
import pytest
|
||||
from jsonschema import Draft202012Validator
|
||||
import json
|
||||
from importlib.resources import files
|
||||
|
||||
from viano_bug_analysis.package.manifest import manifest_issues, parse_manifest
|
||||
|
||||
|
||||
def test_schema_is_valid():
|
||||
Draft202012Validator.check_schema(json.loads(files("viano_bug_analysis.schemas").joinpath("manifest-v1.schema.json").read_text()))
|
||||
|
||||
|
||||
@pytest.mark.parametrize("field,value", [("format_version", 2), ("format_version", True), ("package_generation", -1), ("analysis_revision", True), ("state", "UNKNOWN"), ("created_at", "2026-01-01"), ("package_id", " ")])
|
||||
def test_invalid_fields(manifest, field, value):
|
||||
manifest[field] = value
|
||||
assert manifest_issues(manifest)
|
||||
|
||||
|
||||
def test_missing_state_and_routing(manifest):
|
||||
del manifest["state"]
|
||||
manifest["status"] = "NEW"
|
||||
assert manifest_issues(manifest)
|
||||
|
||||
|
||||
def test_independent_generation_revision_and_routing(manifest):
|
||||
manifest.update(package_generation=15, analysis_revision=2, state="BLOCKED")
|
||||
manifest["routing"].update(destination="inbox", reason="Dependency unavailable", retry_count=3, retry_not_before="2026-01-02T00:00:00Z")
|
||||
assert not manifest_issues(manifest)
|
||||
|
||||
|
||||
def test_time_order(manifest):
|
||||
manifest["updated_at"] = "2025-01-01T00:00:00Z"
|
||||
assert manifest_issues(manifest)[0].code == "MANIFEST_TIME"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("raw", [b'{"a":1,"a":2}', b'{"a":NaN}', b'\xff'])
|
||||
def test_strict_json(raw):
|
||||
with pytest.raises((ValueError, UnicodeError)):
|
||||
parse_manifest(raw)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("timestamp", ["2026-02-30T00:00:00Z", "2026-01-01T00:00:00", "2026-01-01T00:00:00+00:60", "2026-01-01T00:00:60Z"])
|
||||
def test_invalid_timestamps(timestamp, manifest):
|
||||
manifest["routing"]["retry_not_before"] = timestamp
|
||||
assert manifest_issues(manifest)
|
||||
|
||||
|
||||
def test_lowercase_timestamp(manifest):
|
||||
manifest["created_at"] = "2026-01-01t00:00:00z"
|
||||
assert not manifest_issues(manifest)
|
||||
@@ -0,0 +1,36 @@
|
||||
import pytest
|
||||
|
||||
from viano_bug_analysis.package import PackageReader, PackageReadError
|
||||
|
||||
|
||||
def test_read_only_and_filename_independent(make_package):
|
||||
path = make_package(entries={"unknown/data.bin": b"\x00\xff"})
|
||||
before = path.read_bytes()
|
||||
renamed = path.with_name("different.name")
|
||||
path.rename(renamed)
|
||||
with PackageReader(renamed) as reader:
|
||||
assert reader.manifest["package_id"] == "stable-package"
|
||||
assert reader.read_bytes("unknown/data.bin") == b"\x00\xff"
|
||||
modified = reader.manifest
|
||||
modified["case_id"] = "changed"
|
||||
assert reader.manifest["case_id"] == "case-A"
|
||||
with pytest.raises(PackageReadError):
|
||||
reader.read_bytes("../outside")
|
||||
assert renamed.read_bytes() == before
|
||||
with pytest.raises(RuntimeError):
|
||||
reader.read_bytes("manifest.json")
|
||||
|
||||
|
||||
def test_reader_rejection_and_cleanup(make_package):
|
||||
reader = PackageReader(make_package(raw=b"{}"))
|
||||
with pytest.raises(PackageReadError) as error:
|
||||
with reader:
|
||||
pytest.fail("Invalid package opened")
|
||||
assert not error.value.result.valid
|
||||
assert reader._source is None
|
||||
|
||||
|
||||
def test_reader_missing_file(tmp_path):
|
||||
with pytest.raises(PackageReadError):
|
||||
with PackageReader(tmp_path / "absent"):
|
||||
pass
|
||||
@@ -0,0 +1,119 @@
|
||||
import dataclasses
|
||||
import stat
|
||||
import struct
|
||||
import warnings
|
||||
import zipfile
|
||||
|
||||
import pytest
|
||||
|
||||
from viano_bug_analysis.package import PackageLimits, PackageReadError, PackageReader, PackageValidator
|
||||
|
||||
|
||||
@pytest.mark.parametrize("name", ["../escape", "a/../../escape", "/absolute", "C:/escape", "//server/file", "a\\b", "a/./b", "a//b", "a:stream", "a\x01b"])
|
||||
def test_unsafe_paths(name, make_package):
|
||||
path = make_package(entries={name.replace("\\", "/"): b"x"})
|
||||
if "\\" in name:
|
||||
path.write_bytes(path.read_bytes().replace(name.replace("\\", "/").encode(), name.encode()))
|
||||
assert PackageValidator().validate(path).issues[0].code == "UNSAFE_PATH"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("entries", [{"A": b"x", "a": b"y"}, {"a": b"x", "a/b": b"y"}])
|
||||
def test_collisions(entries, make_package):
|
||||
assert PackageValidator().validate(make_package(entries=entries)).issues[0].code == "ENTRY_COLLISION"
|
||||
|
||||
|
||||
def test_duplicate_manifest(make_package):
|
||||
path = make_package()
|
||||
with warnings.catch_warnings():
|
||||
warnings.simplefilter("ignore", UserWarning)
|
||||
with zipfile.ZipFile(path, "a") as archive:
|
||||
archive.writestr("manifest.json", "{}")
|
||||
assert PackageValidator().validate(path).issues[0].code == "ENTRY_COLLISION"
|
||||
|
||||
|
||||
def test_symlink(make_package):
|
||||
path = make_package()
|
||||
with zipfile.ZipFile(path, "a") as archive:
|
||||
info = zipfile.ZipInfo("link")
|
||||
info.create_system = 3
|
||||
info.external_attr = (stat.S_IFLNK | 0o777) << 16
|
||||
archive.writestr(info, "../outside")
|
||||
assert PackageValidator().validate(path).issues[0].code == "SPECIAL_ENTRY"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("field,value,code", [("max_archive_bytes", 1, "ARCHIVE_LIMIT"), ("max_entries", 1, "ENTRY_LIMIT"), ("max_file_bytes", 1, "FILE_LIMIT"), ("max_manifest_bytes", 1, "FILE_LIMIT"), ("max_total_bytes", 1, "TOTAL_LIMIT")])
|
||||
def test_configurable_limits(field, value, code, make_package):
|
||||
limits = dataclasses.replace(PackageLimits(), **{field: value})
|
||||
path = make_package(entries={"data": b"abc"})
|
||||
assert PackageValidator(limits).validate(path).issues[0].code == code
|
||||
assert PackageValidator().validate(path).valid
|
||||
|
||||
|
||||
def test_compression_ratio(make_package):
|
||||
path = make_package(entries={"data": b"a" * 100_000}, compression=zipfile.ZIP_DEFLATED)
|
||||
assert PackageValidator().validate(path).issues[0].code == "RATIO_LIMIT"
|
||||
assert PackageValidator(PackageLimits(max_compression_ratio=10000)).validate(path).valid
|
||||
|
||||
|
||||
@pytest.mark.parametrize("value", [0, -1, float("inf"), True, 1.5])
|
||||
def test_invalid_limit_configuration(value):
|
||||
with pytest.raises(ValueError):
|
||||
PackageLimits(max_entries=value)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("damage", ["not_zip", "truncated", "crc"])
|
||||
def test_corrupt_archive(damage, make_package):
|
||||
path = make_package(entries={"data": b"abcdef"})
|
||||
raw = bytearray(path.read_bytes())
|
||||
if damage == "not_zip":
|
||||
raw = bytearray(b"not an archive")
|
||||
elif damage == "truncated":
|
||||
raw = raw[:-25]
|
||||
else:
|
||||
with zipfile.ZipFile(path) as archive:
|
||||
offset = archive.getinfo("data").header_offset
|
||||
name_len, extra_len = struct.unpack_from("<HH", raw, offset + 26)
|
||||
raw[offset + 30 + name_len + extra_len] ^= 1
|
||||
path.write_bytes(raw)
|
||||
assert not PackageValidator().validate(path).valid
|
||||
with pytest.raises(PackageReadError):
|
||||
with PackageReader(path):
|
||||
pass
|
||||
|
||||
|
||||
def test_unsupported_compression(make_package):
|
||||
path = make_package(entries={"data": b"abc"}, compression=zipfile.ZIP_BZIP2)
|
||||
assert PackageValidator().validate(path).issues[0].code == "UNSUPPORTED_ENTRY"
|
||||
|
||||
|
||||
def test_missing_root_manifest(tmp_path):
|
||||
path = tmp_path / "package"
|
||||
with zipfile.ZipFile(path, "w") as archive:
|
||||
archive.writestr("nested/manifest.json", "{}")
|
||||
assert PackageValidator().validate(path).issues[0].code == "MANIFEST_MISSING"
|
||||
|
||||
|
||||
def test_encrypted_flag(make_package):
|
||||
path = make_package()
|
||||
raw = bytearray(path.read_bytes())
|
||||
for signature, field_offset in [(b"PK\x03\x04", 6), (b"PK\x01\x02", 8)]:
|
||||
offset = raw.index(signature) + field_offset
|
||||
struct.pack_into("<H", raw, offset, struct.unpack_from("<H", raw, offset)[0] | 1)
|
||||
path.write_bytes(raw)
|
||||
assert PackageValidator().validate(path).issues[0].code == "UNSUPPORTED_ENTRY"
|
||||
|
||||
|
||||
def test_nul_filename(make_package):
|
||||
path = make_package(entries={"nulXname": b"x"})
|
||||
path.write_bytes(path.read_bytes().replace(b"nulXname", b"nul\x00name"))
|
||||
assert PackageValidator().validate(path).issues[0].code == "UNSAFE_PATH"
|
||||
|
||||
|
||||
def test_explicit_directories(make_package):
|
||||
assert PackageValidator().validate(make_package(entries={"unknown/": b"", "unknown/file": b"x"})).valid
|
||||
|
||||
|
||||
def test_bounded_read():
|
||||
import io
|
||||
with pytest.raises(ValueError):
|
||||
PackageValidator._read_bounded(io.BytesIO(b"abcd"), 3)
|
||||
@@ -0,0 +1,50 @@
|
||||
import hashlib
|
||||
|
||||
import pytest
|
||||
|
||||
from viano_bug_analysis.package import PackageValidator
|
||||
|
||||
|
||||
def test_minimal_package(make_package):
|
||||
result = PackageValidator().validate(make_package())
|
||||
assert result.valid
|
||||
assert result.manifest["case_id"] == "case-A"
|
||||
|
||||
|
||||
def test_returned_package_and_unknown_data(manifest, make_package):
|
||||
payload = b"immutable case"
|
||||
manifest.update(state="REOPENED", package_generation=8, analysis_revision=3)
|
||||
manifest["artifacts"]["case"] = "case/case.json"
|
||||
manifest["history"]["analysis_revisions"] = 3
|
||||
manifest["integrity"]["case/case.json"] = {"size": len(payload), "sha256": hashlib.sha256(payload).hexdigest()}
|
||||
manifest["extension"] = {"vendor": "anything"}
|
||||
result = PackageValidator().validate(make_package(entries={"case/case.json": payload, "unknown/data.bin": b"opaque"}))
|
||||
assert result.valid
|
||||
assert result.manifest["extension"] == {"vendor": "anything"}
|
||||
assert "unknown/data.bin" in result.files
|
||||
|
||||
|
||||
@pytest.mark.parametrize("kind", ["missing", "hash", "size"])
|
||||
def test_invalid_integrity(kind, manifest, make_package):
|
||||
manifest["integrity"]["original/raw"] = {"size": 3, "sha256": hashlib.sha256(b"abc").hexdigest()}
|
||||
entries = {} if kind == "missing" else {"original/raw": b"abc"}
|
||||
if kind == "hash":
|
||||
manifest["integrity"]["original/raw"]["sha256"] = "0" * 64
|
||||
if kind == "size":
|
||||
manifest["integrity"]["original/raw"]["size"] = 4
|
||||
assert not PackageValidator().validate(make_package(entries=entries)).valid
|
||||
|
||||
|
||||
@pytest.mark.parametrize("ref", ["absent.json", "../escape", "/absolute", "manifest.json"])
|
||||
def test_invalid_pointer(ref, manifest, make_package):
|
||||
manifest["artifacts"]["case"] = ref
|
||||
assert not PackageValidator().validate(make_package()).valid
|
||||
|
||||
|
||||
@pytest.mark.parametrize("raw", [b"{}", b"[]", b"{", b'{"state":"NEW","state":"REOPENED"}', b"\xff", b"[" * 2000])
|
||||
def test_invalid_manifest_controlled(raw, make_package):
|
||||
assert not PackageValidator().validate(make_package(raw=raw)).valid
|
||||
|
||||
|
||||
def test_missing_archive(tmp_path):
|
||||
assert PackageValidator().validate(tmp_path / "missing").issues[0].code == "ARCHIVE_ERROR"
|
||||
Reference in new issue
Block a user