Add expected behavior resolver and read-only research contracts
This commit is contained in:
1 parent
741c1ca4a5
commit
50b9d6e021
12 files changed
+640
-9
No files matched your search
@@ -22,3 +22,5 @@ python -m venv .venv
|
||||
The lifecycle foundation supports one-pass inbox processing and resumable package
|
||||
checkpoints and a deterministic generic CaseBuilder. It includes no analysis worker or LLM,
|
||||
case interpretation, hypothesis generation or investigation tools.
|
||||
|
||||
Expected-behavior resolution and the read-only provider contract: [expected-behavior-v1.md](docs/expected-behavior-v1.md).
|
||||
@@ -0,0 +1,23 @@
|
||||
# Expected behavior and research v1
|
||||
|
||||
The resolver is part of CaseBuilder, project independent and deterministic. Architecture sections 10 and 27 remain authoritative. No LLM, online provider, interpretation or investigation is implemented here.
|
||||
|
||||
Tester declarations are retained verbatim with stable E IDs and source provenance. Only explicit `expected_behavior_complete: true` is sufficient; partial, unspecified, missing or contradictory declarations require research. Natural-language completeness and contradictions are not inferred. Explicit claim conflicts remain visible and cannot be cleared by research. Tester statements are never replaced by external statements.
|
||||
|
||||
`research.request` in the canonical case contains a versioned request or null for complete tester declarations. Requests contain identities, revision, project metadata, report titles/summaries, entity types/IDs and tester expectations. Local source, database and runtime payloads are excluded. Report text remains untrusted data; a future provider must enforce the intended-behavior policy independently. External sources establish intended behavior, never the state of a local implementation.
|
||||
|
||||
## Provider boundary
|
||||
|
||||
`ExternalResearchProvider.lookup(request, timeout_seconds=...)` is read-only. Providers must implement their transport deadline. This synchronous interface cannot forcibly terminate arbitrary Python provider code. `ResearchService` invokes the provider once, validates the entire response, verifies the request hash and citation indices, applies configurable result limits and returns deterministic append-only request/result artifacts. There is no hidden retry loop. Timeout/network errors become `RESEARCH_UNAVAILABLE`; other provider failures are controlled errors. A future worker can use the existing CheckpointStore BLOCKED/routing/retry mechanism; the standalone service does not change lifecycle state on failure.
|
||||
|
||||
Research-result v1 requires provider identity, request hash, intended-behavior scope, complete/partial/not_found status and source URL/title/retrieval timestamp/excerpt for every cited statement. Citation indices must exist. Completeness is the provider's explicit declaration, not a verification of external truth. No search, network fetching or source credibility assessment occurs in this milestone.
|
||||
|
||||
## Package lifecycle
|
||||
|
||||
Before CaseBuilder completion, advance the admitted case into BUILDING_CASE, then call `ResearchService(provider).enrich(active_package, destination)`. It builds the bounded request, stores immutable request/result files under `research/<content-hash>/`, adds a new input index and updates only the manifest's case_inputs pointer through PackageWriter. Intake that new generation into CheckpointStore and run CaseBuilder. Existing files remain byte-identical, package_id and analysis_revision remain stable and package_generation increases. A completed revision must be reopened before enrichment. Complete tester expectations bypass automatic research.
|
||||
|
||||
The explicit `case-inputs.expected_research` list designates structured research-result documents. Arbitrary context.research files remain opaque and cannot silently become expected behavior. Each result has a sibling request.json; its hash, identity and nonfuture revision are checked. Historical research remains in the case across revisions; it is traceable historical input, not silently re-fetched or asserted current. Applicability to changed project versions will require provider/policy decisions in a later milestone.
|
||||
|
||||
CaseBuilder includes external statements in expected_behavior, with tester/external_research/combined provenance and complete/partial status, source details and stable IDs. Facts and implementation context are not modified by research. The deterministic summary presents citations and excerpts without additional interpretation. Completed CaseBuilder retries reuse confirmed output.
|
||||
|
||||
ResearchLimits defaults (1 MiB result, 30 seconds provider deadline) are configurable operational safeguards, not format limits. Generic CaseBuilder and ZIP limits still apply. A test provider is exercised in tests; no production provider or dependencies on external accounts are introduced.
|
||||
@@ -9,7 +9,7 @@ from viano_bug_analysis.package.validator import CHUNK_SIZE
|
||||
|
||||
from .contracts import read_json, require_file, validate
|
||||
from .errors import CaseBuildError
|
||||
from .expected_behavior import collect_expected_behavior
|
||||
from .expected_behavior import ExpectedBehaviorResolver, collect_expected_behavior
|
||||
from .report_collector import collect_statements, provenance
|
||||
from .reproduction_collector import collect_reproductions
|
||||
|
||||
@@ -135,7 +135,7 @@ class CaseAssembler:
|
||||
raise CaseBuildError("PREVIOUS_INPUT_CHANGED", "A source referenced by the prior case changed", record["path"])
|
||||
if index_ref:
|
||||
add(index_ref, "index")
|
||||
for field, role in (("reports", "report"), ("reproductions", "reproduction"), ("attachments", "attachment")):
|
||||
for field, role in (("reports", "report"), ("reproductions", "reproduction"), ("attachments", "attachment"), ("expected_research", "expected_research")):
|
||||
paths = index.get(field, [])
|
||||
if len(paths) != len(set(paths)):
|
||||
raise CaseBuildError("INPUT_DUPLICATE", "A source is listed twice in the same role", index_ref)
|
||||
@@ -167,6 +167,8 @@ class CaseAssembler:
|
||||
hypotheses = collect_statements(reports, "tester_hypotheses", "T")
|
||||
expected = collect_expected_behavior(reports)
|
||||
reproduction_records, facts = collect_reproductions(reproductions, self.limits.max_inline_fact_bytes)
|
||||
research_documents = [(path, load(path, "research-result")) for path, role in roles if role == "expected_research"]
|
||||
expected = ExpectedBehaviorResolver().resolve(expected, research_documents)
|
||||
if previous:
|
||||
old_humans = previous["human_observations"]
|
||||
observations = _stable_records(old_humans["observations"], observations, "O")
|
||||
@@ -182,6 +184,15 @@ class CaseAssembler:
|
||||
record = entities.setdefault(identity, {"type": identity[0], "id": identity[1], "roles": [], "provenance": []})
|
||||
record["roles"] = sorted(set(record["roles"]) | set(entity.get("roles", [])))
|
||||
record["provenance"].append(provenance(path, f"/entities/{index}", "report" if document["format"] == "viano-bug-report" else "reproduction_record", entity))
|
||||
research_request = ExpectedBehaviorResolver().request(manifest, [d for _, d in reports], [entities[k] for k in sorted(entities)], collect_expected_behavior(reports))
|
||||
for path, result in research_documents:
|
||||
request_ref = path.rsplit("/", 1)[0] + "/request.json"
|
||||
request = load(request_ref, "research-request")
|
||||
add(request_ref, "research")
|
||||
if hashlib.sha256(json_bytes(request)).hexdigest() != result["request_hash"]:
|
||||
raise CaseBuildError("RESEARCH_REQUEST_MISMATCH", "Stored research request hash does not match", path)
|
||||
if request["case_id"] != manifest["case_id"] or request["package_id"] != manifest["package_id"] or request["analysis_revision"] > manifest["analysis_revision"]:
|
||||
raise CaseBuildError("RESEARCH_IDENTITY", "Research belongs to another case or future revision", path)
|
||||
issues = []
|
||||
|
||||
def issue(code, message, refs=None):
|
||||
@@ -199,7 +210,7 @@ class CaseAssembler:
|
||||
issue("MISSING_OBSERVATIONS", "No direct observations were supplied")
|
||||
if not reproductions:
|
||||
issue("MISSING_REPRODUCTION", "No structured reproduction was supplied")
|
||||
if expected["completeness"] != "reported_complete":
|
||||
if expected["completeness"] not in {"reported_complete", "researched_complete"}:
|
||||
issue("EXPECTED_BEHAVIOR_" + expected["completeness"].upper(), "Expected-behavior completeness is " + expected["completeness"],
|
||||
[{"source_ref": path, "json_pointer": "/expected_behavior_complete"} for path, report in reports if "expected_behavior_complete" in report])
|
||||
for category, records in (("observation", observations), ("expected_behavior", expected["statements"])):
|
||||
@@ -214,6 +225,12 @@ class CaseAssembler:
|
||||
issue("CONFLICTING_CLAIMS", f"Different explicit values for {category} claim {claim_key}", refs)
|
||||
if category == "expected_behavior":
|
||||
expected["completeness"] = "conflicting"
|
||||
if expected["completeness"] == "researched_complete":
|
||||
research_request = None
|
||||
if expected["completeness"] == "conflicting":
|
||||
tester_expected = collect_expected_behavior(reports)
|
||||
tester_expected["completeness"] = "conflicting"
|
||||
research_request = ExpectedBehaviorResolver().request(manifest, [d for _, d in reports], [entities[k] for k in sorted(entities)], tester_expected)
|
||||
records = [inventory[path] | {"role": role} for path, role in roles]
|
||||
classified = {path for path, _ in roles}
|
||||
raw_inventory = [value for name, value in inventory.items() if name not in excluded]
|
||||
@@ -234,7 +251,7 @@ class CaseAssembler:
|
||||
"entities": [entities[identity] for identity in sorted(entities)],
|
||||
"runtime_facts": {"facts": facts, "raw_refs": refs_for("runtime")},
|
||||
"database_context": {"refs": refs_for("database")}, "source_context": {"refs": refs_for("source")},
|
||||
"research": {"refs": refs_for("research"), "performed": False},
|
||||
"research": {"refs": refs_for("research") + refs_for("expected_research"), "performed": bool(research_documents), "request": research_request},
|
||||
"data_quality": {"issues": issues, "coverage": {"reports": len(reports), "reproductions": len(reproductions), "observations": len(observations)}},
|
||||
"inputs": {"index_ref": index_ref, "records": records,
|
||||
"unclassified": [value for name, value in inventory.items() if name not in classified and name not in excluded],
|
||||
|
||||
@@ -17,3 +17,41 @@ def collect_expected_behavior(reports: list[tuple[str, dict]]) -> dict:
|
||||
status = "unspecified"
|
||||
return {"statements": statements, "completeness": status,
|
||||
"provenance": "tester" if statements else "none"}
|
||||
|
||||
|
||||
class ExpectedBehaviorResolver:
|
||||
"""Resolve explicit declarations without interpreting natural language."""
|
||||
|
||||
def request(self, identity: dict, reports: list[dict], entities: list[dict], expected: dict) -> dict | None:
|
||||
if expected["completeness"] in {"reported_complete", "researched_complete"}:
|
||||
return None
|
||||
return {"format": "viano-research-request", "format_version": 1,
|
||||
"case_id": identity["case_id"], "package_id": identity["package_id"],
|
||||
"analysis_revision": identity["analysis_revision"],
|
||||
"purpose": "expected_behavior", "scope": "intended_behavior_only",
|
||||
"project": identity["project"],
|
||||
"reports": [{"title": r.get("title"), "summary": r.get("summary")} for r in reports],
|
||||
"entities": [{"type": e["type"], "id": e["id"]} for e in entities],
|
||||
"tester_expectations": [r["text"] for r in expected["statements"] if r["provenance"]["kind"] == "tester"]}
|
||||
|
||||
def resolve(self, expected: dict, research: list[tuple[str, dict]]) -> dict:
|
||||
import copy
|
||||
from .errors import CaseBuildError
|
||||
result = copy.deepcopy(expected)
|
||||
statuses = []
|
||||
for path, document in research:
|
||||
for index, statement in enumerate(document["statements"]):
|
||||
if any(i >= len(document["sources"]) for i in statement["source_indices"]):
|
||||
raise CaseBuildError("RESEARCH_SOURCE", "Research statement references a missing source", path)
|
||||
record = {"id": f"E{len(result['statements']) + 1:04d}", "text": statement["text"],
|
||||
"provenance": {"kind": "external_research", "source_ref": path,
|
||||
"json_pointer": f"/statements/{index}", "author": None, "observed_at": None},
|
||||
"external_sources": [copy.deepcopy(document["sources"][i]) for i in statement["source_indices"]]}
|
||||
result["statements"].append(record)
|
||||
if document["statements"]:
|
||||
statuses.append(document["status"])
|
||||
if statuses:
|
||||
result["provenance"] = "combined" if expected["statements"] else "external_research"
|
||||
if result["completeness"] not in {"reported_complete", "conflicting"}:
|
||||
result["completeness"] = "researched_complete" if set(statuses) == {"complete"} else "researched_partial"
|
||||
return result
|
||||
@@ -46,6 +46,9 @@ def render_summary(case: dict) -> str:
|
||||
f" Author: {_literal(source['author']) if source['author'] is not None else '(not supplied)'}",
|
||||
f" Observed at: {_literal(source['observed_at']) if source['observed_at'] is not None else '(not supplied)'}"]
|
||||
lines.extend(_origin_lines(source.get("origin")))
|
||||
for external in record.get("external_sources", []):
|
||||
lines.append(f" External source: {_literal(external['title'])} {_literal(external['url'])}; retrieved: {_literal(external['retrieved_at'])}")
|
||||
lines.append(f" Source excerpt: {_literal(external['excerpt'])}")
|
||||
if "claim" in record:
|
||||
lines.append(f" Claim: {_literal(record['claim']['key'])} = {_literal(json.dumps(record['claim']['value'], sort_keys=True, ensure_ascii=False))}")
|
||||
lines += ["", f"Expected-behavior completeness: {case['expected_behavior']['completeness']}", "", "## Reproductions", ""]
|
||||
@@ -64,6 +67,9 @@ def render_summary(case: dict) -> str:
|
||||
for source in entity["provenance"]:
|
||||
lines.append(f" Source: {_literal(source['source_ref'])} {_literal(source['json_pointer'])}")
|
||||
lines.extend(_origin_lines(source.get("origin")))
|
||||
for external in record.get("external_sources", []):
|
||||
lines.append(f" External source: {_literal(external['title'])} {_literal(external['url'])}; retrieved: {_literal(external['retrieved_at'])}")
|
||||
lines.append(f" Source excerpt: {_literal(external['excerpt'])}")
|
||||
lines += ["", "## Facts supplied by reproduction records", ""]
|
||||
for fact in case["runtime_facts"]["facts"]:
|
||||
source = fact["provenance"]
|
||||
|
||||
@@ -0,0 +1,3 @@
|
||||
from .provider import ExternalResearchProvider, ResearchError, ResearchLimits, ResearchService
|
||||
|
||||
__all__ = ["ExternalResearchProvider", "ResearchError", "ResearchLimits", "ResearchService"]
|
||||
@@ -0,0 +1,95 @@
|
||||
"""Read-only provider boundary; no network implementation or shell access."""
|
||||
import copy
|
||||
import hashlib
|
||||
from dataclasses import dataclass
|
||||
from typing import Protocol
|
||||
|
||||
from viano_bug_analysis.case.contracts import validate
|
||||
from viano_bug_analysis.case.errors import CaseBuildError
|
||||
from viano_bug_analysis.lifecycle.storage import json_bytes
|
||||
|
||||
|
||||
class ResearchError(CaseBuildError):
|
||||
pass
|
||||
|
||||
|
||||
class ExternalResearchProvider(Protocol):
|
||||
def lookup(self, request: dict, *, timeout_seconds: float) -> dict:
|
||||
"""Return a research-result v1 document; respect the supplied deadline."""
|
||||
...
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ResearchLimits:
|
||||
max_result_bytes: int = 1024 * 1024
|
||||
timeout_seconds: float = 30.0
|
||||
|
||||
def __post_init__(self):
|
||||
if isinstance(self.max_result_bytes, bool) or not isinstance(self.max_result_bytes, int) or self.max_result_bytes <= 0:
|
||||
raise ValueError("Result limit must be a positive integer")
|
||||
if isinstance(self.timeout_seconds, bool) or not isinstance(self.timeout_seconds, (float, int)) or not 0 < self.timeout_seconds < float("inf"):
|
||||
raise ValueError("Timeout must be positive and finite")
|
||||
|
||||
|
||||
class ResearchService:
|
||||
def __init__(self, provider: ExternalResearchProvider, limits: ResearchLimits | None = None):
|
||||
self.provider = provider
|
||||
self.limits = limits or ResearchLimits()
|
||||
|
||||
def collect(self, request: dict) -> tuple[str, dict[str, bytes]]:
|
||||
validate(request, "research-request")
|
||||
expected_hash = hashlib.sha256(json_bytes(request)).hexdigest()
|
||||
try:
|
||||
result = copy.deepcopy(self.provider.lookup(copy.deepcopy(request), timeout_seconds=self.limits.timeout_seconds))
|
||||
except (TimeoutError, ConnectionError, OSError) as error:
|
||||
raise ResearchError("RESEARCH_UNAVAILABLE", "Research provider temporarily unavailable") from error
|
||||
except Exception as error:
|
||||
raise ResearchError("RESEARCH_PROVIDER", "Research provider failed") from error
|
||||
try:
|
||||
data = json_bytes(result)
|
||||
except (ValueError, TypeError, RecursionError) as error:
|
||||
raise ResearchError("RESEARCH_JSON", "Provider returned unserializable data") from error
|
||||
if len(data) > self.limits.max_result_bytes:
|
||||
raise ResearchError("RESEARCH_LIMIT", "Result exceeds configured operational limit")
|
||||
validate(result, "research-result")
|
||||
if result["request_hash"] != expected_hash:
|
||||
raise ResearchError("RESEARCH_REQUEST_MISMATCH", "Provider result belongs to another request")
|
||||
for statement in result["statements"]:
|
||||
if any(i >= len(result["sources"]) for i in statement["source_indices"]):
|
||||
raise ResearchError("RESEARCH_SOURCE", "Provider cited a missing source")
|
||||
base = "research/" + hashlib.sha256(json_bytes(request) + data).hexdigest()
|
||||
return base + "/result.json", {base + "/request.json": json_bytes(request), base + "/result.json": data}
|
||||
|
||||
def enrich(self, source, destination):
|
||||
"""Publish research inputs before CaseBuilder completion; source stays intact.
|
||||
|
||||
Intake the resulting generation into CheckpointStore, then run CaseBuilder.
|
||||
A completed revision must be reopened before enrichment.
|
||||
"""
|
||||
from viano_bug_analysis.case.assembler import CaseAssembler
|
||||
from viano_bug_analysis.case.contracts import read_json
|
||||
from viano_bug_analysis.lifecycle.checkpoints import checkpoint_chain
|
||||
from viano_bug_analysis.package import PackageReader, PackageWriter
|
||||
|
||||
with PackageReader(source) as reader:
|
||||
manifest = reader.manifest
|
||||
if manifest["state"] != "BUILDING_CASE":
|
||||
raise ResearchError("RESEARCH_STATE", "Research enrichment requires BUILDING_CASE")
|
||||
if any(cp["details"].get("operation") == "case-built" and cp["analysis_revision"] == manifest["analysis_revision"] for _, cp in checkpoint_chain(reader)):
|
||||
raise ResearchError("STALE_CASE", "Reopen the completed revision before research enrichment")
|
||||
assembler = CaseAssembler()
|
||||
case = assembler.assemble(reader)
|
||||
request = case["research"]["request"]
|
||||
if request is None:
|
||||
raise ResearchError("RESEARCH_NOT_REQUIRED", "Complete tester expectations need no automatic research")
|
||||
index_ref = manifest["artifacts"].get("case_inputs")
|
||||
index = read_json(reader, index_ref, "case-inputs", assembler.limits.max_json_bytes) if index_ref else {"format": "viano-case-inputs", "format_version": 1}
|
||||
result_ref, additions = self.collect(request)
|
||||
index.setdefault("expected_research", [])
|
||||
if result_ref in index["expected_research"]:
|
||||
raise ResearchError("RESEARCH_EXISTS", "Identical research already recorded")
|
||||
index["expected_research"].append(result_ref)
|
||||
new_index = result_ref.rsplit("/", 1)[0] + "/case-inputs.json"
|
||||
additions[new_index] = json_bytes(index)
|
||||
return PackageWriter().write(source, destination, additions=additions,
|
||||
manifest_updates={"artifacts": {**manifest["artifacts"], "case_inputs": new_index}})
|
||||
@@ -66,6 +66,15 @@
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
}
|
||||
},
|
||||
"expected_research": {
|
||||
"type": "array",
|
||||
"uniqueItems": true,
|
||||
"items": {
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"pattern": "\\S"
|
||||
}
|
||||
}
|
||||
},
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema"
|
||||
|
||||
@@ -161,13 +161,17 @@
|
||||
"unspecified",
|
||||
"reported_complete",
|
||||
"reported_partial",
|
||||
"conflicting"
|
||||
"conflicting",
|
||||
"researched_complete",
|
||||
"researched_partial"
|
||||
]
|
||||
},
|
||||
"provenance": {
|
||||
"enum": [
|
||||
"tester",
|
||||
"none"
|
||||
"none",
|
||||
"external_research",
|
||||
"combined"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -393,7 +397,7 @@
|
||||
}
|
||||
},
|
||||
"performed": {
|
||||
"const": false
|
||||
"type": "boolean"
|
||||
}
|
||||
}
|
||||
},
|
||||
@@ -523,7 +527,8 @@
|
||||
"database",
|
||||
"source",
|
||||
"research",
|
||||
"attachment"
|
||||
"attachment",
|
||||
"expected_research"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -564,7 +569,8 @@
|
||||
"enum": [
|
||||
"tester",
|
||||
"reproduction_record",
|
||||
"report"
|
||||
"report",
|
||||
"external_research"
|
||||
]
|
||||
},
|
||||
"source_ref": {
|
||||
@@ -659,6 +665,40 @@
|
||||
},
|
||||
"claim": {
|
||||
"$ref": "#/$defs/claim"
|
||||
},
|
||||
"external_sources": {
|
||||
"type": "array",
|
||||
"minItems": 1,
|
||||
"items": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"url",
|
||||
"title",
|
||||
"retrieved_at",
|
||||
"excerpt"
|
||||
],
|
||||
"properties": {
|
||||
"url": {
|
||||
"type": "string",
|
||||
"pattern": "^https?://[^\\s/]+(?:/[^\\s]*)?$"
|
||||
},
|
||||
"title": {
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"pattern": "\\S"
|
||||
},
|
||||
"retrieved_at": {
|
||||
"type": "string",
|
||||
"format": "date-time"
|
||||
},
|
||||
"excerpt": {
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"pattern": "\\S"
|
||||
}
|
||||
},
|
||||
"additionalProperties": false
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
@@ -0,0 +1,105 @@
|
||||
{
|
||||
"type": "object",
|
||||
"required": [
|
||||
"format",
|
||||
"format_version",
|
||||
"case_id",
|
||||
"package_id",
|
||||
"analysis_revision",
|
||||
"purpose",
|
||||
"scope",
|
||||
"project",
|
||||
"reports",
|
||||
"entities",
|
||||
"tester_expectations"
|
||||
],
|
||||
"properties": {
|
||||
"format": {
|
||||
"const": "viano-research-request"
|
||||
},
|
||||
"format_version": {
|
||||
"const": 1
|
||||
},
|
||||
"case_id": {
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"pattern": "\\S"
|
||||
},
|
||||
"package_id": {
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"pattern": "\\S"
|
||||
},
|
||||
"analysis_revision": {
|
||||
"type": "integer",
|
||||
"minimum": 1
|
||||
},
|
||||
"purpose": {
|
||||
"const": "expected_behavior"
|
||||
},
|
||||
"scope": {
|
||||
"const": "intended_behavior_only"
|
||||
},
|
||||
"project": {
|
||||
"type": "object"
|
||||
},
|
||||
"reports": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"title",
|
||||
"summary"
|
||||
],
|
||||
"properties": {
|
||||
"title": {
|
||||
"type": [
|
||||
"string",
|
||||
"null"
|
||||
]
|
||||
},
|
||||
"summary": {
|
||||
"type": [
|
||||
"string",
|
||||
"null"
|
||||
]
|
||||
}
|
||||
},
|
||||
"additionalProperties": false
|
||||
}
|
||||
},
|
||||
"entities": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"type",
|
||||
"id"
|
||||
],
|
||||
"properties": {
|
||||
"type": {
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"pattern": "\\S"
|
||||
},
|
||||
"id": {
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"pattern": "\\S"
|
||||
}
|
||||
},
|
||||
"additionalProperties": false
|
||||
}
|
||||
},
|
||||
"tester_expectations": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"pattern": "\\S"
|
||||
}
|
||||
}
|
||||
},
|
||||
"additionalProperties": false,
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema"
|
||||
}
|
||||
@@ -0,0 +1,127 @@
|
||||
{
|
||||
"type": "object",
|
||||
"required": [
|
||||
"format",
|
||||
"format_version",
|
||||
"request_hash",
|
||||
"provider",
|
||||
"scope",
|
||||
"status",
|
||||
"sources",
|
||||
"statements"
|
||||
],
|
||||
"properties": {
|
||||
"format": {
|
||||
"const": "viano-research-result"
|
||||
},
|
||||
"format_version": {
|
||||
"const": 1
|
||||
},
|
||||
"request_hash": {
|
||||
"type": "string",
|
||||
"pattern": "^[0-9a-f]{64}$"
|
||||
},
|
||||
"provider": {
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"pattern": "\\S"
|
||||
},
|
||||
"scope": {
|
||||
"const": "intended_behavior_only"
|
||||
},
|
||||
"status": {
|
||||
"enum": [
|
||||
"complete",
|
||||
"partial",
|
||||
"not_found"
|
||||
]
|
||||
},
|
||||
"sources": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"url",
|
||||
"title",
|
||||
"retrieved_at",
|
||||
"excerpt"
|
||||
],
|
||||
"properties": {
|
||||
"url": {
|
||||
"type": "string",
|
||||
"pattern": "^https?://[^\\s/]+(?:/[^\\s]*)?$"
|
||||
},
|
||||
"title": {
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"pattern": "\\S"
|
||||
},
|
||||
"retrieved_at": {
|
||||
"type": "string",
|
||||
"format": "date-time"
|
||||
},
|
||||
"excerpt": {
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"pattern": "\\S"
|
||||
}
|
||||
},
|
||||
"additionalProperties": false
|
||||
}
|
||||
},
|
||||
"statements": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"text",
|
||||
"source_indices"
|
||||
],
|
||||
"properties": {
|
||||
"text": {
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"pattern": "\\S"
|
||||
},
|
||||
"source_indices": {
|
||||
"type": "array",
|
||||
"minItems": 1,
|
||||
"uniqueItems": true,
|
||||
"items": {
|
||||
"type": "integer",
|
||||
"minimum": 0
|
||||
}
|
||||
}
|
||||
},
|
||||
"additionalProperties": false
|
||||
}
|
||||
}
|
||||
},
|
||||
"additionalProperties": false,
|
||||
"allOf": [
|
||||
{
|
||||
"if": {
|
||||
"properties": {
|
||||
"status": {
|
||||
"const": "not_found"
|
||||
}
|
||||
}
|
||||
},
|
||||
"then": {
|
||||
"properties": {
|
||||
"statements": {
|
||||
"maxItems": 0
|
||||
}
|
||||
}
|
||||
},
|
||||
"else": {
|
||||
"properties": {
|
||||
"statements": {
|
||||
"minItems": 1
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema"
|
||||
}
|
||||
@@ -0,0 +1,166 @@
|
||||
import hashlib
|
||||
import json
|
||||
|
||||
import pytest
|
||||
|
||||
from viano_bug_analysis.case.expected_behavior import ExpectedBehaviorResolver, collect_expected_behavior
|
||||
from viano_bug_analysis.case import CaseAssembler, CaseBuildError
|
||||
from viano_bug_analysis.lifecycle import State
|
||||
from viano_bug_analysis.lifecycle.storage import json_bytes
|
||||
from viano_bug_analysis.package import PackageReader, PackageWriter, PackageValidator
|
||||
from viano_bug_analysis.research import ResearchError, ResearchLimits, ResearchService
|
||||
from test_case_builder import case_setup
|
||||
|
||||
|
||||
def request():
|
||||
return ExpectedBehaviorResolver().request(
|
||||
{"case_id": "case-A", "package_id": "stable-package", "analysis_revision": 1, "project": {}}, [], [],
|
||||
{"statements": [], "completeness": "missing", "provenance": "none"})
|
||||
|
||||
|
||||
def result(req, status="complete"):
|
||||
return {"format": "viano-research-result", "format_version": 1, "request_hash": hashlib.sha256(json_bytes(req)).hexdigest(),
|
||||
"provider": "fixture", "scope": "intended_behavior_only", "status": status,
|
||||
"sources": [{"url": "https://example.org/manual", "title": "Manual", "retrieved_at": "2026-01-01T00:00:00Z", "excerpt": "The switch turns on."}],
|
||||
"statements": [] if status == "not_found" else [{"text": "The switch turns on.", "source_indices": [0]}]}
|
||||
|
||||
|
||||
class FixtureProvider:
|
||||
def __init__(self, response):
|
||||
self.response = response
|
||||
|
||||
def lookup(self, request, *, timeout_seconds):
|
||||
assert timeout_seconds > 0
|
||||
return self.response
|
||||
|
||||
|
||||
@pytest.mark.parametrize("flag,status", [(True,"reported_complete"),(False,"reported_partial"),(None,"unspecified")])
|
||||
def test_explicit_completeness(flag, status):
|
||||
expected = collect_expected_behavior([("report", {"expected_behavior": [{"text":"Expected"}], "expected_behavior_complete": flag})])
|
||||
assert expected["completeness"] == status
|
||||
req = ExpectedBehaviorResolver().request({"case_id":"c","package_id":"p","analysis_revision":1,"project":{}}, [], [], expected)
|
||||
assert (req is None) == (flag is True)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("status,completeness", [("complete","researched_complete"),("partial","researched_partial"),("not_found","missing")])
|
||||
def test_resolution(status, completeness):
|
||||
expected = {"statements":[],"completeness":"missing","provenance":"none"}
|
||||
resolved = ExpectedBehaviorResolver().resolve(expected, [("research/result.json",result(request(), status))])
|
||||
assert resolved["completeness"] == completeness
|
||||
assert expected["statements"] == []
|
||||
if resolved["statements"]:
|
||||
assert resolved["statements"][0]["external_sources"][0]["url"].startswith("https:")
|
||||
|
||||
|
||||
def test_combined_does_not_override_tester_or_conflict():
|
||||
expected = collect_expected_behavior([("r", {"expected_behavior":[{"text":"Tester statement"}],"expected_behavior_complete":False})])
|
||||
resolved = ExpectedBehaviorResolver().resolve(expected, [("research/result.json",result(request()))])
|
||||
assert resolved["provenance"] == "combined"
|
||||
assert resolved["statements"][0] == expected["statements"][0]
|
||||
expected["completeness"] = "conflicting"
|
||||
assert ExpectedBehaviorResolver().resolve(expected, [("r",result(request()))])["completeness"] == "conflicting"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("mutation", ["identity","source","scope","time"])
|
||||
def test_invalid_provider_is_rejected(mutation):
|
||||
response = result(request())
|
||||
if mutation == "identity": response["request_hash"] = "0" * 64
|
||||
if mutation == "source": response["statements"][0]["source_indices"] = [2]
|
||||
if mutation == "scope": response["scope"] = "local_implementation"
|
||||
if mutation == "time": response["sources"][0]["retrieved_at"] = "2026-01-01"
|
||||
with pytest.raises(CaseBuildError): ResearchService(FixtureProvider(response)).collect(request())
|
||||
|
||||
|
||||
def test_provider_failure_controlled():
|
||||
class Offline:
|
||||
def lookup(self, request, *, timeout_seconds): raise TimeoutError()
|
||||
with pytest.raises(ResearchError, match="temporarily unavailable"):
|
||||
ResearchService(Offline()).collect(request())
|
||||
with pytest.raises(ResearchError):
|
||||
ResearchService(FixtureProvider(result(request())), ResearchLimits(max_result_bytes=1)).collect(request())
|
||||
|
||||
|
||||
def test_collection_deterministic():
|
||||
service = ResearchService(FixtureProvider(result(request())))
|
||||
assert service.collect(request()) == service.collect(request())
|
||||
|
||||
|
||||
def test_package_roundtrip_and_restart(case_setup, tmp_path):
|
||||
store, builder, _, entries, _, index = case_setup
|
||||
active = store.advance("case-A","stable-package",State.BUILDING_CASE,step_id="prepare-research")
|
||||
with PackageReader(active) as reader:
|
||||
case = CaseAssembler().assemble(reader)
|
||||
# Research can also be explicitly supplied to corroborate a complete tester report.
|
||||
req = request()
|
||||
req["reports"] = [{"title":"Generic problem","summary":"Reported behavior"}]
|
||||
req["entities"] = [{"type":"component","id":"42"}]
|
||||
req["tester_expectations"] = ["The switch turns on"]
|
||||
req["project"] = reader.manifest["project"]
|
||||
artifacts = reader.manifest["artifacts"]
|
||||
ref, additions = ResearchService(FixtureProvider(result(req))).collect(req)
|
||||
index = dict(index, expected_research=[ref])
|
||||
additions["research/new-index.json"] = json_bytes(index)
|
||||
enriched = tmp_path / "enriched.zip"
|
||||
PackageWriter().write(active,enriched,additions=additions,manifest_updates={"artifacts": {**artifacts,"case_inputs":"research/new-index.json"}})
|
||||
store.intake(enriched)
|
||||
built = builder.build("case-A","stable-package")
|
||||
assert PackageValidator().validate(built.package).valid
|
||||
with PackageReader(built.package) as reader:
|
||||
canonical = json.loads(reader.read_bytes(built.case_ref))
|
||||
assert canonical["expected_behavior"]["provenance"] == "combined"
|
||||
assert canonical["research"]["performed"] is True
|
||||
for path, value in entries.items():
|
||||
assert reader.read_bytes(path) == (value.encode() if isinstance(value,str) else value)
|
||||
assert builder.build("case-A","stable-package").reused
|
||||
|
||||
|
||||
def test_service_enrichment_missing_expectations(manifest, make_package, tmp_path):
|
||||
from viano_bug_analysis.lifecycle import CheckpointStore, Workspace
|
||||
from viano_bug_analysis.case import CaseBuilder
|
||||
manifest["artifacts"]["case_inputs"] = "input.json"
|
||||
source = make_package(entries={"input.json": json_bytes({"format":"viano-case-inputs","format_version":1,"reports":["report.json"]}),
|
||||
"report.json": json_bytes({"format":"viano-bug-report","format_version":1,"title":"Switch remains off"}),
|
||||
"unknown.bin": b"preserve"})
|
||||
store = CheckpointStore(Workspace(tmp_path / "workspace"))
|
||||
store.intake(source)
|
||||
active = store.advance("case-A","stable-package",State.BUILDING_CASE,step_id="research-start")
|
||||
class DynamicProvider:
|
||||
calls = 0
|
||||
def lookup(self, request, *, timeout_seconds):
|
||||
self.calls += 1
|
||||
assert "source_context" not in request and "database_context" not in request
|
||||
return result(request)
|
||||
provider = DynamicProvider()
|
||||
service = ResearchService(provider)
|
||||
destination = tmp_path / "researched.zip"
|
||||
service.enrich(active, destination)
|
||||
assert provider.calls == 1
|
||||
assert PackageValidator().validate(destination).valid
|
||||
with PackageReader(active) as before, PackageReader(destination) as after:
|
||||
assert after.manifest["package_generation"] == before.manifest["package_generation"] + 1
|
||||
assert after.manifest["package_id"] == before.manifest["package_id"]
|
||||
for path in before.files:
|
||||
if path != "manifest.json": assert before.read_bytes(path) == after.read_bytes(path)
|
||||
store.intake(destination)
|
||||
built = CaseBuilder(store).build("case-A","stable-package")
|
||||
with PackageReader(built.package) as reader:
|
||||
case = json.loads(reader.read_bytes(built.case_ref))
|
||||
assert case["expected_behavior"]["completeness"] == "researched_complete"
|
||||
assert case["expected_behavior"]["provenance"] == "external_research"
|
||||
assert case["runtime_facts"]["facts"] == []
|
||||
with pytest.raises(ResearchError, match="Reopen"):
|
||||
service.enrich(built.package, tmp_path / "invalid.zip")
|
||||
assert provider.calls == 1
|
||||
|
||||
|
||||
@pytest.mark.parametrize("value", [0, -1, True, float("inf"), float("nan")])
|
||||
def test_invalid_timeout(value):
|
||||
with pytest.raises(ValueError): ResearchLimits(timeout_seconds=value)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("contract", ["research-request", "research-result"])
|
||||
def test_research_schemas(contract):
|
||||
from importlib.resources import files
|
||||
from jsonschema import Draft202012Validator
|
||||
schema = json.loads(files("viano_bug_analysis.schemas").joinpath(contract + "-v1.schema.json").read_text())
|
||||
Draft202012Validator.check_schema(schema)
|
||||
Reference in new issue
Block a user