Add authenticated case intake and completed-case return API
This commit is contained in:
1 parent
830dabe870
commit
27feda6d7f
14 files changed
+1098
-1
No files matched your search
@@ -10,6 +10,9 @@ Generic case construction and its input contracts are documented in
|
||||
[docs/case-builder-v1.md](docs/case-builder-v1.md).
|
||||
Vcore imports for wowdiag/wowsrc are documented in
|
||||
[docs/vcore-adapters-v1.md](docs/vcore-adapters-v1.md).
|
||||
Authenticated submission of new cases and completed-case feedback is documented in
|
||||
[docs/intake-api.md](docs/intake-api.md). The HTTP API creates and queues packages;
|
||||
external callers do not need to implement the internal package format.
|
||||
|
||||
Requires Python 3.11 or newer.
|
||||
|
||||
|
||||
@@ -0,0 +1,20 @@
|
||||
[api]
|
||||
workspace = "../service-data/service"
|
||||
host = "127.0.0.1"
|
||||
port = 8080
|
||||
|
||||
[[projects]]
|
||||
id = "viano-core"
|
||||
type = "wow"
|
||||
profile = "vcore"
|
||||
adapter = "vcore"
|
||||
# Trusted server-side analysis settings; never accepted from upload metadata.
|
||||
# [projects.settings]
|
||||
# project_config = "project.local.toml"
|
||||
# llm_config = "llm.early-review.local.toml"
|
||||
# investigation_config = "investigation.bounded.toml"
|
||||
|
||||
[[clients]]
|
||||
id = "wow-capture-worker"
|
||||
token_env = "VBA_INTAKE_TOKEN"
|
||||
projects = ["viano-core"]
|
||||
@@ -0,0 +1,168 @@
|
||||
# Übergabe-API v1
|
||||
|
||||
Die API erstellt interne Case-Pakete und veröffentlicht sie atomar in der Inbox.
|
||||
Externe Programme liefern Metadaten und Dateien; sie erzeugen und bearbeiten keine
|
||||
internen Manifeste. Die API führt keine Analyse und keine LLM-Aufrufe aus. Ein
|
||||
dauerhaft laufender Analyseworker und die allgemeine Weboberfläche sind folgende
|
||||
Ausbauschritte. Bis dahin übernimmt der vorhandene Inbox-Prozessor die Aufnahme.
|
||||
|
||||
## Lokaler Start
|
||||
|
||||
```powershell
|
||||
.\.venv\Scripts\python.exe -m pip install -e '.[api]'
|
||||
Copy-Item config/api.example.toml config/api.local.toml
|
||||
$env:VBA_INTAKE_TOKEN = [Convert]::ToHexString([System.Security.Cryptography.RandomNumberGenerator]::GetBytes(32))
|
||||
.\.venv\Scripts\python.exe -m viano_bug_analysis.api check --config config/api.local.toml
|
||||
.\.venv\Scripts\python.exe -m viano_bug_analysis.api serve --config config/api.local.toml
|
||||
```
|
||||
|
||||
Der API-Token ist ein eigener Zugangsschlüssel, kein Gemini-Key. Programme erhalten
|
||||
je einen Client und einen Token aus einer serverseitigen Umgebungsvariable. Der
|
||||
Token benötigt 32–256 Zeichen aus `A-Z a-z 0-9 _ -`. Auf älterem Windows PowerShell
|
||||
kann ein solcher Token über einen Passwortmanager erzeugt und als Variable gesetzt
|
||||
werden. Der Beispielstart bindet nur an `127.0.0.1:8080`. Für entfernte Nutzung sind
|
||||
HTTPS und ein passend konfigurierter Reverseproxy erforderlich. Tokens nicht in
|
||||
URLs oder Git-Konfiguration ablegen. Zugriff auf das Workspace-Dateisystem bleibt
|
||||
auf den Dienst und vertrauenswürdige Betreiber beschränkt.
|
||||
|
||||
`projects` ist die serverseitige Freigabeliste. Jeder Auftrag benennt `project_id`
|
||||
und `profile_id`. Er kann keine Repository-URLs, DB-Zugänge, Budgets oder
|
||||
Konfigurationspfade überschreiben. Optionale `projects.settings` halten die spätere
|
||||
Worker-Konfiguration bereit; die Eingangsschicht verwendet diese Dateien nicht für
|
||||
Analysen. Aktuell ist der Importadapter `vcore` für `wow/vcore` implementiert.
|
||||
Weitere Projekte benötigen passende Adapter/Profile, keine separat gepflegten Forks.
|
||||
|
||||
## Endpunkte
|
||||
|
||||
| Methode/Pfad | Bedeutung |
|
||||
|---|---|
|
||||
| `GET /healthz` | Minimale öffentliche Erreichbarkeitsprüfung |
|
||||
| `GET /v1/profiles` | Für diesen Client freigegebene Projekte/Profile |
|
||||
| `GET /v1/schema` | Öffentliches JSON-Schema der Übergabemetadaten |
|
||||
| `POST /v1/submissions` | Multipart-Upload, `201` bei neuer Einreihung, `200` bei Wiederholung |
|
||||
| `GET /v1/submissions/{id}` | Quittung und aktueller verwalteter Zustand |
|
||||
| `GET /v1/submissions/{id}/package` | Abgeschlossene Akte herunterladen; sonst `409` |
|
||||
|
||||
Alle Endpunkte außer Health benötigen `Authorization: Bearer TOKEN`.
|
||||
POST benötigt zusätzlich `Idempotency-Key: frei-gewaehlte-eindeutige-id` (1–128
|
||||
Zeichen, Buchstaben/Ziffern sowie Punkt, Unterstrich, Bindestrich; erstes Zeichen
|
||||
alphanumerisch). Derselbe Client/Schlüssel mit denselben Metadaten und Dateiinhalten
|
||||
liefert dieselbe Quittung, auch nach einem Neustart. Abweichender Inhalt ergibt
|
||||
`409 API_IDEMPOTENCY_CONFLICT`. Dateinamen und Multipart-Grenzen zählen nicht als
|
||||
Inhaltsänderung; Rollen und Reihenfolge der Dateien zählen. Ein zweiter neuer
|
||||
Auftrag für dasselbe Projekt und dieselbe Case-ID wird zurückgewiesen.
|
||||
|
||||
## Neuer Auftrag
|
||||
|
||||
`metadata` ist ein UTF-8-JSON-Textfeld, kein Datei-Upload. Dateien: genau einmal
|
||||
`wowdiag`, optional einmal `wowsrc`, optional mehrfach `attachments`. Für Vcore muss
|
||||
die Case-ID dem Bug des Erfassungsarchivs entsprechen (`BUG-000020`). Beide
|
||||
Erfassungsarchive müssen zueinander passen. Beispiel `submission.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"kind": "new_case",
|
||||
"project_id": "viano-core",
|
||||
"profile_id": "vcore",
|
||||
"case_id": "BUG-000020",
|
||||
"report": {
|
||||
"title": "Wolpertinger fehlen",
|
||||
"observations": ["Nach Questannahme waren im untersuchten Gebiet keine Tiere sichtbar."],
|
||||
"expected_behavior": ["Fangbare Tiere sind im Gebiet vorhanden."]
|
||||
},
|
||||
"versions": {
|
||||
"reported_server": "commit-des-gemeldeten-serverstands",
|
||||
"capture_server": "commit-des-erfassungsstands"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```powershell
|
||||
curl.exe --fail-with-body -H "Authorization: Bearer $env:VBA_INTAKE_TOKEN" -H "Idempotency-Key: bug20-capture-001" -F "metadata=<submission.json" -F "wowdiag=@wowdiag.tar.gz" -F "wowsrc=@wowsrc.tar.gz" http://127.0.0.1:8080/v1/submissions
|
||||
```
|
||||
|
||||
Versionsangaben werden als unbestätigte Angaben des Einlieferers gespeichert, nicht
|
||||
als verifizierte technische Fakten. Sie ersetzen keine Prüfung des Deployments.
|
||||
Zusätzliche Anhänge werden unter generierten internen Namen unverändert erhalten;
|
||||
sie werden nicht ausgeführt. Originalarchive bleiben ebenfalls erhalten.
|
||||
|
||||
## Rückläufer nach Abschluss
|
||||
|
||||
Dateien: genau einmal `case_package` (unveränderte abgeschlossene ZIP-Akte), optional
|
||||
mehrfach `attachments`. Kein erneutes Erzeugen interner JSON-Dateien nötig:
|
||||
|
||||
```json
|
||||
{
|
||||
"kind": "case_return",
|
||||
"project_id": "viano-core",
|
||||
"profile_id": "vcore",
|
||||
"case_id": "BUG-000020",
|
||||
"feedback": {
|
||||
"outcome": "patch_failed",
|
||||
"summary": "Der Patch wurde getestet, das Verhalten besteht weiter.",
|
||||
"observations": ["Die Tiere bleiben auch nach Neustart unsichtbar."],
|
||||
"patch_id": "pr-123",
|
||||
"tested_build": "commit-des-getesteten-builds"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```powershell
|
||||
curl.exe --fail-with-body -H "Authorization: Bearer $env:VBA_INTAKE_TOKEN" -H "Idempotency-Key: bug20-patch123-feedback" -F "metadata=<return.json" -F "case_package=@case.zip" http://127.0.0.1:8080/v1/submissions
|
||||
```
|
||||
|
||||
Zulässig sind `OUTPUT_READY`, `HANDED_OFF`, `FAILED_TO_ANALYZE`. Die API übernimmt
|
||||
bei `OUTPUT_READY` die interne Übergabemarkierung, ergänzt Feedback, Bericht und
|
||||
Anhänge und wechselt anschließend nach `REOPENED`. Die Paketidentität bleibt,
|
||||
`analysis_revision` steigt genau einmal. Bisherige Artefakte bleiben erhalten und
|
||||
der Casebuilder sieht die neuen Beobachtungen. Das neue Paket wird in die Inbox
|
||||
gelegt; `QUEUED` bedeutet ausschließlich angenommen, nicht erfolgreich analysiert.
|
||||
|
||||
Ein verwalteter Fall darf nur mit seiner exakten aktuellen Akte zurückgegeben
|
||||
werden. Veraltete/divergierende Akten und parallele Rückgaben werden abgewiesen.
|
||||
Ohne lokale Fallhistorie ist eine gültige abgeschlossene, passend zugeordnete Akte
|
||||
importierbar. Alte manuelle Testpakete ohne `project.id` benötigen vorerst eine
|
||||
betreiberseitige Migration; sie werden nicht stillschweigend einem Projekt zugeordnet.
|
||||
BLOCKED-Fälle, laufende Fälle und asynchrone Testerantworten innerhalb einer
|
||||
Untersuchung sind keine Abschlussrückläufer und werden hier nicht wiedereröffnet.
|
||||
|
||||
## Grenzen und Wiederanlauf
|
||||
|
||||
Standard: 768 MiB gesamter HTTP-Body, 512 MiB je Datei, 64 KiB Metadaten, 12 Dateien,
|
||||
120 Sekunden Uploadzeit und zwei gleichzeitige Uploads pro Prozess. Konfigurierbar
|
||||
über `[limits]` mit den Feldnamen aus `IntakeLimits`. Die existierenden Import- und
|
||||
ZIP-Validatoren setzen zusätzlich ihre eigenen Grenzen durch (Vcore-Import ist
|
||||
enger begrenzt). Dies sind Betriebsschutzwerte, keine fachlichen Analysebudgets.
|
||||
|
||||
Uploads werden vor dem Parsen authentifiziert. Auch Requests ohne Content-Length
|
||||
werden anhand tatsächlich empfangener Bytes begrenzt. Multipart-Spools werden bei
|
||||
Fehler/Timeout geschlossen, eigene temporäre Arbeitskopien wieder entfernt. Der
|
||||
Dienst prüft freien Speicher vor Aufnahme und Paketbau. Für den ersten Betrieb
|
||||
einen einzelnen Uvicorn-Prozess verwenden; die Upload-Parallelitätsgrenze gilt pro
|
||||
Prozess. Der Workspace verwendet zusätzlich OS-Dateisperren für Veröffentlichung.
|
||||
|
||||
Unter `runtime/intake/requests` bleiben kleine Quittungen und Hashes. Vor der
|
||||
atomaren Inbox-Veröffentlichung wird eine PREPARED-Quittung mit fertigem Paket
|
||||
gesichert. Beim Neustart werden solche Veröffentlichungen fortgesetzt. Nach
|
||||
Einreihung wird die zusätzliche Paketkopie entfernt. Bereits vom Worker übernommene
|
||||
Pakete werden über ihre Checkpoint-Kette erkannt und nicht erneut eingereiht.
|
||||
Nach hartem Prozessabbruch können verwaiste Arbeitsverzeichnisse unter
|
||||
`runtime/intake/staging` und Betriebssystem-Temporärdateien verbleiben; deren
|
||||
betriebliche Bereinigung/Retention gehört zum folgenden Worker-/Dienst-Ausbau.
|
||||
|
||||
Fehlerantworten enthalten stabile Codes, keine Dateipfade oder Secrets. Wesentliche
|
||||
Fälle: `401` Zugriff, `403` Projektfreigabe, `409` Konflikt/Zustand, `413` Größe,
|
||||
`422` ungültige Metadaten/Archive, `503` beschäftigt/Speicherfehler, `507` freier
|
||||
Speicher zu gering. Bei `503` mit demselben Idempotenzschlüssel wiederholen.
|
||||
|
||||
## Lokale Prüfung
|
||||
|
||||
```powershell
|
||||
.\.venv\Scripts\python.exe -m pip install -e '.[api,api-test]'
|
||||
.\.venv\Scripts\python.exe -m pytest tests/profiles/vcore/test_intake_api.py -q
|
||||
```
|
||||
|
||||
Die Tests verwenden künstliche Erfassungsarchive und HTTP-Testtransport. Sie prüfen
|
||||
auch Feedbackübernahme durch den echten Casebuilder, Idempotenz, veraltete
|
||||
Rückläufer, Uploadgrenzen sowie Abstürze vor und nach Queue-Veröffentlichung. Es
|
||||
finden keine kostenpflichtigen LLM-Aufrufe statt.
|
||||
@@ -12,6 +12,8 @@ dependencies = ["jsonschema>=4.18,<5"]
|
||||
[project.optional-dependencies]
|
||||
test = ["pytest>=8,<10"]
|
||||
research = ["PyMySQL>=1.1,<2", "zstandard>=0.23,<1"]
|
||||
api = ["starlette>=0.52.1,<0.53", "uvicorn>=0.35,<1", "python-multipart>=0.0.32,<1"]
|
||||
api-test = ["httpx>=0.28,<1", "pytest>=8,<10"]
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
where = ["src"]
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
"""Authenticated intake boundary; no model calls or analysis worker."""
|
||||
|
||||
from .config import APIConfig, Client, Project, IntakeLimits, APIError
|
||||
from .service import IntakeService
|
||||
|
||||
__all__ = ['APIConfig', 'Client', 'Project', 'IntakeLimits', 'APIError', 'IntakeService']
|
||||
@@ -0,0 +1,27 @@
|
||||
import argparse
|
||||
import json
|
||||
from .config import APIConfig, APIError
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description='Authenticated BugAnalysis intake; no analysis worker')
|
||||
parser.add_argument('command', choices=['check', 'serve'])
|
||||
parser.add_argument('--config', required=True)
|
||||
args = parser.parse_args()
|
||||
try:
|
||||
config = APIConfig.load(args.config)
|
||||
config.tokens()
|
||||
if args.command == 'check':
|
||||
print(json.dumps({'status': 'OK', 'projects': [p.id for p in config.projects]}))
|
||||
return 0
|
||||
import uvicorn
|
||||
from .app import create_app
|
||||
uvicorn.run(create_app(config), host=config.host, port=config.port, access_log=False)
|
||||
return 0
|
||||
except APIError as error:
|
||||
print(json.dumps({'status': 'ERROR', 'code': error.code}))
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,168 @@
|
||||
import asyncio
|
||||
import functools
|
||||
import hmac
|
||||
import shutil
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
from starlette.applications import Starlette
|
||||
from starlette.datastructures import UploadFile
|
||||
from starlette.exceptions import HTTPException
|
||||
from starlette.requests import Request
|
||||
from starlette.responses import FileResponse, JSONResponse
|
||||
from starlette.routing import Route
|
||||
from starlette.concurrency import run_in_threadpool
|
||||
from starlette.formparsers import MultiPartParser, MultiPartException
|
||||
|
||||
from .config import APIError
|
||||
from .contracts import schema
|
||||
from .service import IntakeService
|
||||
|
||||
|
||||
def create_app(config):
|
||||
service = IntakeService(config)
|
||||
tokens = config.tokens()
|
||||
service.recover()
|
||||
active = 0
|
||||
|
||||
def authenticate(request):
|
||||
header = request.headers.get('authorization', '')
|
||||
if not header.startswith('Bearer ') or not header.isascii():
|
||||
raise APIError('API_UNAUTHORIZED', 401)
|
||||
for client, token in tokens:
|
||||
if hmac.compare_digest(header[7:], token):
|
||||
return client
|
||||
raise APIError('API_UNAUTHORIZED', 401)
|
||||
|
||||
async def health(request):
|
||||
return JSONResponse({'status': 'OK'})
|
||||
|
||||
async def profiles(request):
|
||||
client = authenticate(request)
|
||||
return JSONResponse({'projects': [{'project_id': p.id, 'profile_id': p.profile,
|
||||
'adapter': p.adapter} for p in config.projects if p.id in client.projects]})
|
||||
|
||||
async def contract(request):
|
||||
authenticate(request)
|
||||
return JSONResponse(schema())
|
||||
|
||||
async def status(request):
|
||||
client = authenticate(request)
|
||||
return JSONResponse(await run_in_threadpool(service.status, request.path_params['submission'], client))
|
||||
|
||||
async def package(request):
|
||||
client = authenticate(request)
|
||||
path = await run_in_threadpool(service.package, request.path_params['submission'], client)
|
||||
return FileResponse(path, media_type='application/zip', filename='case.zip')
|
||||
|
||||
async def submit(request):
|
||||
nonlocal active
|
||||
client = authenticate(request) # never parse an unauthenticated upload
|
||||
if active >= config.limits.max_inflight_uploads:
|
||||
raise APIError('API_BUSY', 503)
|
||||
if request.headers.get('content-type', '').split(';')[0] != 'multipart/form-data':
|
||||
raise APIError('API_CONTENT_TYPE', 415)
|
||||
active += 1
|
||||
form = None
|
||||
try:
|
||||
if shutil.disk_usage(service.root).free < config.limits.min_free_bytes:
|
||||
raise APIError('API_STORAGE_LOW', 507)
|
||||
async with asyncio.timeout(config.limits.upload_timeout_seconds):
|
||||
parser = UploadParser(request.headers, request.stream(),
|
||||
max_files=config.limits.max_files, max_fields=1,
|
||||
max_part_size=config.limits.max_metadata_bytes,
|
||||
file_limit=config.limits.max_file_bytes)
|
||||
form = await parser.parse()
|
||||
raw = form.get('metadata')
|
||||
if not isinstance(raw, str):
|
||||
raise APIError('API_METADATA_INVALID', 422)
|
||||
with tempfile.TemporaryDirectory(dir=service.staging) as temporary:
|
||||
uploads = []
|
||||
for i, (role, value) in enumerate(form.multi_items()):
|
||||
if role == 'metadata':
|
||||
continue
|
||||
if not isinstance(value, UploadFile):
|
||||
raise APIError('API_UPLOAD_ROLES', 422)
|
||||
path = Path(temporary) / str(i)
|
||||
size = 0
|
||||
with path.open('wb') as output:
|
||||
while chunk := await value.read(65536):
|
||||
size += len(chunk)
|
||||
if size > config.limits.max_file_bytes:
|
||||
raise APIError('API_FILE_LIMIT', 413)
|
||||
output.write(chunk)
|
||||
uploads.append((role, path))
|
||||
result, replay = await run_in_threadpool(functools.partial(service.submit, client,
|
||||
request.headers.get('idempotency-key'), raw.encode('utf-8'), uploads))
|
||||
return JSONResponse(result, status_code=200 if replay else 201)
|
||||
except TimeoutError:
|
||||
raise APIError('API_UPLOAD_TIMEOUT', 408) from None
|
||||
except MultiPartException:
|
||||
raise APIError('API_MULTIPART_INVALID', 400) from None
|
||||
finally:
|
||||
if form is not None:
|
||||
await form.close()
|
||||
active -= 1
|
||||
|
||||
async def error(request, exc):
|
||||
return JSONResponse({'status': 'ERROR', 'code': exc.code}, status_code=exc.status)
|
||||
|
||||
async def http_error(request, exc):
|
||||
return JSONResponse({'status': 'ERROR', 'code': 'API_BAD_REQUEST'}, status_code=exc.status_code)
|
||||
|
||||
async def storage_error(request, exc):
|
||||
return JSONResponse({'status': 'ERROR', 'code': 'API_STORAGE_ERROR'}, status_code=503)
|
||||
|
||||
app = Starlette(routes=[Route('/healthz', health), Route('/v1/profiles', profiles),
|
||||
Route('/v1/schema', contract), Route('/v1/submissions', submit, methods=['POST']),
|
||||
Route('/v1/submissions/{submission}', status), Route('/v1/submissions/{submission}/package', package)],
|
||||
exception_handlers={APIError: error, HTTPException: http_error, OSError: storage_error})
|
||||
app.state.intake = service
|
||||
return BodyLimit(app, config.limits.max_body_bytes)
|
||||
|
||||
|
||||
class UploadParser(MultiPartParser):
|
||||
"""Bound file parts while parsing and close partial spools on cancellation."""
|
||||
def __init__(self, *args, file_limit, **kwargs):
|
||||
super().__init__(*args, **kwargs)
|
||||
self.file_limit = file_limit
|
||||
self.file_bytes = 0
|
||||
|
||||
def on_part_begin(self):
|
||||
super().on_part_begin()
|
||||
self.file_bytes = 0
|
||||
|
||||
def on_part_data(self, data, start, end):
|
||||
if self._current_part.file is not None:
|
||||
self.file_bytes += end - start
|
||||
if self.file_bytes > self.file_limit:
|
||||
raise APIError('API_FILE_LIMIT', 413)
|
||||
super().on_part_data(data, start, end)
|
||||
|
||||
async def parse(self):
|
||||
try:
|
||||
return await super().parse()
|
||||
except BaseException:
|
||||
for file in self._files_to_close_on_error:
|
||||
file.close()
|
||||
raise
|
||||
|
||||
|
||||
class BodyLimit:
|
||||
"""Count actual ASGI bytes, also for chunked requests without Content-Length."""
|
||||
def __init__(self, app, maximum):
|
||||
self.app, self.maximum = app, maximum
|
||||
self.state = app.state
|
||||
|
||||
async def __call__(self, scope, receive, send):
|
||||
if scope['type'] != 'http':
|
||||
return await self.app(scope, receive, send)
|
||||
total = 0
|
||||
async def bounded():
|
||||
nonlocal total
|
||||
message = await receive()
|
||||
total += len(message.get('body', b''))
|
||||
if total > self.maximum:
|
||||
raise APIError('API_BODY_LIMIT', 413)
|
||||
return message
|
||||
await self.app(scope, bounded, send)
|
||||
@@ -0,0 +1,132 @@
|
||||
import os
|
||||
import re
|
||||
import stat
|
||||
import tomllib
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
|
||||
IDENTIFIER = re.compile(r'[A-Za-z0-9][A-Za-z0-9_.-]{0,127}')
|
||||
SETTINGS = {'project_config', 'llm_config', 'investigation_config', 'budget_config',
|
||||
'analysis_config', 'diagnosis_config', 'change_request_config', 'correction_config'}
|
||||
|
||||
|
||||
class APIError(Exception):
|
||||
def __init__(self, code, status=400):
|
||||
self.code, self.status = code, status
|
||||
super().__init__(code)
|
||||
|
||||
|
||||
def directory(path):
|
||||
"""Service-owned paths must not pass through links/junctions."""
|
||||
path = Path(path).absolute()
|
||||
for part in [*reversed(path.parents), path]:
|
||||
if part.exists() or part.is_symlink():
|
||||
info = part.lstat()
|
||||
if stat.S_ISLNK(info.st_mode) or getattr(info, 'st_file_attributes', 0) & 0x400:
|
||||
raise APIError('API_UNSAFE_STORAGE', 500)
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
return path.resolve()
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class IntakeLimits:
|
||||
# Operational safety defaults, not permanent package-format constraints.
|
||||
max_body_bytes: int = 768 * 1024**2
|
||||
max_file_bytes: int = 512 * 1024**2
|
||||
max_metadata_bytes: int = 64 * 1024
|
||||
max_files: int = 12
|
||||
upload_timeout_seconds: int = 120
|
||||
max_inflight_uploads: int = 2
|
||||
min_free_bytes: int = 512 * 1024**2
|
||||
|
||||
def __post_init__(self):
|
||||
if any(type(v) is not int or v <= 0 for v in vars(self).values()):
|
||||
raise APIError('API_CONFIG', 500)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Project:
|
||||
id: str
|
||||
type: str
|
||||
profile: str
|
||||
adapter: str = 'vcore'
|
||||
settings: dict = field(default_factory=dict)
|
||||
|
||||
def __post_init__(self):
|
||||
if (any(not isinstance(v, str) or not IDENTIFIER.fullmatch(v)
|
||||
for v in (self.id, self.type, self.profile))
|
||||
or self.adapter != 'vcore' or (self.type, self.profile) != ('wow', 'vcore')
|
||||
or set(self.settings) - SETTINGS):
|
||||
raise APIError('API_PROFILE_CONFIG', 500)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Client:
|
||||
id: str
|
||||
token_env: str
|
||||
projects: tuple[str, ...]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class APIConfig:
|
||||
workspace: Path
|
||||
projects: tuple[Project, ...]
|
||||
clients: tuple[Client, ...]
|
||||
host: str = '127.0.0.1'
|
||||
port: int = 8080
|
||||
limits: IntakeLimits = field(default_factory=IntakeLimits)
|
||||
|
||||
def __post_init__(self):
|
||||
ids = {p.id for p in self.projects}
|
||||
if (not ids or len(ids) != len(self.projects) or not self.clients
|
||||
or len({c.id for c in self.clients}) != len(self.clients)
|
||||
or type(self.port) is not int or not 0 < self.port < 65536
|
||||
or not isinstance(self.host, str) or not self.host):
|
||||
raise APIError('API_CONFIG', 500)
|
||||
for c in self.clients:
|
||||
if (not isinstance(c.id, str) or not IDENTIFIER.fullmatch(c.id)
|
||||
or not isinstance(c.token_env, str)
|
||||
or not re.fullmatch(r'[A-Za-z_][A-Za-z0-9_]*', c.token_env)
|
||||
or not c.projects or not set(c.projects) <= ids):
|
||||
raise APIError('API_CONFIG', 500)
|
||||
|
||||
def tokens(self):
|
||||
result = []
|
||||
for client in self.clients:
|
||||
token = os.environ.get(client.token_env, '')
|
||||
if not re.fullmatch(r'[A-Za-z0-9_-]{32,256}', token):
|
||||
raise APIError('API_CREDENTIALS', 500)
|
||||
if any(token == existing for _, existing in result):
|
||||
raise APIError('API_CREDENTIALS', 500)
|
||||
result.append((client, token))
|
||||
return result
|
||||
|
||||
@classmethod
|
||||
def load(cls, path):
|
||||
try:
|
||||
path = Path(path).resolve()
|
||||
data = tomllib.loads(path.read_text(encoding='utf-8'))
|
||||
if set(data) - {'api', 'projects', 'clients', 'limits'}:
|
||||
raise ValueError()
|
||||
api = data['api']
|
||||
if set(api) - {'workspace', 'host', 'port'}:
|
||||
raise ValueError()
|
||||
projects = []
|
||||
for row in data['projects']:
|
||||
row = dict(row)
|
||||
settings = row.pop('settings', {})
|
||||
if set(settings) - SETTINGS:
|
||||
raise ValueError()
|
||||
resolved = {}
|
||||
for key, value in settings.items():
|
||||
target = (path.parent / value).resolve()
|
||||
if not target.is_file():
|
||||
raise ValueError()
|
||||
resolved[key] = target
|
||||
projects.append(Project(**row, settings=resolved))
|
||||
clients = [Client(**(dict(c) | {'projects': tuple(c['projects'])})) for c in data['clients']]
|
||||
return cls(workspace=(path.parent / api['workspace']).resolve(),
|
||||
projects=tuple(projects), clients=tuple(clients), host=api.get('host', '127.0.0.1'),
|
||||
port=api.get('port', 8080), limits=IntakeLimits(**data.get('limits', {})))
|
||||
except (OSError, ValueError, TypeError, KeyError):
|
||||
raise APIError('API_CONFIG', 500) from None
|
||||
@@ -0,0 +1,23 @@
|
||||
"""Public metadata contract; clients never describe internal ZIP members."""
|
||||
from importlib.resources import files
|
||||
import json
|
||||
|
||||
from jsonschema import Draft202012Validator
|
||||
from viano_bug_analysis.package.manifest import parse_manifest
|
||||
from .config import APIError
|
||||
|
||||
|
||||
def schema():
|
||||
return json.loads(files('viano_bug_analysis.schemas').joinpath('api-submission-v1.schema.json').read_text(encoding='utf-8'))
|
||||
|
||||
|
||||
def metadata(raw, limit):
|
||||
try:
|
||||
if len(raw) > limit:
|
||||
raise APIError('API_METADATA_LIMIT', 413)
|
||||
result = parse_manifest(raw)
|
||||
if not Draft202012Validator(schema()).is_valid(result):
|
||||
raise APIError('API_METADATA_INVALID', 422)
|
||||
return result
|
||||
except (ValueError, TypeError, UnicodeError, RecursionError):
|
||||
raise APIError('API_METADATA_INVALID', 422) from None
|
||||
@@ -0,0 +1,105 @@
|
||||
"""Trusted conversion of public submissions into append-only case artifacts."""
|
||||
import copy
|
||||
from pathlib import Path
|
||||
|
||||
from viano_bug_analysis.lifecycle.checkpoints import CheckpointStore
|
||||
from viano_bug_analysis.lifecycle.job import timestamp
|
||||
from viano_bug_analysis.lifecycle.storage import Workspace, json_bytes
|
||||
from viano_bug_analysis.package import PackageReader
|
||||
from viano_bug_analysis.package.factory import PackageFactory
|
||||
from viano_bug_analysis.profiles.vcore.importer import VcoreImporter
|
||||
from viano_bug_analysis.case.contracts import validate
|
||||
from .config import APIError
|
||||
|
||||
TERMINAL = {'OUTPUT_READY', 'HANDED_OFF', 'FAILED_TO_ANALYZE'}
|
||||
|
||||
|
||||
def documents(data, client, submission, uploads, index):
|
||||
prefix = f'intake/submissions/{submission}'
|
||||
request_ref = prefix + '/request.json'
|
||||
additions = {request_ref: json_bytes({'format': 'viano-api-submission', 'format_version': 1,
|
||||
'received_at': timestamp(), 'actor': client.id, 'metadata': data})}
|
||||
report = {'format': 'viano-bug-report', 'format_version': 1}
|
||||
supplied = data.get('report', {})
|
||||
if 'title' in supplied:
|
||||
report['title'] = supplied['title']
|
||||
if 'description' in supplied:
|
||||
report['summary'] = supplied['description']
|
||||
for source, target in [('observations', 'observations'), ('expected_behavior', 'expected_behavior'),
|
||||
('tester_assumptions', 'tester_hypotheses'), ('reproduction_steps', 'comments')]:
|
||||
report[target] = [{'text': text, 'author': client.id,
|
||||
'origin': {'source_ref': request_ref, 'json_pointer': f'/metadata/report/{source}/{i}'}}
|
||||
for i, text in enumerate(supplied.get(source, []))]
|
||||
if feedback := data.get('feedback'):
|
||||
report['comments'].append({'text': f"{feedback['outcome']}: {feedback['summary']}", 'author': client.id})
|
||||
for field in ('patch_id', 'tested_build'):
|
||||
if field in feedback:
|
||||
report['comments'].append({'text': f"Client-supplied {field}: {feedback[field]}", 'author': client.id})
|
||||
report['observations'].extend({'text': text, 'author': client.id,
|
||||
'origin': {'source_ref': request_ref, 'json_pointer': f'/metadata/feedback/observations/{i}'}}
|
||||
for i, text in enumerate(feedback['observations']))
|
||||
if data.get('versions'):
|
||||
report['comments'].append({'text': 'Client-supplied, unverified versions: ' +
|
||||
json_bytes(data['versions']).decode(), 'author': client.id})
|
||||
report_ref = prefix + '/report.json'
|
||||
validate(report, 'report')
|
||||
additions[report_ref] = json_bytes(report)
|
||||
index = copy.deepcopy(index)
|
||||
index.setdefault('reports', []).append(report_ref)
|
||||
index.setdefault('attachments', []).append(request_ref)
|
||||
for i, (role, path) in enumerate(uploads):
|
||||
if role == 'attachments':
|
||||
ref = prefix + f'/attachments/{i:04d}.bin'
|
||||
additions[ref] = Path(path).read_bytes()
|
||||
index['attachments'].append(ref)
|
||||
index_ref = prefix + '/case-inputs.json'
|
||||
validate(index, 'case-inputs')
|
||||
additions[index_ref] = json_bytes(index)
|
||||
return additions, {'case_inputs': index_ref, 'intake_request': request_ref}
|
||||
|
||||
|
||||
def build_new(destination, data, client, project, submission, package_id, uploads):
|
||||
roles = dict(uploads)
|
||||
plan = VcoreImporter().prepare(roles['wowdiag'], roles.get('wowsrc'))
|
||||
# Do not associate a collector archive with a different externally named bug.
|
||||
if data['case_id'] != f'BUG-{plan.bug_id:06d}':
|
||||
raise APIError('API_CASE_ID_MISMATCH', 422)
|
||||
import json
|
||||
additions, pointers = documents(data, client, submission, uploads,
|
||||
json.loads(plan.additions[plan.index_ref]))
|
||||
now = timestamp()
|
||||
manifest = {'format': 'viano-bug-case-package', 'format_version': 1, 'package_id': package_id,
|
||||
'package_generation': 0, 'case_id': data['case_id'],
|
||||
'project': {'id': project.id, 'type': project.type, 'profile': project.profile},
|
||||
'created_at': now, 'updated_at': now, 'state': 'NEW', 'analysis_revision': 0,
|
||||
'routing': {'destination': 'inbox', 'reason': None, 'retry_count': 0, 'retry_not_before': None},
|
||||
'artifacts': {'case': None, 'diagnosis': None, 'change_request': None,
|
||||
'vcore_import': plan.receipt_ref, **pointers},
|
||||
'history': {'analysis_revisions': 0, 'reproductions': plan.reproductions_count,
|
||||
'evidence': 0, 'handoffs': 0, 'feedback': 0, 'events_ref': None}, 'integrity': {}}
|
||||
PackageFactory().create(destination, manifest, plan.additions | additions)
|
||||
|
||||
|
||||
def build_return(destination, scratch, data, client, project, submission, uploads):
|
||||
source = dict(uploads)['case_package']
|
||||
with PackageReader(source) as reader:
|
||||
original = reader.manifest
|
||||
if (original['case_id'] != data['case_id'] or original['project'] !=
|
||||
{'id': project.id, 'type': project.type, 'profile': project.profile}):
|
||||
raise APIError('API_CASE_IDENTITY', 422)
|
||||
if original['state'] not in TERMINAL:
|
||||
raise APIError('API_CASE_NOT_COMPLETED', 409)
|
||||
import json
|
||||
index = json.loads(reader.read_bytes(original['artifacts']['case_inputs']))
|
||||
validate(index, 'case-inputs')
|
||||
additions, pointers = documents(data, client, submission, uploads, index)
|
||||
store = CheckpointStore(Workspace(scratch))
|
||||
store.intake(source)
|
||||
case_id, package_id = original['case_id'], original['package_id']
|
||||
if original['state'] == 'OUTPUT_READY':
|
||||
store.advance(case_id, package_id, 'HANDED_OFF', step_id=f'api:{submission}:handoff')
|
||||
result = store.advance(case_id, package_id, 'REOPENED', step_id=f'api:{submission}:reopen',
|
||||
details={'operation': 'api-case-return', 'submission_id': submission}, additions=additions,
|
||||
artifacts=pointers, feedback_increment=1)
|
||||
import shutil
|
||||
shutil.copyfile(result, destination)
|
||||
@@ -0,0 +1,193 @@
|
||||
"""Durable, idempotent inbox publication. This boundary never invokes an LLM."""
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
from viano_bug_analysis.lifecycle.checkpoints import CheckpointStore, checkpoint_chain, fingerprint
|
||||
from viano_bug_analysis.lifecycle.errors import LifecycleError, LockBusy
|
||||
from viano_bug_analysis.lifecycle.locking import FileLock
|
||||
from viano_bug_analysis.lifecycle.storage import Workspace, atomic_json, json_bytes, key
|
||||
from viano_bug_analysis.package import PackageReader, PackageReadError, PackageWriteError
|
||||
from viano_bug_analysis.profiles.vcore.errors import AdapterError
|
||||
from viano_bug_analysis.case.errors import CaseBuildError
|
||||
from .config import APIError, IDENTIFIER, directory
|
||||
from .contracts import metadata
|
||||
from .packages import TERMINAL, build_new, build_return
|
||||
|
||||
|
||||
def digest(path):
|
||||
with open(path, 'rb') as stream:
|
||||
return hashlib.file_digest(stream, 'sha256').hexdigest()
|
||||
|
||||
|
||||
class IntakeService:
|
||||
def __init__(self, config):
|
||||
self.config = config
|
||||
directory(config.workspace)
|
||||
self.workspace = Workspace(config.workspace)
|
||||
for name in ('inbox', 'cases', 'runtime', 'runtime/locks', 'runtime/intake'):
|
||||
directory(self.workspace.root / name)
|
||||
self.root = self.workspace.runtime / 'intake'
|
||||
self.requests = directory(self.root / 'requests')
|
||||
self.staging = directory(self.root / 'staging')
|
||||
self.store = CheckpointStore(self.workspace)
|
||||
|
||||
def project(self, client, data):
|
||||
if data['project_id'] not in client.projects:
|
||||
raise APIError('API_PROJECT_FORBIDDEN', 403)
|
||||
project = next(p for p in self.config.projects if p.id == data['project_id'])
|
||||
if project.profile != data['profile_id']:
|
||||
raise APIError('API_PROFILE_MISMATCH', 422)
|
||||
return project
|
||||
|
||||
def _receipt(self, submission, client):
|
||||
if not IDENTIFIER.fullmatch(submission):
|
||||
raise APIError('API_NOT_FOUND', 404)
|
||||
path = self.requests / submission / 'receipt.json'
|
||||
if not path.is_file():
|
||||
raise APIError('API_NOT_FOUND', 404)
|
||||
receipt = json.loads(path.read_bytes())
|
||||
if receipt['project_id'] not in client.projects:
|
||||
raise APIError('API_NOT_FOUND', 404)
|
||||
return receipt
|
||||
|
||||
def status(self, submission, client):
|
||||
receipt = self._receipt(submission, client)
|
||||
status = receipt['status']
|
||||
latest = self.store.latest(receipt['case_id'], receipt['package_id'])
|
||||
if self._known(receipt, latest):
|
||||
with PackageReader(latest) as reader:
|
||||
status = reader.manifest['state']
|
||||
return {k: receipt[k] for k in ('submission_id', 'project_id', 'profile_id', 'case_id',
|
||||
'package_id', 'analysis_revision')} | {'status': status}
|
||||
|
||||
def package(self, submission, client):
|
||||
receipt = self._receipt(submission, client)
|
||||
latest = self.store.latest(receipt['case_id'], receipt['package_id'])
|
||||
if self._known(receipt, latest):
|
||||
with PackageReader(latest) as reader:
|
||||
if reader.manifest['state'] in TERMINAL:
|
||||
return latest
|
||||
raise APIError('API_RESULT_NOT_READY', 409)
|
||||
|
||||
def _known(self, receipt, head):
|
||||
if not head:
|
||||
return False
|
||||
with PackageReader(head) as reader:
|
||||
return (fingerprint(reader) == receipt['output_fingerprint'] or
|
||||
any(c['source_fingerprint'] == receipt['output_fingerprint']
|
||||
for _, c in checkpoint_chain(reader)))
|
||||
|
||||
def _publish(self, receipt):
|
||||
folder = self.requests / receipt['submission_id']
|
||||
inbox = self.workspace.inbox / (receipt['submission_id'] + '.zip')
|
||||
with self.store._locks(receipt['case_id']):
|
||||
head = self.store.latest(receipt['case_id'], receipt['package_id'])
|
||||
if inbox.exists():
|
||||
if digest(inbox) != receipt['output_sha256']:
|
||||
raise APIError('API_QUEUE_CONFLICT', 409)
|
||||
elif not self._known(receipt, head):
|
||||
actual = None
|
||||
if head:
|
||||
with PackageReader(head) as reader:
|
||||
actual = fingerprint(reader)
|
||||
if actual != receipt['base_fingerprint']:
|
||||
raise APIError('API_STALE_CASE', 409)
|
||||
prepared = folder / 'package.zip'
|
||||
if not prepared.is_file() or digest(prepared) != receipt['output_sha256']:
|
||||
raise APIError('API_PREPARED_PACKAGE_MISSING', 500)
|
||||
# Hard-link publication is atomic and cannot replace a queue entry.
|
||||
os.link(prepared, inbox)
|
||||
receipt['status'] = 'QUEUED'
|
||||
atomic_json(folder / 'receipt.json', receipt)
|
||||
(folder / 'package.zip').unlink(missing_ok=True)
|
||||
|
||||
def recover(self):
|
||||
"""Resume only committed PREPARED submissions after an interrupted publish."""
|
||||
with FileLock(self.workspace.runtime / 'locks/api-intake.lock'):
|
||||
for path in self.requests.glob('*/receipt.json'):
|
||||
receipt = json.loads(path.read_bytes())
|
||||
if receipt['status'] == 'PREPARED':
|
||||
self._publish(receipt)
|
||||
|
||||
def submit(self, client, idempotency_key, raw_metadata, uploads):
|
||||
if not isinstance(idempotency_key, str) or not IDENTIFIER.fullmatch(idempotency_key):
|
||||
raise APIError('API_IDEMPOTENCY_KEY', 400)
|
||||
data = metadata(raw_metadata, self.config.limits.max_metadata_bytes)
|
||||
project = self.project(client, data)
|
||||
roles = [r for r, _ in uploads]
|
||||
required = 'wowdiag' if data['kind'] == 'new_case' else 'case_package'
|
||||
allowed = {'wowdiag', 'wowsrc', 'attachments'} if data['kind'] == 'new_case' else {'case_package', 'attachments'}
|
||||
if (required not in roles or set(roles) - allowed or len(roles) > self.config.limits.max_files
|
||||
or any(roles.count(r) > 1 for r in allowed - {'attachments'})):
|
||||
raise APIError('API_UPLOAD_ROLES', 422)
|
||||
if any(Path(p).stat().st_size > self.config.limits.max_file_bytes for _, p in uploads):
|
||||
raise APIError('API_FILE_LIMIT', 413)
|
||||
submission = key(client.id + ':' + idempotency_key)
|
||||
request_hash = key(json_bytes({'metadata': data, 'files': [
|
||||
[r, Path(p).stat().st_size, digest(p)] for r, p in uploads]}).decode())
|
||||
try:
|
||||
with FileLock(self.workspace.runtime / 'locks/api-intake.lock'):
|
||||
folder = directory(self.requests / submission)
|
||||
receipt_path = folder / 'receipt.json'
|
||||
if receipt_path.exists():
|
||||
receipt = json.loads(receipt_path.read_bytes())
|
||||
if receipt['request_hash'] != request_hash:
|
||||
raise APIError('API_IDEMPOTENCY_CONFLICT', 409)
|
||||
if receipt['status'] == 'PREPARED':
|
||||
self._publish(receipt)
|
||||
return self.status(submission, client), True
|
||||
package_id = 'api-' + key(project.id + ':' + data['case_id'])
|
||||
base = None
|
||||
if data['kind'] == 'case_return':
|
||||
with PackageReader(dict(uploads)['case_package']) as reader:
|
||||
package_id = reader.manifest['package_id']
|
||||
incoming = fingerprint(reader)
|
||||
head = self.store.latest(data['case_id'], package_id)
|
||||
if head:
|
||||
with PackageReader(head) as reader:
|
||||
base = fingerprint(reader)
|
||||
if base != incoming:
|
||||
raise APIError('API_STALE_CASE', 409)
|
||||
elif self.store.latest(data['case_id'], package_id):
|
||||
raise APIError('API_CASE_EXISTS', 409)
|
||||
# An unconsumed submission, including a crash-prepared one, owns this identity.
|
||||
for path in self.requests.glob('*/receipt.json'):
|
||||
old = json.loads(path.read_bytes())
|
||||
if old['case_id'] == data['case_id'] and old['project_id'] == project.id:
|
||||
if data['kind'] == 'new_case':
|
||||
raise APIError('API_CASE_EXISTS', 409)
|
||||
if old['status'] == 'PREPARED' or (self.workspace.inbox / (old['submission_id'] + '.zip')).exists():
|
||||
raise APIError('API_CASE_PENDING', 409)
|
||||
reserve = sum(Path(p).stat().st_size for _, p in uploads) * 5
|
||||
if shutil.disk_usage(self.root).free < self.config.limits.min_free_bytes + reserve:
|
||||
raise APIError('API_STORAGE_LOW', 507)
|
||||
with tempfile.TemporaryDirectory(dir=self.staging) as temporary:
|
||||
destination = Path(temporary) / 'package.zip'
|
||||
if data['kind'] == 'new_case':
|
||||
build_new(destination, data, client, project, submission, package_id, uploads)
|
||||
else:
|
||||
build_return(destination, Path(temporary) / 'scratch', data, client, project, submission, uploads)
|
||||
with PackageReader(destination) as reader:
|
||||
output_fingerprint = fingerprint(reader)
|
||||
revision = reader.manifest['analysis_revision']
|
||||
# Unreferenced package from an interrupted pre-receipt build can be replaced.
|
||||
os.replace(destination, folder / 'package.zip')
|
||||
with open(folder / 'package.zip', 'r+b') as stream:
|
||||
os.fsync(stream.fileno())
|
||||
receipt = {'submission_id': submission, 'status': 'PREPARED', 'client_id': client.id,
|
||||
'project_id': project.id, 'profile_id': project.profile, 'case_id': data['case_id'],
|
||||
'package_id': package_id, 'analysis_revision': revision, 'request_hash': request_hash,
|
||||
'base_fingerprint': base, 'output_fingerprint': output_fingerprint,
|
||||
'output_sha256': digest(folder / 'package.zip')}
|
||||
atomic_json(receipt_path, receipt)
|
||||
self._publish(receipt)
|
||||
return self.status(submission, client), False
|
||||
except LockBusy:
|
||||
raise APIError('API_BUSY', 503) from None
|
||||
except (PackageReadError, PackageWriteError, AdapterError, LifecycleError,
|
||||
CaseBuildError, ValueError, TypeError, KeyError, RecursionError):
|
||||
raise APIError('API_PACKAGE_INVALID', 422) from None
|
||||
@@ -283,7 +283,8 @@ class CheckpointStore:
|
||||
reason: str | None = None, retry_not_before: str | None = None,
|
||||
details: dict | None = None, additions: dict[str, bytes] | None = None,
|
||||
artifacts: dict[str, str | None] | None = None,
|
||||
expected_generation: int | None = None, manual_retry: bool = False) -> Path:
|
||||
expected_generation: int | None = None, manual_retry: bool = False,
|
||||
feedback_increment: int = 0) -> Path:
|
||||
try:
|
||||
if not isinstance(step_id, str) or not step_id.strip():
|
||||
raise LifecycleError("INVALID_STEP", "A nonblank stable step_id is required")
|
||||
@@ -295,11 +296,15 @@ class CheckpointStore:
|
||||
raise LifecycleError("INVALID_GENERATION", "expected_generation must be a nonnegative integer")
|
||||
if type(manual_retry) is not bool:
|
||||
raise LifecycleError("INVALID_RETRY", "manual_retry must be boolean")
|
||||
if type(feedback_increment) is not int or feedback_increment not in (0, 1) or (feedback_increment and target != State.REOPENED):
|
||||
raise LifecycleError("INVALID_FEEDBACK", "Feedback may only be appended when reopening")
|
||||
if manual_retry:
|
||||
details = {**(details or {}), "manual_retry": True}
|
||||
payload = {"target": target, "reason": reason, "retry_not_before": retry_not_before,
|
||||
"details": details or {}, "artifacts": artifacts or {},
|
||||
"additions": {name: hashlib.sha256(data).hexdigest() for name, data in (additions or {}).items()}}
|
||||
if feedback_increment:
|
||||
payload['feedback_increment'] = feedback_increment
|
||||
request_hash = hashlib.sha256(json_bytes(payload)).hexdigest()
|
||||
with self._locks(case_id):
|
||||
latest = self.latest(case_id, package_id)
|
||||
@@ -362,6 +367,8 @@ class CheckpointStore:
|
||||
if target == State.REOPENED or (manifest["state"] == State.VALIDATED and target == State.BUILDING_CASE and revision == 0):
|
||||
revision += 1
|
||||
history = {"analysis_revisions": max(int(manifest["history"]["analysis_revisions"]), revision)}
|
||||
if feedback_increment:
|
||||
history['feedback'] = int(manifest['history']['feedback']) + feedback_increment
|
||||
return self._commit(reader, step_id, request_hash, target, details=details, additions=additions,
|
||||
resume_state=resume, updates={"routing": routing, "analysis_revision": revision,
|
||||
"history": history, "artifacts": artifacts or {}})
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"type": "object", "additionalProperties": false,
|
||||
"required": ["kind", "project_id", "profile_id", "case_id"],
|
||||
"properties": {
|
||||
"kind": {"enum": ["new_case", "case_return"]},
|
||||
"project_id": {"$ref": "#/$defs/id"}, "profile_id": {"$ref": "#/$defs/id"},
|
||||
"case_id": {"$ref": "#/$defs/id"},
|
||||
"report": {"type": "object", "additionalProperties": false, "properties": {
|
||||
"title": {"$ref": "#/$defs/text"}, "description": {"$ref": "#/$defs/text"},
|
||||
"observations": {"$ref": "#/$defs/texts"}, "expected_behavior": {"$ref": "#/$defs/texts"},
|
||||
"tester_assumptions": {"$ref": "#/$defs/texts"}, "reproduction_steps": {"$ref": "#/$defs/texts"}
|
||||
}},
|
||||
"versions": {"type": "object", "additionalProperties": false, "properties": {
|
||||
"reported_server": {"$ref": "#/$defs/text"}, "capture_server": {"$ref": "#/$defs/text"},
|
||||
"tested_build": {"$ref": "#/$defs/text"}
|
||||
}},
|
||||
"feedback": {"type": "object", "additionalProperties": false, "required": ["outcome", "summary", "observations"], "properties": {
|
||||
"outcome": {"enum": ["patch_failed", "validation_failed", "review_feedback", "new_evidence"]},
|
||||
"summary": {"$ref": "#/$defs/text"}, "observations": {"$ref": "#/$defs/texts"},
|
||||
"patch_id": {"$ref": "#/$defs/text"}, "tested_build": {"$ref": "#/$defs/text"}
|
||||
}}
|
||||
},
|
||||
"allOf": [{"if": {"properties": {"kind": {"const": "case_return"}}},
|
||||
"then": {"required": ["feedback"]}, "else": {"not": {"required": ["feedback"]}}}],
|
||||
"$defs": {
|
||||
"id": {"type": "string", "pattern": "^[A-Za-z0-9][A-Za-z0-9_.-]{0,127}$"},
|
||||
"text": {"type": "string", "minLength": 1, "maxLength": 12000, "pattern": "\\S"},
|
||||
"texts": {"type": "array", "minItems": 1, "maxItems": 100, "items": {"$ref": "#/$defs/text"}}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,212 @@
|
||||
import json
|
||||
from dataclasses import replace
|
||||
|
||||
import pytest
|
||||
pytest.importorskip('starlette')
|
||||
pytest.importorskip('httpx')
|
||||
pytest.importorskip('multipart')
|
||||
from starlette.testclient import TestClient
|
||||
|
||||
from viano_bug_analysis.api import APIConfig, Client, Project, IntakeLimits
|
||||
from viano_bug_analysis.api.app import create_app
|
||||
from viano_bug_analysis.api.config import APIError
|
||||
from viano_bug_analysis.lifecycle.checkpoints import CheckpointStore
|
||||
from viano_bug_analysis.lifecycle.storage import Workspace
|
||||
from viano_bug_analysis.package import PackageReader
|
||||
|
||||
TOKEN = 'test_token_' + 'x' * 40
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def api(tmp_path, monkeypatch):
|
||||
monkeypatch.setenv('TEST_INTAKE_TOKEN', TOKEN)
|
||||
config = APIConfig(tmp_path / 'service', (Project('core', 'wow', 'vcore'),),
|
||||
(Client('capture', 'TEST_INTAKE_TOKEN', ('core',)),), limits=IntakeLimits(min_free_bytes=1))
|
||||
app = create_app(config)
|
||||
return TestClient(app), app.state.intake, config
|
||||
|
||||
|
||||
def data(kind='new_case'):
|
||||
result = {'kind': kind, 'project_id': 'core', 'profile_id': 'vcore', 'case_id': 'BUG-000020',
|
||||
'report': {'observations': ['The observed bug persists.']}}
|
||||
if kind == 'case_return':
|
||||
result['feedback'] = {'outcome': 'patch_failed', 'summary': 'Still broken',
|
||||
'observations': ['The tested patch did not fix it.']}
|
||||
return result
|
||||
|
||||
|
||||
def post(client, source, *, metadata=None, key='first', role='wowdiag', token=TOKEN):
|
||||
return client.post('/v1/submissions', headers={'Authorization': 'Bearer ' + token,
|
||||
'Idempotency-Key': key}, data={'metadata': json.dumps(metadata or data())},
|
||||
files={role: ('../untrusted.bin', source.read_bytes(), 'application/octet-stream')})
|
||||
|
||||
|
||||
def test_create_replay_conflict_and_private_status(api, source_pair):
|
||||
client, service, _ = api
|
||||
source = source_pair[0]
|
||||
response = post(client, source)
|
||||
assert response.status_code == 201, response.text
|
||||
receipt = response.json()
|
||||
assert receipt['status'] == 'QUEUED'
|
||||
assert post(client, source).status_code == 200
|
||||
assert len(list(service.workspace.inbox.glob('*.zip'))) == 1
|
||||
changed = data() | {'report': {'title': 'Changed'}}
|
||||
assert post(client, source, metadata=changed).status_code == 409
|
||||
assert post(client, source, key='different').status_code == 409
|
||||
assert post(client, source, token='bad').status_code == 401
|
||||
assert client.get('/v1/submissions/' + receipt['submission_id']).status_code == 401
|
||||
with PackageReader(next(service.workspace.inbox.glob('*.zip'))) as reader:
|
||||
assert reader.manifest['project']['id'] == 'core'
|
||||
assert reader.manifest['state'] == 'NEW'
|
||||
assert b'The observed bug persists.' in reader.read_bytes(
|
||||
'intake/submissions/' + receipt['submission_id'] + '/report.json')
|
||||
|
||||
|
||||
def test_return_preserves_history_and_increments_revision_once(api, source_pair, tmp_path):
|
||||
client, service, _ = api
|
||||
first = post(client, source_pair[0]).json()
|
||||
inbox = next(service.workspace.inbox.glob('*.zip'))
|
||||
# Standalone terminal case simulates transfer back from a different deployment.
|
||||
from viano_bug_analysis.package import PackageWriter
|
||||
terminal = tmp_path / 'terminal.zip'
|
||||
PackageWriter().write(inbox, terminal, manifest_updates={'state': 'OUTPUT_READY',
|
||||
'analysis_revision': 1, 'history': {'analysis_revisions': 1},
|
||||
'routing': {'destination': 'outbox'}})
|
||||
inbox.unlink()
|
||||
response = post(client, terminal, metadata=data('case_return'), key='return', role='case_package')
|
||||
assert response.status_code == 201, response.text
|
||||
returned = next(service.workspace.inbox.glob('*.zip'))
|
||||
with PackageReader(terminal) as old, PackageReader(returned) as new:
|
||||
assert new.manifest['state'] == 'REOPENED'
|
||||
assert new.manifest['analysis_revision'] == 2
|
||||
assert new.manifest['history']['feedback'] == 1
|
||||
assert new.manifest['package_id'] == first['package_id']
|
||||
for ref in old.files:
|
||||
if ref != 'manifest.json':
|
||||
assert old.read_bytes(ref) == new.read_bytes(ref)
|
||||
index = json.loads(new.read_bytes(new.manifest['artifacts']['case_inputs']))
|
||||
assert any(b'The tested patch did not fix it.' in new.read_bytes(ref) for ref in index['reports'])
|
||||
assert post(client, terminal, metadata=data('case_return'), key='return', role='case_package').status_code == 200
|
||||
assert post(client, terminal, metadata=data('case_return'), key='return2', role='case_package').status_code == 409
|
||||
from viano_bug_analysis.case import CaseBuilder
|
||||
isolated = CheckpointStore(Workspace(tmp_path / 'analysis'))
|
||||
isolated.intake(returned)
|
||||
built = CaseBuilder(isolated).build('BUG-000020', first['package_id'])
|
||||
with PackageReader(built.package) as reader:
|
||||
case = json.loads(reader.read_bytes(built.case_ref))
|
||||
assert any(o['text'] == 'The tested patch did not fix it.'
|
||||
for o in case['human_observations']['observations'])
|
||||
|
||||
|
||||
def test_active_case_cannot_be_returned(api, source_pair):
|
||||
client, service, _ = api
|
||||
post(client, source_pair[0])
|
||||
source = next(service.workspace.inbox.glob('*.zip'))
|
||||
assert post(client, source, metadata=data('case_return'), key='return', role='case_package').status_code == 409
|
||||
|
||||
|
||||
def test_limits_and_contract(api, source_pair):
|
||||
client, _, config = api
|
||||
assert client.get('/healthz').json() == {'status': 'OK'}
|
||||
assert client.get('/v1/schema', headers={'Authorization': 'Bearer ' + TOKEN}).status_code == 200
|
||||
assert post(client, source_pair[0], metadata=data() | {'budget': 99}).status_code == 422
|
||||
assert post(client, source_pair[0], metadata=data() | {'project_id': 'elsewhere'}).status_code == 403
|
||||
tiny = replace(config, workspace=config.workspace.parent / 'tiny',
|
||||
limits=replace(config.limits, max_body_bytes=100))
|
||||
limited = TestClient(create_app(tiny))
|
||||
assert post(limited, source_pair[0]).status_code == 413
|
||||
|
||||
|
||||
def test_recover_prepared_after_publication_interruption(api, source_pair, monkeypatch):
|
||||
client, service, _ = api
|
||||
publish = service._publish
|
||||
def interrupted(receipt):
|
||||
raise OSError('simulated crash')
|
||||
monkeypatch.setattr(service, '_publish', interrupted)
|
||||
with pytest.raises(OSError):
|
||||
service.submit(service.config.clients[0], 'crash', json.dumps(data()).encode(), [('wowdiag', source_pair[0])])
|
||||
monkeypatch.setattr(service, '_publish', publish)
|
||||
service.recover()
|
||||
assert len(list(service.workspace.inbox.glob('*.zip'))) == 1
|
||||
result, replay = service.submit(service.config.clients[0], 'crash', json.dumps(data()).encode(), [('wowdiag', source_pair[0])])
|
||||
assert replay and result['status'] == 'QUEUED'
|
||||
|
||||
|
||||
def test_published_receipt_crash_replay_after_worker_consumes(api, source_pair, monkeypatch):
|
||||
import viano_bug_analysis.api.service as module
|
||||
client, service, _ = api
|
||||
original = module.atomic_json
|
||||
def interrupted(path, value):
|
||||
if value.get('status') == 'QUEUED':
|
||||
raise OSError('crash after publication')
|
||||
original(path, value)
|
||||
monkeypatch.setattr(module, 'atomic_json', interrupted)
|
||||
with pytest.raises(OSError):
|
||||
service.submit(service.config.clients[0], 'crash2', json.dumps(data()).encode(), [('wowdiag', source_pair[0])])
|
||||
inbox = next(service.workspace.inbox.glob('*.zip'))
|
||||
service.store.intake(inbox)
|
||||
inbox.unlink()
|
||||
monkeypatch.setattr(module, 'atomic_json', original)
|
||||
service.recover()
|
||||
assert not list(service.workspace.inbox.glob('*.zip'))
|
||||
result, replay = service.submit(service.config.clients[0], 'crash2', json.dumps(data()).encode(), [('wowdiag', source_pair[0])])
|
||||
assert replay and result['status'] == 'VALIDATED'
|
||||
|
||||
|
||||
def test_file_limit_and_partial_upload_cleanup(api, source_pair):
|
||||
_, _, config = api
|
||||
tiny = replace(config, workspace=config.workspace.parent / 'files',
|
||||
limits=replace(config.limits, max_file_bytes=5))
|
||||
app = create_app(tiny)
|
||||
response = post(TestClient(app), source_pair[0])
|
||||
assert response.status_code == 413
|
||||
assert not list(app.state.intake.staging.iterdir())
|
||||
|
||||
|
||||
def test_return_with_stale_managed_head_is_rejected(api, source_pair, tmp_path):
|
||||
client, service, _ = api
|
||||
first = post(client, source_pair[0]).json()
|
||||
inbox = next(service.workspace.inbox.glob('*.zip'))
|
||||
from viano_bug_analysis.package import PackageWriter
|
||||
terminal = tmp_path / 'terminal.zip'
|
||||
PackageWriter().write(inbox, terminal, manifest_updates={'state': 'OUTPUT_READY',
|
||||
'analysis_revision': 1, 'history': {'analysis_revisions': 1}, 'routing': {'destination': 'outbox'}})
|
||||
service.store.intake(terminal)
|
||||
inbox.unlink()
|
||||
response = post(client, terminal, metadata=data('case_return'), key='stale', role='case_package')
|
||||
assert response.status_code == 409
|
||||
head = service.store.latest('BUG-000020', first['package_id'])
|
||||
response = post(client, head, metadata=data('case_return'), key='fresh', role='case_package')
|
||||
assert response.status_code == 201, response.text
|
||||
assert response.json()['status'] == 'QUEUED' # not the previous OUTPUT_READY result
|
||||
auth = {'Authorization': 'Bearer ' + TOKEN}
|
||||
assert client.get('/v1/submissions/' + response.json()['submission_id'] + '/package', headers=auth).status_code == 409
|
||||
|
||||
|
||||
def test_scope_invalid_json_and_mismatched_source(api, source_pair):
|
||||
client, service, _ = api
|
||||
response = post(client, source_pair[0], metadata=data() | {'case_id': 'BUG-000021'})
|
||||
assert response.status_code == 422
|
||||
assert not list(service.workspace.inbox.glob('*.zip'))
|
||||
raw = client.post('/v1/submissions', headers={'Authorization': 'Bearer ' + TOKEN,
|
||||
'Idempotency-Key': 'duplicate-json'}, data={'metadata': '{"kind":"new_case","kind":"case_return"}'},
|
||||
files={'wowdiag': ('source', source_pair[0].read_bytes())})
|
||||
assert raw.status_code == 422
|
||||
first = post(client, source_pair[0]).json()
|
||||
stranger = Client('stranger', 'OTHER', ('other-project',))
|
||||
with pytest.raises(APIError) as exc:
|
||||
service.status(first['submission_id'], stranger)
|
||||
assert exc.value.status == 404
|
||||
|
||||
|
||||
def test_pair_and_attachments_are_retained(api, source_pair):
|
||||
client, service, _ = api
|
||||
response = client.post('/v1/submissions', headers={'Authorization': 'Bearer ' + TOKEN,
|
||||
'Idempotency-Key': 'paired'}, data={'metadata': json.dumps(data())}, files=[
|
||||
('wowdiag', ('diag', source_pair[0].read_bytes())),
|
||||
('wowsrc', ('src', source_pair[1].read_bytes())),
|
||||
('attachments', ('a', b'opaque-human-notes'))])
|
||||
assert response.status_code == 201, response.text
|
||||
with PackageReader(next(service.workspace.inbox.glob('*.zip'))) as reader:
|
||||
index = json.loads(reader.read_bytes(reader.manifest['artifacts']['case_inputs']))
|
||||
assert any(reader.read_bytes(ref) == b'opaque-human-notes' for ref in index['attachments'])
|
||||
Reference in new issue
Block a user