108 lines
6.2 KiB
Markdown
108 lines
6.2 KiB
Markdown
# Asynchronous test handoff v1
|
|
|
|
This contract supports externally performed human and automated tests. The human
|
|
HTTP adapter and dedicated web page are now documented in
|
|
[Web console](web-console.md#menschliche-testanfragen). Automated execution,
|
|
notifications, bot steering and client automation are not provided by that page.
|
|
`docs/architecture.md` remains unchanged and authoritative: VianoBugAnalysis
|
|
does not steer labs. It requests observations through a future external adapter.
|
|
|
|
## Durable contract
|
|
|
|
`AsyncTestCoordinator` is imported from `viano_bug_analysis.lifecycle.async_tests`.
|
|
It is a trusted controller service, not a callable operation exposed directly to
|
|
an LLM. The investigation controller validates requests before calling it. The
|
|
web adapter authenticates/authorizes users and confirms feedback before calling
|
|
`confirm_feedback`; a payload claiming `confirmed = true` is not authentication.
|
|
User-supplied observations are untrusted data, not executable instructions.
|
|
|
|
Two bundled JSON schemas define the case-local exchange:
|
|
|
|
- `async-test-request-v1`: stable request/action/hypothesis IDs, case/package IDs,
|
|
revision, executor (`human` or `automated`), question, prerequisites, steps,
|
|
requested observations, environment, deadline and reserved budget ledger ref.
|
|
- `async-test-feedback-v1`: matching identity/executor, observed environment,
|
|
tester ID, outcome (`completed` or `unable_to_test`), observations, attachment
|
|
references/hashes, and confirming actor/time. Unsupported operations, unknown
|
|
fields and unconfirmed feedback are rejected.
|
|
|
|
The controller reserves budget first, using `testing.request_human` or
|
|
`testing.request_automated`, then calls `submit(request)`. These actions require
|
|
explicit cost overrides in the frozen budget policy. The example budget config
|
|
sets human tests to 40 units and automated tests to 20. These are tunable initial
|
|
values; they do not start any test execution. Existing revisions need their
|
|
existing policy and cannot silently adopt a new cost table.
|
|
|
|
One outstanding case-level test is allowed at present. A reservation and its
|
|
submission are separate durable steps. A failed submission does not silently
|
|
refund the reservation: the controller can retry the same valid request or
|
|
explicitly fail an undispatched reservation under existing budget rules. Future
|
|
delivery/claim/cancellation policies will need explicit contracts; none are
|
|
inferred from a deadline or idle time.
|
|
|
|
## Waiting and confirmation
|
|
|
|
```text
|
|
INVESTIGATING_HYPOTHESES + matching reserved action
|
|
-> request stored + BLOCKED/AWAITING_TEST_FEEDBACK
|
|
-> worker continues with other cases
|
|
-> authenticated external adapter confirms matching feedback
|
|
-> feedback stored + INVESTIGATING_HYPOTHESES (same analysis_revision)
|
|
-> future controller evaluates result and settles the reserved action
|
|
```
|
|
|
|
Submission stores an immutable request under
|
|
`analysis/async-tests/NNNN/<hashed-request-id>/request.json`, sets the manifest
|
|
pointer `async_test_request`, and publishes a wait checkpoint. The checkpoint
|
|
has `automatic_retry = false`. The deadline is recorded as `retry_not_before`
|
|
for scheduling visibility, but it does not authorize an automatic resume.
|
|
The inbox worker explicitly skips `AWAITING_TEST_FEEDBACK`; ordinary lifecycle
|
|
`advance` cannot resume such a case. Deadline expiry neither consumes budget
|
|
nor creates a failure, new analysis revision, refund or replacement request.
|
|
Deadline escalation/notifications are future adapter/controller policy.
|
|
|
|
`confirm_feedback(feedback)` verifies the original handoff, matching identities,
|
|
pending reservation, unchanged ledger, schemas and attachment hashes. It stores
|
|
`feedback.json`, sets `async_test_feedback`, and resumes the investigation even
|
|
before `retry_not_before`. It deliberately uses a narrowly controlled event-driven
|
|
resume path rather than disabling retry protection for other blocked reasons.
|
|
|
|
While waiting, remaining balance, expenses, rewards, low-information counters
|
|
and pending reservations remain unchanged. Confirmation also leaves the budget
|
|
ledger unchanged. Only subsequent `record_result`/`settle` applies novelty,
|
|
semantic scoring and possible reward. Waiting hours never cost extra credits;
|
|
confirmation never restores start budgets. Unavailable tests can return a
|
|
confirmed `unable_to_test` observation and be assessed as a low-information
|
|
result. Unrelated post-diagnosis review or patch rejection still uses the
|
|
existing `REOPENED` path and creates a new revision.
|
|
|
|
Both APIs are idempotent. The same document reuses the current package;
|
|
conflicting content under the same request ID is rejected. Unknown, wrong-case,
|
|
wrong-revision, unmatched and unconfirmed feedback cannot wake the case. Events
|
|
increase only `package_generation` and preserve all previous payloads. A case
|
|
waiting for external feedback can be admitted on another service before its
|
|
deadline and still stays blocked until confirmed feedback arrives.
|
|
|
|
## Future web/bot adapter responsibilities
|
|
|
|
The UI will list persisted requests, allow authorized testers to claim them,
|
|
capture observations and upload attachments, and call the confirmation API.
|
|
External adapters need reliable/idempotent delivery, authentication, authorization,
|
|
attachment size/type policy and notification/escalation handling. Delivery state
|
|
must not replace the case-local original request and feedback history. Draft
|
|
feedback stays outside the authoritative package until confirmed; no console
|
|
entry or SSH workflow is required.
|
|
|
|
Attachments must already exist in the package when confirmation is called, with
|
|
matching hashes. The future upload importer will append them using the existing
|
|
safe writer, preferably under the same request's `attachments/` directory.
|
|
Unknown files are preserved. Checkpoint-declared request/feedback artifacts and
|
|
their async-test attachment references are excluded from raw CaseBuilder input
|
|
identity, so expected investigation feedback does not force case regeneration.
|
|
Existing original-source attachments retain their original input classification.
|
|
|
|
The coordinator's configurable `max_json_bytes` (default 1 MiB) is an operational
|
|
safeguard, not a permanent limit of either v1 format. Case locking, generation
|
|
comparison and atomic package publication use the existing lifecycle foundation.
|
|
No lock is held while a human, bot, network provider or model is working.
|