Prepare asynchronous test handoffs with budget-preserving resume
This commit is contained in:
1 parent
af18f36c24
commit
798925d25d
11 files changed
+655
-2
No files matched your search
@@ -0,0 +1,106 @@
|
||||
# Asynchronous test handoff v1
|
||||
|
||||
This addition prepares the existing project for externally performed human and
|
||||
automated tests. No web UI, HTTP feedback endpoint, notifications, bot steering,
|
||||
client automation, Tool Router or Investigation Loop is implemented here.
|
||||
`docs/architecture.md` remains unchanged and authoritative: VianoBugAnalysis
|
||||
does not steer labs. It requests observations through a future external adapter.
|
||||
|
||||
## Durable contract
|
||||
|
||||
`AsyncTestCoordinator` is imported from `viano_bug_analysis.lifecycle.async_tests`.
|
||||
It is a trusted controller service, not a callable operation exposed directly to
|
||||
an LLM. The future router must validate requests before calling it. The future
|
||||
web adapter must authenticate/authorize users and confirm feedback before calling
|
||||
`confirm_feedback`; a payload claiming `confirmed = true` is not authentication.
|
||||
User-supplied observations are untrusted data, not executable instructions.
|
||||
|
||||
Two bundled JSON schemas define the case-local exchange:
|
||||
|
||||
- `async-test-request-v1`: stable request/action/hypothesis IDs, case/package IDs,
|
||||
revision, executor (`human` or `automated`), question, prerequisites, steps,
|
||||
requested observations, environment, deadline and reserved budget ledger ref.
|
||||
- `async-test-feedback-v1`: matching identity/executor, observed environment,
|
||||
tester ID, outcome (`completed` or `unable_to_test`), observations, attachment
|
||||
references/hashes, and confirming actor/time. Unsupported operations, unknown
|
||||
fields and unconfirmed feedback are rejected.
|
||||
|
||||
The controller reserves budget first, using `testing.request_human` or
|
||||
`testing.request_automated`, then calls `submit(request)`. These actions require
|
||||
explicit cost overrides in the frozen budget policy. The example budget config
|
||||
sets human tests to 40 units and automated tests to 20. These are tunable initial
|
||||
values; they do not start any test execution. Existing revisions need their
|
||||
existing policy and cannot silently adopt a new cost table.
|
||||
|
||||
One outstanding case-level test is allowed at present. A reservation and its
|
||||
submission are separate durable steps. A failed submission does not silently
|
||||
refund the reservation: the controller can retry the same valid request or
|
||||
explicitly fail an undispatched reservation under existing budget rules. Future
|
||||
delivery/claim/cancellation policies will need explicit contracts; none are
|
||||
inferred from a deadline or idle time.
|
||||
|
||||
## Waiting and confirmation
|
||||
|
||||
```text
|
||||
INVESTIGATING_HYPOTHESES + matching reserved action
|
||||
-> request stored + BLOCKED/AWAITING_TEST_FEEDBACK
|
||||
-> worker continues with other cases
|
||||
-> authenticated external adapter confirms matching feedback
|
||||
-> feedback stored + INVESTIGATING_HYPOTHESES (same analysis_revision)
|
||||
-> future controller evaluates result and settles the reserved action
|
||||
```
|
||||
|
||||
Submission stores an immutable request under
|
||||
`analysis/async-tests/NNNN/<hashed-request-id>/request.json`, sets the manifest
|
||||
pointer `async_test_request`, and publishes a wait checkpoint. The checkpoint
|
||||
has `automatic_retry = false`. The deadline is recorded as `retry_not_before`
|
||||
for scheduling visibility, but it does not authorize an automatic resume.
|
||||
The inbox worker explicitly skips `AWAITING_TEST_FEEDBACK`; ordinary lifecycle
|
||||
`advance` cannot resume such a case. Deadline expiry neither consumes budget
|
||||
nor creates a failure, new analysis revision, refund or replacement request.
|
||||
Deadline escalation/notifications are future adapter/controller policy.
|
||||
|
||||
`confirm_feedback(feedback)` verifies the original handoff, matching identities,
|
||||
pending reservation, unchanged ledger, schemas and attachment hashes. It stores
|
||||
`feedback.json`, sets `async_test_feedback`, and resumes the investigation even
|
||||
before `retry_not_before`. It deliberately uses a narrowly controlled event-driven
|
||||
resume path rather than disabling retry protection for other blocked reasons.
|
||||
|
||||
While waiting, remaining balance, expenses, rewards, low-information counters
|
||||
and pending reservations remain unchanged. Confirmation also leaves the budget
|
||||
ledger unchanged. Only subsequent `record_result`/`settle` applies novelty,
|
||||
semantic scoring and possible reward. Waiting hours never cost extra credits;
|
||||
confirmation never restores start budgets. Unavailable tests can return a
|
||||
confirmed `unable_to_test` observation and be assessed as a low-information
|
||||
result. Unrelated post-diagnosis review or patch rejection still uses the
|
||||
existing `REOPENED` path and creates a new revision.
|
||||
|
||||
Both APIs are idempotent. The same document reuses the current package;
|
||||
conflicting content under the same request ID is rejected. Unknown, wrong-case,
|
||||
wrong-revision, unmatched and unconfirmed feedback cannot wake the case. Events
|
||||
increase only `package_generation` and preserve all previous payloads. A case
|
||||
waiting for external feedback can be admitted on another service before its
|
||||
deadline and still stays blocked until confirmed feedback arrives.
|
||||
|
||||
## Future web/bot adapter responsibilities
|
||||
|
||||
The UI will list persisted requests, allow authorized testers to claim them,
|
||||
capture observations and upload attachments, and call the confirmation API.
|
||||
External adapters need reliable/idempotent delivery, authentication, authorization,
|
||||
attachment size/type policy and notification/escalation handling. Delivery state
|
||||
must not replace the case-local original request and feedback history. Draft
|
||||
feedback stays outside the authoritative package until confirmed; no console
|
||||
entry or SSH workflow is required.
|
||||
|
||||
Attachments must already exist in the package when confirmation is called, with
|
||||
matching hashes. The future upload importer will append them using the existing
|
||||
safe writer, preferably under the same request's `attachments/` directory.
|
||||
Unknown files are preserved. Checkpoint-declared request/feedback artifacts and
|
||||
their async-test attachment references are excluded from raw CaseBuilder input
|
||||
identity, so expected investigation feedback does not force case regeneration.
|
||||
Existing original-source attachments retain their original input classification.
|
||||
|
||||
The coordinator's configurable `max_json_bytes` (default 1 MiB) is an operational
|
||||
safeguard, not a permanent limit of either v1 format. Case locking, generation
|
||||
comparison and atomic package publication use the existing lifecycle foundation.
|
||||
No lock is held while a human, bot, network provider or model is working.
|
||||
Reference in new issue
Block a user