Add persistent lifecycle checkpoints and resumable inbox processing
This commit is contained in:
Nico committed
1 parent
f832b038a5
commit
e985e1d5f0
15 files changed
+1401
-1
No files matched your search
@@ -0,0 +1,166 @@
|
||||
# Lifecycle foundation v1
|
||||
|
||||
`architecture.md` remains the authoritative design. This milestone provides only
|
||||
deterministic job state, checkpoint storage, locking and inbox admission. It does
|
||||
not execute CaseBuilder, interpretation, hypotheses, diagnosis, tools or LLMs.
|
||||
State APIs allow future phase implementations to confirm their work; recording
|
||||
a state is not evidence that this foundation executed that phase.
|
||||
|
||||
## Package state and service state
|
||||
|
||||
The package manifest is authoritative for identity, generation, revision, state
|
||||
and routing. Detailed progress is stored as immutable JSON artifacts at
|
||||
`history/checkpoints/<hash-of-step-id>.json`. The manifest's
|
||||
`artifacts.lifecycle_checkpoint` points to the latest artifact. Each checkpoint
|
||||
links to its predecessor, with format/version, identities, output generation,
|
||||
analysis revision, previous/current state, routing snapshot, optional resume state, step identity,
|
||||
request hash, source generation and content fingerprint, creation time and opaque
|
||||
structured `details`. `checkpoint-v1.schema.json` defines the JSON contract.
|
||||
Readers additionally check paths, cycles, state values, generation/revision order,
|
||||
identity and agreement with the manifest for a matching generation. Checkpoint
|
||||
JSON reads use the configurable manifest-byte limit as their security cap.
|
||||
|
||||
Existing `analysis/current.json` is not repurposed as worker state. The checkpoint
|
||||
namespace and manifest checkpoint pointer are reserved to the lifecycle service.
|
||||
Checkpoints are outside frozen analysis revision directories. Every successful
|
||||
checkpoint uses PackageWriter to create and validate one new package generation;
|
||||
all old artifacts remain. No checkpoint exists only in an external database.
|
||||
|
||||
Service layout:
|
||||
|
||||
```text
|
||||
<workspace>/
|
||||
inbox/ incoming files of arbitrary names/extensions
|
||||
quarantine/<unique-id>/ package.zip and validation_error.json
|
||||
cache/<raw-archive-sha256>.zip exact immutable input archives
|
||||
cases/<case-hash>/<package-hash>/
|
||||
g00000001.zip immutable confirmed generations
|
||||
current.json reconstructible local lookup index
|
||||
outbox/ reserved for later analyzed-package delivery
|
||||
review/ reserved for later human-review delivery
|
||||
runtime/
|
||||
current_job.json coarse worker summary, or null when idle
|
||||
locks/worker.lock persistent lock inode/file
|
||||
locks/case-<case-hash>.lock
|
||||
```
|
||||
|
||||
Opaque IDs are hashed before use in filesystem paths, including for locking.
|
||||
Stored-generation filenames are internal lookup conventions and are always
|
||||
checked against the manifest; incoming filenames never supply identity or state.
|
||||
Runtime includes identities, input generation, analysis revision, target stage,
|
||||
start time and last confirmed checkpoint time. It contains no detailed progress.
|
||||
Runtime and local index JSON are written via temporary file, flush/fsync and
|
||||
atomic replacement. They can be stale after a crash and are never authoritative.
|
||||
Current generation is recovered from fully validated managed packages. Corrupt
|
||||
managed storage is reported rather than silently rolling back to an older file.
|
||||
|
||||
## Transitions and retry
|
||||
|
||||
Normal transitions follow the architecture's linear sequence from NEW through
|
||||
HANDED_OFF. REOPENED proceeds to BUILDING_CASE. Active phases may checkpoint
|
||||
progress in the same state, become BLOCKED, or become FAILED_TO_ANALYZE.
|
||||
HANDED_OFF and FAILED_TO_ANALYZE can become REOPENED. Skips, regressions and
|
||||
unrelated BLOCKED resumes are rejected.
|
||||
|
||||
The initial VALIDATED to BUILDING_CASE transition starts revision 1 if it is
|
||||
still zero. REOPENED increments analysis revision exactly once; subsequently
|
||||
entering BUILDING_CASE preserves it. The revision summary counter is maintained
|
||||
without altering other history counters. Package generation changes independently.
|
||||
These are explicit controller operations, not inferred from filenames.
|
||||
|
||||
BLOCKED requires a nonblank reason and timezone-aware retry timestamp. The
|
||||
checkpoint records the interrupted active state; routing becomes inbox with an
|
||||
incremented retry count. Resume returns only to that state, only after the retry
|
||||
time, and clears the reason/timestamp while retaining retry count. The retry
|
||||
clock is UTC-aware. FAILED_TO_ANALYZE routes logically to review; OUTPUT_READY
|
||||
and HANDED_OFF route logically to outbox; active/reopened work routes to cases.
|
||||
The state and routing fields remain separate, and intake preserves a returning
|
||||
package's routing. Only fresh NEW packages are advanced to VALIDATED on admission.
|
||||
|
||||
This milestone does not create diagnoses, failure reports or Change Requests and
|
||||
does not deliver packages to coding agents. Outbox/review are initialized but
|
||||
physical output delivery belongs to the future phase controller. Detailed
|
||||
checkpoint payloads and artifact pointer updates are caller-supplied contracts;
|
||||
they receive structural validation, not diagnostic interpretation.
|
||||
|
||||
## Idempotence, locking and recovery
|
||||
|
||||
The content fingerprint hashes sorted internal paths and uncompressed bytes,
|
||||
with canonical JSON for the manifest. Archive compression, order and filename
|
||||
do not affect identity. Re-delivery of a known input generation and fingerprint
|
||||
does not create a new generation. Each checkpoint carries its source receipt,
|
||||
so this remains true when the package moves to a service without old local files.
|
||||
A seen generation with different content is quarantined as a conflict. Newer
|
||||
returning packages must preserve current protected files, checkpoint lineage and
|
||||
analysis revision; they cannot replace prior content or erase history. The input
|
||||
manifest remains authoritative for externally performed work on newer generations.
|
||||
|
||||
`advance` requires a stable nonblank `step_id`. Repeating it with the same target,
|
||||
details, artifact content and retry request returns the latest confirmed package
|
||||
without rerunning a write. Reusing the ID for a different request is an error.
|
||||
Artifact additions and the checkpoint are committed in the same package. External
|
||||
side effects are outside this transaction and need their own idempotence policy.
|
||||
|
||||
Nonblocking native OS locks use msvcrt on Windows and flock on POSIX. The coarse
|
||||
runtime file models one worker, guarded by a worker lock. Case locks additionally
|
||||
use case identity, independent of package identity. There is no timeout-based lock
|
||||
stealing and no waiting on blocked cases. The OS releases locks when a process
|
||||
exits, including a crash. Lock files remain in place to avoid inode replacement
|
||||
races; do not delete them while workers are active. Shared network filesystems
|
||||
require a separate distributed locking strategy and are not supported here.
|
||||
|
||||
Commit sequence:
|
||||
|
||||
1. Validate input, checkpoint lineage and requested transition under locks.
|
||||
2. Atomically write the coarse runtime summary.
|
||||
3. Publish the fully validated next generation via PackageWriter.
|
||||
4. Update the reconstructible local index, then clear runtime to null.
|
||||
5. For inbox admission, acknowledge the original only after publication, checking
|
||||
that its raw bytes still match the cached snapshot.
|
||||
|
||||
Before step 3, retry repeats only the unconfirmed operation. After step 3, the
|
||||
published package and embedded receipt prove completion even if steps 4 or 5
|
||||
never happened. A retry repairs local metadata and does not increment generation.
|
||||
File fsync and atomic visibility are provided, without claiming portable
|
||||
directory-metadata durability through power loss. Crashes may leave unpublished
|
||||
temporary files or incomplete quarantine directories; those are not treated as
|
||||
confirmed generations. Garbage collection is outside this milestone.
|
||||
|
||||
## Public API
|
||||
|
||||
```python
|
||||
from viano_bug_analysis.lifecycle import (
|
||||
Workspace, InboxProcessor, CheckpointStore, State, LifecycleError,
|
||||
)
|
||||
|
||||
workspace = Workspace("service-data")
|
||||
processor = InboxProcessor(workspace)
|
||||
results = processor.poll(max_jobs=100)
|
||||
|
||||
store = CheckpointStore(workspace)
|
||||
try:
|
||||
package = store.advance(
|
||||
"case-A", "stable-package", State.BUILDING_CASE,
|
||||
step_id="revision-1:start-case-building",
|
||||
details={"inputs_collected": []},
|
||||
)
|
||||
except LifecycleError as error:
|
||||
print(error.code, str(error))
|
||||
```
|
||||
|
||||
`poll` is a single pass, not a daemon. It validates bounded cached snapshots,
|
||||
admits valid cases, preserves invalid originals in quarantine with a structured
|
||||
error report, and continues to the next input after per-input errors. Oversized
|
||||
archives are moved directly to quarantine without creating an oversized cache
|
||||
copy. Due BLOCKED local cases are resumed once; future retries remain deferred.
|
||||
Deferred or busy inputs do not consume the `max_jobs` admission limit, so they
|
||||
cannot indefinitely crowd fresh work out of a poll. Such a pass may return more
|
||||
than `max_jobs` results and inspect all pending files. No diagnostic work runs.
|
||||
|
||||
Results are ACCEPTED, DUPLICATE, QUARANTINED, DEFERRED, BUSY, RESUMED or ERROR,
|
||||
with paths and optional error code/message. Conflicting/corrupt incoming records
|
||||
are quarantined; infrastructure failures leave inbox input pending. A changed
|
||||
producer file is not acknowledged. Workspace storage must be service-owned on a
|
||||
local filesystem supporting hard links, and producers must publish complete
|
||||
files atomically without modifying them during processing. Symlink inputs are
|
||||
rejected. This is not a hardened multi-tenant filesystem boundary.
|
||||
Reference in new issue
Block a user