Add operational telemetry with safe diagnostics and passive monitoring

docs/telemetry-v1.md

@@ -0,0 +1,162 @@
+# Operational telemetry v1
+
+Telemetry observes existing library execution, not case truth. It never resumes
+BLOCKED cases, refills budgets, changes routing, retries a model, edits packages,
+executes tools or deletes case artifacts. `docs/architecture.md` is unchanged.
+A continuous analysis worker/scheduler remains the next integration milestone.
+The observation heartbeat is not a substitute for that worker or evidence that
+an analysis is progressing.
+
+## Enable observations
+
+```python
+from viano_bug_analysis.lifecycle import Workspace, CheckpointStore
+from viano_bug_analysis.telemetry import Telemetry, TelemetryConfig
+from viano_bug_analysis.analysis import CaseInterpreter
+
+workspace = Workspace("work")
+config = TelemetryConfig.load("config/telemetry.local.toml")
+with Telemetry(workspace, config) as telemetry:
+    store = CheckpointStore(workspace, telemetry=telemetry)
+    # In an async driver, use existing APIs normally:
+    # await CaseInterpreter(store, gateway).interpret(case_id, package_id)
+```
+
+Telemetry is explicitly opt-in. Without an attached running session, existing API
+behavior is unchanged. InboxProcessor also accepts telemetry=telemetry. Stores
+can be attached after construction with store.telemetry=telemetry. Use one
+observation session per workspace; a nonblocking OS lease prevents competing
+log rotations/status writers. Concurrent threads and async tasks in that process
+have independent, correlated operation spans. There is no global worker lock
+across observation or model calls. Stop sessions normally to close files/threads.
+
+The wrappers cover intake, checkpoints, CaseBuilder, interpretation, hypotheses,
+investigation, diagnosis, Change Requests, output validation/publication,
+LLM runner, budget/evidence operations and asynchronous test submit/confirmation.
+The session records span/parent IDs, operation, duration, known result/error code,
+replay marker and hashed case/package identities. Checkpoints supply manifest
+state, analysis revision and generation; actual LLM rounds supply usage metadata.
+Budget observations retain current verified revision balances, spend, reward and
+pending actions. Evidence publication counts exclude replay. Invocation counters
+include replay; actual model-round/token counters do not double-count successful
+stored calls. All counters start fresh with the observation session; service_id
+and start time identify resets. Durable lifetime history remains in the case.
+
+## Where to inspect
+
+Service-owned files under runtime/telemetry/:
+
+- service.jsonl: correlated structured JSON events; rotated by configured byte
+  size and backup count (default 5 MiB plus five backups).
+- status.json: atomically replaced observation status, active operations, last
+  confirmed progress/error, latest observed cases/budgets and queue inventory.
+- metrics.prom: atomically replaced Prometheus text exposition, suitable for a
+  textfile collector or later monitoring adapter. No HTTP endpoint is opened.
+
+```powershell
+python -m viano_bug_analysis.telemetry status --workspace C:/vba/work
+python -m viano_bug_analysis.telemetry errors --workspace C:/vba/work --case-id CASE-ID
+python -m viano_bug_analysis.telemetry events --workspace C:/vba/work --limit 100
+python -m viano_bug_analysis.telemetry events --workspace C:/vba/work --source-name incoming.zip
+python -m viano_bug_analysis.telemetry metrics --workspace C:/vba/work
+python -m viano_bug_analysis.telemetry scan --workspace C:/vba/work
+```
+
+Commands are passive. scan validates the latest managed case candidates and
+checkpoint identity from their manifests; filenames only select service-owned
+candidates. It does not perform intake, reconstruct service indices or run any
+analysis API. A missing observer status returns NOT_STARTED. Unreadable or unsafe
+files return a controlled TELEMETRY_READ and exit 2. status can successfully read
+an unhealthy/stopped service (exit 0); inspect its health fields, not exit 0 alone.
+errors selects ERROR events; events also includes expected warnings such as
+quarantine, provider retries and rejected model output. Quarantine events retain
+only hashed source names and the generated local quarantine directory key, so
+operators can find validation_error.json without logging the input filename.
+Readers tolerate incomplete last records and report corrupt records explicitly;
+startup separates an interrupted log tail before appending new records.
+
+## Liveness, progress and waiting
+
+An independent heartbeat thread updates every five seconds by default. It does
+not wait for model requests or archive inspection. A separate inventory thread
+samples every 60 seconds; slow archive checks cannot directly park the heartbeat.
+The sample has observed_at/complete/error fields and becomes unknown/stale rather
+than pretending an incomplete scan means zero work. Case checkpoint callbacks
+immediately update generation/state; inventory merging cannot regress a newer
+observed generation. Current output/review/handed-off cases are distinguished
+from runnable analysis, delayed retries, human feedback and manual provider waits.
+Expired human/manual waits never become automatically runnable.
+
+observer is LIVE, STOPPED, HEARTBEAT_STALE or NOT_STARTED. Work distinguishes
+WORKING, NO_RECENT_PROGRESS, INPUT_PENDING, READY_NOT_PROCESSING,
+WAITING_EXTERNAL, IDLE and QUEUE_UNKNOWN. Active operations expose start time,
+elapsed time, per-case progress age and a configurable stalled marker. Progress
+on another case cannot hide an old active phase with no confirmed checkpoints.
+A long phase is a diagnostic warning, not proof of deadlock. Nothing is restarted
+or cancelled automatically. Public read_status recomputes heartbeat/inventory age
+at read time, so a dead process does not remain LIVE because a stale JSON file
+was written previously.
+
+health distinguishes observation freshness, recent errors, incomplete inventory,
+no recent progress and telemetry sink failure. Last errors remain inspectable;
+a recent-error warning ages out. Recovered log/snapshot channels can become
+healthy again, while loss of observation integrity remains visible. This is an
+operational hint, not permission to terminate a worker or alter a case.
+
+Nested-operation snapshot writes are throttled by snapshot_interval_seconds
+(default one second). Top-level phase boundaries, heartbeat and explicit
+telemetry.flush() force a snapshot. JSONL events are still emitted immediately.
+All queue/progress views are sampled observations, never scheduler authority.
+
+## Metrics and data quality
+
+Metrics include operation counts/duration sums and counts, checkpoint counts,
+active operations, heartbeat/start/last-progress times, fresh inventory/inbox/case
+categories, telemetry failures, published evidence, observed budget units,
+actual model rounds, provider failures, reported tokens and estimated monetary
+costs. Monetary cost currencies are kept separate; investigation budget is not
+money. Unknown token/cost fields and usage of failed provider attempts have
+explicit counters. Estimates use configured rates only; they are not invoices.
+No token count, cost or model capacity is inferred from response size.
+
+Do not alert on a stale metrics file's observer_alive=1 alone. Monitor freshness
+using time() - vba_observer_heartbeat_time_seconds against your configured stale
+threshold. The same issue applies to all textfile collectors after process death.
+Unknown/stale inventories omit queue gauges instead of exporting fabricated zero.
+Case IDs, provider/model names and arbitrary error text are not metric labels,
+which bounds cardinality. Counters reset per observation session; current budget
+views carry analysis_revision and must not be treated as lifetime totals.
+
+## Privacy and failures
+
+Case/package/source/provider/model identifiers are hashed. Events use fixed
+operation/status fields and installed code allowlists. Unknown/custom error codes
+are replaced by UNCLASSIFIED_ERROR. Error chains include suppressed context,
+exception type, numeric errno, known schema keyword and bounded internal
+module/function/line frames. They exclude exception messages, source lines,
+locals, arguments, absolute paths, endpoint URLs, prompts, responses and case
+contents. External frames are intentionally masked. A model's raw reply remains
+only in the authorized case archive. Existing legacy LLM logging is replaced by
+safe observations within an active session; legacy gateway provider messages are
+suppressed only in the corresponding task context. Uninstrumented third-party
+application loggers are outside this telemetry sink's privacy guarantee.
+
+After startup, log/snapshot/observer-callback failures are fail-open for case
+execution, are counted, degrade observation status and produce a rate-limited
+safe stderr marker. No secret-bearing exception text is used in that fallback.
+A service manager can capture stderr when the normal sink is unwritable. Startup
+configuration, unsafe paths and an already-held writer lease fail explicitly
+before using the observation session. Snapshot/metric publication is atomic;
+telemetry files are mutable operational artifacts outside the case ZIP.
+
+The defaults in config/telemetry.example.toml are configurable operational limits,
+not case-package format boundaries. Scanner package limits can be supplied to
+Telemetry(package_limits=...) or are adopted from an attached store. max_cases,
+max_frames, snapshot byte limits and bounded metric series prevent uncontrolled
+observation growth; partial observations are reported explicitly.
+
+Tests cover a complete mocked-model pipeline, preserved case bytes/budget/replay,
+independent heartbeat, per-case stalls, passive/manual/human waits, privacy,
+provider failover, rotation, corrupt-tail restart, cancellation, sink recovery,
+callback failure, single-writer lease and CLI/status/schema behavior. Real-model
+reasoning quality and the future continuous scheduler are separate concerns.