VianoBugAnalysis/docs/telemetry-v1.md

Operational telemetry v1

LLM round events expose numeric http_status for nonretryable HTTP failures and for every entry in failed_providers. Use the events CLI to inspect temporary provider failures (warning events). For example, 429 indicates a quota/rate-limit response; 503 indicates service unavailability. Missing status on older records, timeouts or connection errors means unknown, not success. Provider bodies and headers are never logged. This field does not change retries or case budgets.

Telemetry observes existing library execution, not case truth. It never resumes BLOCKED cases, refills budgets, changes routing, retries a model, edits packages, executes tools or deletes case artifacts. docs/architecture.md is unchanged. A continuous analysis worker/scheduler remains the next integration milestone. The observation heartbeat is not a substitute for that worker or evidence that an analysis is progressing.

Enable observations

from viano_bug_analysis.lifecycle import Workspace, CheckpointStore
from viano_bug_analysis.telemetry import Telemetry, TelemetryConfig
from viano_bug_analysis.analysis import CaseInterpreter

workspace = Workspace("work")
config = TelemetryConfig.load("config/telemetry.local.toml")
with Telemetry(workspace, config) as telemetry:
    store = CheckpointStore(workspace, telemetry=telemetry)
    # In an async driver, use existing APIs normally:
    # await CaseInterpreter(store, gateway).interpret(case_id, package_id)

Telemetry is explicitly opt-in. Without an attached running session, existing API behavior is unchanged. InboxProcessor also accepts telemetry=telemetry. Stores can be attached after construction with store.telemetry=telemetry. Use one observation session per workspace; a nonblocking OS lease prevents competing log rotations/status writers. Concurrent threads and async tasks in that process have independent, correlated operation spans. There is no global worker lock across observation or model calls. Stop sessions normally to close files/threads.

The wrappers cover intake, checkpoints, CaseBuilder, interpretation, hypotheses, investigation, diagnosis, Change Requests, output validation/publication, LLM runner, budget/evidence operations and asynchronous test submit/confirmation. The session records span/parent IDs, operation, duration, known result/error code, replay marker and hashed case/package identities. Checkpoints supply manifest state, analysis revision and generation; actual LLM rounds supply usage metadata. Budget observations retain current verified revision balances, spend, reward and pending actions. Evidence publication counts exclude replay. Invocation counters include replay; actual model-round/token counters do not double-count successful stored calls. All counters start fresh with the observation session; service_id and start time identify resets. Durable lifetime history remains in the case.

Where to inspect

Service-owned files under runtime/telemetry/:

• service.jsonl: correlated structured JSON events; rotated by configured byte size and backup count (default 5 MiB plus five backups).
• status.json: atomically replaced observation status, active operations, last confirmed progress/error, latest observed cases/budgets and queue inventory.
• metrics.prom: atomically replaced Prometheus text exposition, suitable for a textfile collector or later monitoring adapter. No HTTP endpoint is opened.

python -m viano_bug_analysis.telemetry status --workspace C:/vba/work
python -m viano_bug_analysis.telemetry errors --workspace C:/vba/work --case-id CASE-ID
python -m viano_bug_analysis.telemetry events --workspace C:/vba/work --limit 100
python -m viano_bug_analysis.telemetry events --workspace C:/vba/work --source-name incoming.zip
python -m viano_bug_analysis.telemetry metrics --workspace C:/vba/work
python -m viano_bug_analysis.telemetry scan --workspace C:/vba/work

Commands are passive. scan validates the latest managed case candidates and checkpoint identity from their manifests; filenames only select service-owned candidates. It does not perform intake, reconstruct service indices or run any analysis API. A missing observer status returns NOT_STARTED. Unreadable or unsafe files return a controlled TELEMETRY_READ and exit 2. status can successfully read an unhealthy/stopped service (exit 0); inspect its health fields, not exit 0 alone. errors selects ERROR events; events also includes expected warnings such as quarantine, provider retries and rejected model output. Quarantine events retain only hashed source names and the generated local quarantine directory key, so operators can find validation_error.json without logging the input filename. Readers tolerate incomplete last records and report corrupt records explicitly; startup separates an interrupted log tail before appending new records.

Liveness, progress and waiting

An independent heartbeat thread updates every five seconds by default. It does not wait for model requests or archive inspection. A separate inventory thread samples every 60 seconds; slow archive checks cannot directly park the heartbeat. The sample has observed_at/complete/error fields and becomes unknown/stale rather than pretending an incomplete scan means zero work. Case checkpoint callbacks immediately update generation/state; inventory merging cannot regress a newer observed generation. Current output/review/handed-off cases are distinguished from runnable analysis, delayed retries, human feedback and manual provider waits. Expired human/manual waits never become automatically runnable.

observer is LIVE, STOPPED, HEARTBEAT_STALE or NOT_STARTED. Work distinguishes WORKING, NO_RECENT_PROGRESS, INPUT_PENDING, READY_NOT_PROCESSING, WAITING_EXTERNAL, IDLE and QUEUE_UNKNOWN. Active operations expose start time, elapsed time, per-case progress age and a configurable stalled marker. Progress on another case cannot hide an old active phase with no confirmed checkpoints. A long phase is a diagnostic warning, not proof of deadlock. Nothing is restarted or cancelled automatically. Public read_status recomputes heartbeat/inventory age at read time, so a dead process does not remain LIVE because a stale JSON file was written previously.

health distinguishes observation freshness, recent errors, incomplete inventory, no recent progress and telemetry sink failure. Last errors remain inspectable; a recent-error warning ages out. Recovered log/snapshot channels can become healthy again, while loss of observation integrity remains visible. This is an operational hint, not permission to terminate a worker or alter a case.

Nested-operation snapshot writes are throttled by snapshot_interval_seconds (default one second). Top-level phase boundaries, heartbeat and explicit telemetry.flush() force a snapshot. JSONL events are still emitted immediately. All queue/progress views are sampled observations, never scheduler authority.

Metrics and data quality

Metrics include operation counts/duration sums and counts, checkpoint counts, active operations, heartbeat/start/last-progress times, fresh inventory/inbox/case categories, telemetry failures, published evidence, observed budget units, actual model rounds, provider failures, reported tokens and estimated monetary costs. Monetary cost currencies are kept separate; investigation budget is not money. Unknown token/cost fields and usage of failed provider attempts have explicit counters. Estimates use configured rates only; they are not invoices. No token count, cost or model capacity is inferred from response size.

Do not alert on a stale metrics file's observer_alive=1 alone. Monitor freshness using time() - vba_observer_heartbeat_time_seconds against your configured stale threshold. The same issue applies to all textfile collectors after process death. Unknown/stale inventories omit queue gauges instead of exporting fabricated zero. Case IDs, provider/model names and arbitrary error text are not metric labels, which bounds cardinality. Counters reset per observation session; current budget views carry analysis_revision and must not be treated as lifetime totals.

Privacy and failures

Case/package/source/provider/model identifiers are hashed. Events use fixed operation/status fields and installed code allowlists. Unknown/custom error codes are replaced by UNCLASSIFIED_ERROR. Error chains include suppressed context, exception type, numeric errno, known schema keyword and bounded internal module/function/line frames. They exclude exception messages, source lines, locals, arguments, absolute paths, endpoint URLs, prompts, responses and case contents. External frames are intentionally masked. A model's raw reply remains only in the authorized case archive. Existing legacy LLM logging is replaced by safe observations within an active session; legacy gateway provider messages are suppressed only in the corresponding task context. Uninstrumented third-party application loggers are outside this telemetry sink's privacy guarantee.

After startup, log/snapshot/observer-callback failures are fail-open for case execution, are counted, degrade observation status and produce a rate-limited safe stderr marker. No secret-bearing exception text is used in that fallback. A service manager can capture stderr when the normal sink is unwritable. Startup configuration, unsafe paths and an already-held writer lease fail explicitly before using the observation session. Snapshot/metric publication is atomic; telemetry files are mutable operational artifacts outside the case ZIP.

The defaults in config/telemetry.example.toml are configurable operational limits, not case-package format boundaries. Scanner package limits can be supplied to Telemetry(package_limits=...) or are adopted from an attached store. max_cases, max_frames, snapshot byte limits and bounded metric series prevent uncontrolled observation growth; partial observations are reported explicitly.

Tests cover a complete mocked-model pipeline, preserved case bytes/budget/replay, independent heartbeat, per-case stalls, passive/manual/human waits, privacy, provider failover, rotation, corrupt-tail restart, cancellation, sink recovery, callback failure, single-writer lease and CLI/status/schema behavior. Real-model reasoning quality and the future continuous scheduler are separate concerns.