163 lines
9.7 KiB
Markdown
163 lines
9.7 KiB
Markdown
# Operational telemetry v1
|
|
|
|
Telemetry observes existing library execution, not case truth. It never resumes
|
|
BLOCKED cases, refills budgets, changes routing, retries a model, edits packages,
|
|
executes tools or deletes case artifacts. `docs/architecture.md` is unchanged.
|
|
A continuous analysis worker/scheduler remains the next integration milestone.
|
|
The observation heartbeat is not a substitute for that worker or evidence that
|
|
an analysis is progressing.
|
|
|
|
## Enable observations
|
|
|
|
```python
|
|
from viano_bug_analysis.lifecycle import Workspace, CheckpointStore
|
|
from viano_bug_analysis.telemetry import Telemetry, TelemetryConfig
|
|
from viano_bug_analysis.analysis import CaseInterpreter
|
|
|
|
workspace = Workspace("work")
|
|
config = TelemetryConfig.load("config/telemetry.local.toml")
|
|
with Telemetry(workspace, config) as telemetry:
|
|
store = CheckpointStore(workspace, telemetry=telemetry)
|
|
# In an async driver, use existing APIs normally:
|
|
# await CaseInterpreter(store, gateway).interpret(case_id, package_id)
|
|
```
|
|
|
|
Telemetry is explicitly opt-in. Without an attached running session, existing API
|
|
behavior is unchanged. InboxProcessor also accepts telemetry=telemetry. Stores
|
|
can be attached after construction with store.telemetry=telemetry. Use one
|
|
observation session per workspace; a nonblocking OS lease prevents competing
|
|
log rotations/status writers. Concurrent threads and async tasks in that process
|
|
have independent, correlated operation spans. There is no global worker lock
|
|
across observation or model calls. Stop sessions normally to close files/threads.
|
|
|
|
The wrappers cover intake, checkpoints, CaseBuilder, interpretation, hypotheses,
|
|
investigation, diagnosis, Change Requests, output validation/publication,
|
|
LLM runner, budget/evidence operations and asynchronous test submit/confirmation.
|
|
The session records span/parent IDs, operation, duration, known result/error code,
|
|
replay marker and hashed case/package identities. Checkpoints supply manifest
|
|
state, analysis revision and generation; actual LLM rounds supply usage metadata.
|
|
Budget observations retain current verified revision balances, spend, reward and
|
|
pending actions. Evidence publication counts exclude replay. Invocation counters
|
|
include replay; actual model-round/token counters do not double-count successful
|
|
stored calls. All counters start fresh with the observation session; service_id
|
|
and start time identify resets. Durable lifetime history remains in the case.
|
|
|
|
## Where to inspect
|
|
|
|
Service-owned files under runtime/telemetry/:
|
|
|
|
- service.jsonl: correlated structured JSON events; rotated by configured byte
|
|
size and backup count (default 5 MiB plus five backups).
|
|
- status.json: atomically replaced observation status, active operations, last
|
|
confirmed progress/error, latest observed cases/budgets and queue inventory.
|
|
- metrics.prom: atomically replaced Prometheus text exposition, suitable for a
|
|
textfile collector or later monitoring adapter. No HTTP endpoint is opened.
|
|
|
|
```powershell
|
|
python -m viano_bug_analysis.telemetry status --workspace C:/vba/work
|
|
python -m viano_bug_analysis.telemetry errors --workspace C:/vba/work --case-id CASE-ID
|
|
python -m viano_bug_analysis.telemetry events --workspace C:/vba/work --limit 100
|
|
python -m viano_bug_analysis.telemetry events --workspace C:/vba/work --source-name incoming.zip
|
|
python -m viano_bug_analysis.telemetry metrics --workspace C:/vba/work
|
|
python -m viano_bug_analysis.telemetry scan --workspace C:/vba/work
|
|
```
|
|
|
|
Commands are passive. scan validates the latest managed case candidates and
|
|
checkpoint identity from their manifests; filenames only select service-owned
|
|
candidates. It does not perform intake, reconstruct service indices or run any
|
|
analysis API. A missing observer status returns NOT_STARTED. Unreadable or unsafe
|
|
files return a controlled TELEMETRY_READ and exit 2. status can successfully read
|
|
an unhealthy/stopped service (exit 0); inspect its health fields, not exit 0 alone.
|
|
errors selects ERROR events; events also includes expected warnings such as
|
|
quarantine, provider retries and rejected model output. Quarantine events retain
|
|
only hashed source names and the generated local quarantine directory key, so
|
|
operators can find validation_error.json without logging the input filename.
|
|
Readers tolerate incomplete last records and report corrupt records explicitly;
|
|
startup separates an interrupted log tail before appending new records.
|
|
|
|
## Liveness, progress and waiting
|
|
|
|
An independent heartbeat thread updates every five seconds by default. It does
|
|
not wait for model requests or archive inspection. A separate inventory thread
|
|
samples every 60 seconds; slow archive checks cannot directly park the heartbeat.
|
|
The sample has observed_at/complete/error fields and becomes unknown/stale rather
|
|
than pretending an incomplete scan means zero work. Case checkpoint callbacks
|
|
immediately update generation/state; inventory merging cannot regress a newer
|
|
observed generation. Current output/review/handed-off cases are distinguished
|
|
from runnable analysis, delayed retries, human feedback and manual provider waits.
|
|
Expired human/manual waits never become automatically runnable.
|
|
|
|
observer is LIVE, STOPPED, HEARTBEAT_STALE or NOT_STARTED. Work distinguishes
|
|
WORKING, NO_RECENT_PROGRESS, INPUT_PENDING, READY_NOT_PROCESSING,
|
|
WAITING_EXTERNAL, IDLE and QUEUE_UNKNOWN. Active operations expose start time,
|
|
elapsed time, per-case progress age and a configurable stalled marker. Progress
|
|
on another case cannot hide an old active phase with no confirmed checkpoints.
|
|
A long phase is a diagnostic warning, not proof of deadlock. Nothing is restarted
|
|
or cancelled automatically. Public read_status recomputes heartbeat/inventory age
|
|
at read time, so a dead process does not remain LIVE because a stale JSON file
|
|
was written previously.
|
|
|
|
health distinguishes observation freshness, recent errors, incomplete inventory,
|
|
no recent progress and telemetry sink failure. Last errors remain inspectable;
|
|
a recent-error warning ages out. Recovered log/snapshot channels can become
|
|
healthy again, while loss of observation integrity remains visible. This is an
|
|
operational hint, not permission to terminate a worker or alter a case.
|
|
|
|
Nested-operation snapshot writes are throttled by snapshot_interval_seconds
|
|
(default one second). Top-level phase boundaries, heartbeat and explicit
|
|
telemetry.flush() force a snapshot. JSONL events are still emitted immediately.
|
|
All queue/progress views are sampled observations, never scheduler authority.
|
|
|
|
## Metrics and data quality
|
|
|
|
Metrics include operation counts/duration sums and counts, checkpoint counts,
|
|
active operations, heartbeat/start/last-progress times, fresh inventory/inbox/case
|
|
categories, telemetry failures, published evidence, observed budget units,
|
|
actual model rounds, provider failures, reported tokens and estimated monetary
|
|
costs. Monetary cost currencies are kept separate; investigation budget is not
|
|
money. Unknown token/cost fields and usage of failed provider attempts have
|
|
explicit counters. Estimates use configured rates only; they are not invoices.
|
|
No token count, cost or model capacity is inferred from response size.
|
|
|
|
Do not alert on a stale metrics file's observer_alive=1 alone. Monitor freshness
|
|
using time() - vba_observer_heartbeat_time_seconds against your configured stale
|
|
threshold. The same issue applies to all textfile collectors after process death.
|
|
Unknown/stale inventories omit queue gauges instead of exporting fabricated zero.
|
|
Case IDs, provider/model names and arbitrary error text are not metric labels,
|
|
which bounds cardinality. Counters reset per observation session; current budget
|
|
views carry analysis_revision and must not be treated as lifetime totals.
|
|
|
|
## Privacy and failures
|
|
|
|
Case/package/source/provider/model identifiers are hashed. Events use fixed
|
|
operation/status fields and installed code allowlists. Unknown/custom error codes
|
|
are replaced by UNCLASSIFIED_ERROR. Error chains include suppressed context,
|
|
exception type, numeric errno, known schema keyword and bounded internal
|
|
module/function/line frames. They exclude exception messages, source lines,
|
|
locals, arguments, absolute paths, endpoint URLs, prompts, responses and case
|
|
contents. External frames are intentionally masked. A model's raw reply remains
|
|
only in the authorized case archive. Existing legacy LLM logging is replaced by
|
|
safe observations within an active session; legacy gateway provider messages are
|
|
suppressed only in the corresponding task context. Uninstrumented third-party
|
|
application loggers are outside this telemetry sink's privacy guarantee.
|
|
|
|
After startup, log/snapshot/observer-callback failures are fail-open for case
|
|
execution, are counted, degrade observation status and produce a rate-limited
|
|
safe stderr marker. No secret-bearing exception text is used in that fallback.
|
|
A service manager can capture stderr when the normal sink is unwritable. Startup
|
|
configuration, unsafe paths and an already-held writer lease fail explicitly
|
|
before using the observation session. Snapshot/metric publication is atomic;
|
|
telemetry files are mutable operational artifacts outside the case ZIP.
|
|
|
|
The defaults in config/telemetry.example.toml are configurable operational limits,
|
|
not case-package format boundaries. Scanner package limits can be supplied to
|
|
Telemetry(package_limits=...) or are adopted from an attached store. max_cases,
|
|
max_frames, snapshot byte limits and bounded metric series prevent uncontrolled
|
|
observation growth; partial observations are reported explicitly.
|
|
|
|
Tests cover a complete mocked-model pipeline, preserved case bytes/budget/replay,
|
|
independent heartbeat, per-case stalls, passive/manual/human waits, privacy,
|
|
provider failover, rotation, corrupt-tail restart, cancellation, sink recovery,
|
|
callback failure, single-writer lease and CLI/status/schema behavior. Real-model
|
|
reasoning quality and the future continuous scheduler are separate concerns.
|