8.9 KiB
Optional context caching and usage accounting
The package remains the complete, authoritative record. Saved requests, evidence,
responses, checkpoints and provider configuration references remain portable.
No cache handle, credential or provider conversation ID becomes required case
state. docs/architecture.md and the investigation full-history policy are
unchanged. All evidence remains in every investigation input.
First implementation
The existing openai_compatible adapter now reads
usage.prompt_tokens_details.cached_tokens and
usage.completion_tokens_details.reasoning_tokens. This exposes implicit cache
hits when the provider actually reports them, without changing existing requests
or enabling explicit caching. An absent count means unknown, not zero.
The optional gemini adapter uses the native GenerateContent API. It moves the
unchanged system instructions and authoritative output schema to the beginning
of the wire request. Only an exact top-level schema duplicate is removed from
the live user JSON; no case facts, identifiers, evidence or assessments are
removed. The archived request is not changed. This is deterministic transport
preparation, not another LLM summary.
With context_cache = true, this instruction/schema prefix can be stored as a
Gemini cachedContents resource. Case-specific input and all research history
are still sent in full. The prefix is independent of the currently selected
hypothesis. Reuse requires an exact match of endpoint, provider ID, model,
credential fingerprint, prefix and TTL. The adapter retains at most eight
entries (including short-lived failed-creation entries), in process memory only.
Different phases have different instructions/schema and therefore different
cache entries. Final diagnosis remains an independent request, without a prior
investigation conversation.
This intentionally does not implement a provider conversation/session or cache the growing evidence history. It establishes measured caching without changing what the model knows. A later session adapter must preserve full replay, provider failover and an independent final evaluation; its quality and total cost need a separate comparison. Caching itself promises no reasoning-quality improvement.
Configuration and fallback
See config/llm.gemini-cache.example.toml. Existing local configuration files
are not edited. The native base URL ends in /v1beta, not /v1beta/openai.
context_cache defaults to false and is allowed only for the native adapter.
cached_input_cost_per_million can also be set for OpenAI-compatible providers
that report cache usage. Prices remain optional, model-specific configuration.
New default fields are omitted from configuration snapshots to preserve frozen
operation hashes and replay of old cases.
cache_min_prefix_chars is an eligibility heuristic, not a token count or API
minimum. A short prefix may therefore obtain no explicit cache saving. The
provider remains authoritative about minimum tokens and model support. There
is no padding to meet a minimum. TTL defaults to five minutes; expiry is not
extended automatically. Restarting a worker rebuilds a cache from complete
requests rather than depending on a remote handle.
Cache creation has a small bounded part of the overall request deadline (at
most five seconds and one quarter of the provider timeout). Rejection, timeout,
unsupported caching or an invalid cache-create response falls back to the full
request. Failed creation is suppressed for sixty seconds. A cached generation
rejected with HTTP 400, 403 or 404 invalidates the handle and retries once with
the full request, within the same overall deadline. Other failures use normal
provider failover/retry. No unbounded transport retry loop is introduced.
Successful allocations expire remotely by TTL even if the worker exits. Keys
are sent in headers; redirects and ambient proxies are disabled. Native
reasoning_effort uses Gemini thinkingLevel (model support must be checked);
none is rejected rather than silently mapped to another setting.
Reading the measurements
For a strict offline-prepared/live-user-run cache test, set cache_required = true.
Optionally restrict it with cache_required_prompt_prefix = "investigation_project_";
this requires cache usage throughout investigation while leaving small earlier
phases optional. Without a prefix, the requirement covers all calls from that
provider. It requires context_cache = true and the native Gemini adapter.
The worker refuses to generate without a handle (LLM_CACHE_REQUIRED), refuses
uncached retry/failover on a required call, and only accepts a response with a
positive, consistent provider-reported cached token count
(LLM_CACHE_USAGE_UNCONFIRMED otherwise). This is a test assertion, not a
promise that Gemini will accept a prefix. A rejected/too-small cache ends the
test explicitly. A response without confirmed usage may already have incurred
API costs. Default behaviour remains optional; old operation hashes stay valid.
Strict cache creation may use half of the configured request timeout, instead
of the five-second optional-optimization allowance. The overall deadline still
applies; a creation HTTP status is retained for diagnosing a rejected cache.
Each round's metadata and safe llm_round telemetry can include:
| Field | Meaning |
|---|---|
input_tokens |
Total effective input, including cached input |
cached_input_tokens |
Provider-reported cached part |
uncached_input_tokens |
Total minus cached part, only if both are known and consistent |
output_tokens |
Billed generated tokens; native Gemini candidates plus thoughts |
reasoning_tokens |
Reported thinking subset, not added to output cost a second time |
estimated_input_cost |
Uncached input estimate |
estimated_cached_input_cost |
Cached input estimate |
estimated_output_cost |
Output estimate, including thinking |
estimated_cost |
Generation estimate; excludes separately reported cache storage |
cache_status |
disabled, below_minimum, created, reused, or fallback |
cache_created_tokens |
Newly allocated tokens, recorded once per allocation |
estimated_cache_storage_cost |
Newly allocated tokens × configured TTL hours × storage rate / 1M |
Token counters are available as vba_llm_tokens_total{direction="..."}; the
directions overlap (total input includes cached input, output includes thinking).
Do not sum all directions. Cost components use
vba_llm_cost_component_total{component="...",currency="..."}. The storage
component is distinct from generation costs. Unknown values remain null and
usage-unknown counters identify incomplete accounting.
Storage is an estimate for the whole reserved TTL, counted on creation only,
not on every hit. It is not a provider invoice. A cache created before a
temporary generation failure is recorded in the failed provider attempt's
cache_usage; ordinary unsuccessful generation attempts can still have unknown
usage. A crash between remote allocation and local persistence can also leave
unrecorded charges. Never treat these estimates as a precise account balance.
If cache creation loses its response, newly allocated tokens remain unknown;
they are not reported as zero.
For old configurations with no cache price and no reported cached count, the legacy full-input-price estimate remains available. With a known positive cache count and no cache rate, or with a configured cache rate and missing cache usage, the exact generation estimate is unknown. No fake discount is inferred.
For an economic comparison, use measured cached/uncached input and output estimates plus newly allocated storage, latency and outcome quality. There is no program-side Euro abort cap. The existing local configuration remains usable for collecting implicit-hit measurements before choosing the native adapter. No paid call is needed to run the offline protocol/accounting regression tests.
An offline inspection of the archived BUG-000017 selected-context trial found an
instruction/schema prefix of 18,703 characters for investigation_project_v9,
with 59,900 characters remaining live in the inspected request. That prefix
passes the default character heuristic; only a real provider response can
confirm token eligibility, hits and savings. The smaller entity-resolution
prefixes (3,730 / 4,340 characters) do not pass it. No live API call was made
for this inspection; it is not a new quality or cost benchmark.