9.8 KiB
Context caching and usage accounting
The package remains the complete, authoritative record. Saved requests, evidence,
responses, checkpoints and provider configuration references remain portable.
No cache handle, credential or provider conversation ID becomes required case
state. docs/architecture.md and the investigation full-history policy are
unchanged. All evidence remains in every investigation input.
Transport and complete case context
The existing openai_compatible adapter now reads
usage.prompt_tokens_details.cached_tokens and
usage.completion_tokens_details.reasoning_tokens. This exposes implicit cache
hits when the provider actually reports them, without changing existing requests
or enabling explicit caching. An absent count means unknown, not zero.
The gemini adapter uses the native GenerateContent API. It moves the
unchanged system instructions and authoritative output schema to the beginning
of the wire request. Only an exact top-level schema duplicate is removed from
the live user JSON; no case facts, identifiers, evidence or assessments are
removed. The archived request is not changed. This is deterministic transport
preparation, not another LLM summary.
By default on the native Gemini adapter, instructions, schema and the stable
case context are stored as a Gemini cachedContents resource. Hypothesis state,
budget, questions and all research history are still sent in full as live input.
See stable case cache for the partition and proof contract. The prefix is independent of the currently selected
hypothesis. Reuse requires an exact match of endpoint, provider ID, model,
credential fingerprint, prefix and TTL. The adapter retains at most eight
entries (including short-lived failed-creation entries), in process memory only.
Different phases have different instructions/schema and therefore different
cache entries. Final diagnosis remains an independent request, without a prior
investigation conversation.
This intentionally does not implement a provider conversation/session or cache the growing evidence history. It establishes measured caching without changing what the model knows. A later session adapter must preserve full replay, provider failover and an independent final evaluation; its quality and total cost need a separate comparison. Caching itself promises no reasoning-quality improvement.
Configuration and fallback
See config/llm.gemini-cache.example.toml. Existing local configuration files
use the adapter selected in each provider entry. The native base URL ends in /v1beta, not /v1beta/openai.
Omitting context_cache enables explicit caching on the native Gemini adapter.
Adapters without explicit-cache support remain usable without it. Explicit true
is allowed only for the native adapter; false remains an operational override.
cached_input_cost_per_million can also be set for OpenAI-compatible providers
that report cache usage. Prices remain optional, model-specific configuration.
The effective enabled choice is persisted for native Gemini, so a change from
uncached processing is visible to frozen-operation configuration checks. Other
adapters retain their old default snapshots. Existing pending operations must
keep their original configuration or be tested in a fresh workspace.
cache_min_prefix_chars is an eligibility heuristic, not a token count or API
minimum. A short prefix may therefore obtain no explicit cache saving. The
provider remains authoritative about minimum tokens and model support. There
is no padding to meet a minimum. TTL defaults to five minutes; expiry is not
extended automatically. Restarting a worker rebuilds a cache from complete
requests rather than depending on a remote handle.
Cache creation has a small bounded part of the overall request deadline (at
most five seconds and one quarter of the provider timeout). Rejection, timeout,
unsupported caching or an invalid cache-create response falls back to the full
request. Failed creation is suppressed for sixty seconds. A cached generation
rejected with HTTP 400, 403 or 404 invalidates the handle and retries once with
the full request, within the same overall deadline. Other failures use normal
provider failover/retry. No unbounded transport retry loop is introduced.
Successful allocations expire remotely by TTL even if the worker exits. Keys
are sent in headers; redirects and ambient proxies are disabled. Native
reasoning_effort uses Gemini thinkingLevel (model support must be checked);
none is rejected rather than silently mapped to another setting.
Reading the measurements
For a strict offline-prepared/live-user-run cache test, set cache_required = true.
Optionally restrict it with cache_required_prompt_prefix = "investigation_project_";
this requires cache usage throughout investigation while leaving small earlier
phases optional. Without a prefix, the requirement covers all calls from that
provider. It requires context_cache = true and the native Gemini adapter.
The worker refuses to generate without a handle (LLM_CACHE_REQUIRED), refuses
uncached retry/failover on a required call, and only accepts a response with a
positive, consistent provider-reported cached token count
(LLM_CACHE_USAGE_UNCONFIRMED otherwise). This is a test assertion, not a
promise that Gemini will accept a prefix. A rejected/too-small cache ends the
test explicitly. A response without confirmed usage may already have incurred
API costs. Default behaviour enables caching where supported with graceful fallback;
strict enforcement is a separate test mode.
Strict cache creation may use half of the configured request timeout, instead
of the five-second optional-optimization allowance. The overall deadline still
applies; a creation HTTP status is retained for diagnosing a rejected cache.
Each round's metadata and safe llm_round telemetry can include:
| Field | Meaning |
|---|---|
input_tokens |
Total effective input, including cached input |
cached_input_tokens |
Provider-reported cached part |
uncached_input_tokens |
Total minus cached part, only if both are known and consistent |
output_tokens |
Billed generated tokens; native Gemini candidates plus thoughts |
reasoning_tokens |
Reported thinking subset, not added to output cost a second time |
estimated_input_cost |
Uncached input estimate |
estimated_cached_input_cost |
Cached input estimate |
estimated_output_cost |
Output estimate, including thinking |
estimated_cost |
Generation estimate; excludes separately reported cache storage |
cache_status |
disabled, below_minimum, created, reused, or fallback |
cache_created_tokens |
Newly allocated tokens, recorded once per allocation |
estimated_cache_storage_cost |
Newly allocated tokens × configured TTL hours × storage rate / 1M |
Token counters are available as vba_llm_tokens_total{direction="..."}; the
directions overlap (total input includes cached input, output includes thinking).
Do not sum all directions. Cost components use
vba_llm_cost_component_total{component="...",currency="..."}. The storage
component is distinct from generation costs. Unknown values remain null and
usage-unknown counters identify incomplete accounting.
Storage is an estimate for the whole reserved TTL, counted on creation only,
not on every hit. It is not a provider invoice. A cache created before a
temporary generation failure is recorded in the failed provider attempt's
cache_usage; ordinary unsuccessful generation attempts can still have unknown
usage. A crash between remote allocation and local persistence can also leave
unrecorded charges. Never treat these estimates as a precise account balance.
If cache creation loses its response, newly allocated tokens remain unknown;
they are not reported as zero.
For old configurations with no cache price and no reported cached count, the legacy full-input-price estimate remains available. With a known positive cache count and no cache rate, or with a configured cache rate and missing cache usage, the exact generation estimate is unknown. No fake discount is inferred.
For an economic comparison, use measured cached/uncached input and output estimates plus newly allocated storage, latency and outcome quality. There is no program-side Euro abort cap. The existing local configuration remains usable for collecting implicit-hit measurements before choosing the native adapter. No paid call is needed to run the offline protocol/accounting regression tests.
An offline inspection of the archived BUG-000017 selected-context trial found an
instruction/schema prefix of 18,703 characters for investigation_project_v9,
with 59,900 characters remaining live in the inspected request. That prefix
passes the default character heuristic; only a real provider response can
confirm token eligibility, hits and savings. The smaller entity-resolution
prefixes (3,730 / 4,340 characters) do not pass it. No live API call was made
for this inspection; it is not a new quality or cost benchmark.
API sources checked 2026-10-16
- Gemini caching guide
- CachedContents reference
- GenerateContent usage metadata
- Thinking configuration
- Model-specific pricing
- Interactions session API, for a separate future adapter
The regular local Gemini configurations were switched to the native adapter for this policy. Small prefixes still respect the configured eligibility heuristic; API minimums are not bypassed by padding. Transient cache failures retain the full-input fallback so an optimization outage does not stop normal analysis.