Enable explicit caching by default on capable Gemini adapter

This commit is contained in:
Nico committed 2026-10-16 16:02:40 +02:00
1 parent 02a7364c58
commit d5dac27f56
5 files changed
+53 -14

No files matched your search

+18 -8
View File
@@ -1,4 +1,4 @@
# Optional context caching and usage accounting
# Context caching and usage accounting
The package remains the complete, authoritative record. Saved requests, evidence,
responses, checkpoints and provider configuration references remain portable.
@@ -14,14 +14,14 @@ The existing `openai_compatible` adapter now reads
hits when the provider actually reports them, without changing existing requests
or enabling explicit caching. An absent count means unknown, not zero.
The optional `gemini` adapter uses the native GenerateContent API. It moves the
The `gemini` adapter uses the native GenerateContent API. It moves the
unchanged system instructions and authoritative output schema to the beginning
of the wire request. Only an exact top-level schema duplicate is removed from
the live user JSON; no case facts, identifiers, evidence or assessments are
removed. The archived request is not changed. This is deterministic transport
preparation, not another LLM summary.
With `context_cache = true`, this instruction/schema prefix can be stored as a
By default on the native Gemini adapter, this instruction/schema prefix is stored as a
Gemini `cachedContents` resource. Case-specific input and all research history
are still sent in full. The prefix is independent of the currently selected
hypothesis. Reuse requires an exact match of endpoint, provider ID, model,
@@ -41,12 +41,16 @@ improvement.
## Configuration and fallback
See `config/llm.gemini-cache.example.toml`. Existing local configuration files
are not edited. The native base URL ends in `/v1beta`, **not** `/v1beta/openai`.
`context_cache` defaults to false and is allowed only for the native adapter.
use the adapter selected in each provider entry. The native base URL ends in `/v1beta`, **not** `/v1beta/openai`.
Omitting `context_cache` enables explicit caching on the native Gemini adapter.
Adapters without explicit-cache support remain usable without it. Explicit true
is allowed only for the native adapter; false remains an operational override.
`cached_input_cost_per_million` can also be set for OpenAI-compatible providers
that report cache usage. Prices remain optional, model-specific configuration.
New default fields are omitted from configuration snapshots to preserve frozen
operation hashes and replay of old cases.
The effective enabled choice is persisted for native Gemini, so a change from
uncached processing is visible to frozen-operation configuration checks. Other
adapters retain their old default snapshots. Existing pending operations must
keep their original configuration or be tested in a fresh workspace.
`cache_min_prefix_chars` is an eligibility heuristic, not a token count or API
minimum. A short prefix may therefore obtain no explicit cache saving. The
@@ -80,7 +84,8 @@ positive, consistent provider-reported cached token count
(`LLM_CACHE_USAGE_UNCONFIRMED` otherwise). This is a test assertion, not a
promise that Gemini will accept a prefix. A rejected/too-small cache ends the
test explicitly. A response without confirmed usage may already have incurred
API costs. Default behaviour remains optional; old operation hashes stay valid.
API costs. Default behaviour enables caching where supported with graceful fallback;
strict enforcement is a separate test mode.
Strict cache creation may use half of the configured request timeout, instead
of the five-second optional-optimization allowance. The overall deadline still
applies; a creation HTTP status is retained for diagnosing a rejected cache.
@@ -145,3 +150,8 @@ for this inspection; it is not a new quality or cost benchmark.
- [Thinking configuration](https://ai.google.dev/gemini-api/docs/generate-content/thinking)
- [Model-specific pricing](https://ai.google.dev/gemini-api/docs/pricing)
- [Interactions session API, for a separate future adapter](https://ai.google.dev/gemini-api/docs/interactions-overview)
The regular local Gemini configurations were switched to the native adapter for
this policy. Small prefixes still respect the configured eligibility heuristic;
API minimums are not bypassed by padding. Transient cache failures retain the
full-input fallback so an optimization outage does not stop normal analysis.