Enable explicit caching by default on capable Gemini adapter
This commit is contained in:
1 parent
02a7364c58
commit
d5dac27f56
5 files changed
+53
-14
No files matched your search
@@ -1,4 +1,4 @@
|
||||
# Optional context caching and usage accounting
|
||||
# Context caching and usage accounting
|
||||
|
||||
The package remains the complete, authoritative record. Saved requests, evidence,
|
||||
responses, checkpoints and provider configuration references remain portable.
|
||||
@@ -14,14 +14,14 @@ The existing `openai_compatible` adapter now reads
|
||||
hits when the provider actually reports them, without changing existing requests
|
||||
or enabling explicit caching. An absent count means unknown, not zero.
|
||||
|
||||
The optional `gemini` adapter uses the native GenerateContent API. It moves the
|
||||
The `gemini` adapter uses the native GenerateContent API. It moves the
|
||||
unchanged system instructions and authoritative output schema to the beginning
|
||||
of the wire request. Only an exact top-level schema duplicate is removed from
|
||||
the live user JSON; no case facts, identifiers, evidence or assessments are
|
||||
removed. The archived request is not changed. This is deterministic transport
|
||||
preparation, not another LLM summary.
|
||||
|
||||
With `context_cache = true`, this instruction/schema prefix can be stored as a
|
||||
By default on the native Gemini adapter, this instruction/schema prefix is stored as a
|
||||
Gemini `cachedContents` resource. Case-specific input and all research history
|
||||
are still sent in full. The prefix is independent of the currently selected
|
||||
hypothesis. Reuse requires an exact match of endpoint, provider ID, model,
|
||||
@@ -41,12 +41,16 @@ improvement.
|
||||
## Configuration and fallback
|
||||
|
||||
See `config/llm.gemini-cache.example.toml`. Existing local configuration files
|
||||
are not edited. The native base URL ends in `/v1beta`, **not** `/v1beta/openai`.
|
||||
`context_cache` defaults to false and is allowed only for the native adapter.
|
||||
use the adapter selected in each provider entry. The native base URL ends in `/v1beta`, **not** `/v1beta/openai`.
|
||||
Omitting `context_cache` enables explicit caching on the native Gemini adapter.
|
||||
Adapters without explicit-cache support remain usable without it. Explicit true
|
||||
is allowed only for the native adapter; false remains an operational override.
|
||||
`cached_input_cost_per_million` can also be set for OpenAI-compatible providers
|
||||
that report cache usage. Prices remain optional, model-specific configuration.
|
||||
New default fields are omitted from configuration snapshots to preserve frozen
|
||||
operation hashes and replay of old cases.
|
||||
The effective enabled choice is persisted for native Gemini, so a change from
|
||||
uncached processing is visible to frozen-operation configuration checks. Other
|
||||
adapters retain their old default snapshots. Existing pending operations must
|
||||
keep their original configuration or be tested in a fresh workspace.
|
||||
|
||||
`cache_min_prefix_chars` is an eligibility heuristic, not a token count or API
|
||||
minimum. A short prefix may therefore obtain no explicit cache saving. The
|
||||
@@ -80,7 +84,8 @@ positive, consistent provider-reported cached token count
|
||||
(`LLM_CACHE_USAGE_UNCONFIRMED` otherwise). This is a test assertion, not a
|
||||
promise that Gemini will accept a prefix. A rejected/too-small cache ends the
|
||||
test explicitly. A response without confirmed usage may already have incurred
|
||||
API costs. Default behaviour remains optional; old operation hashes stay valid.
|
||||
API costs. Default behaviour enables caching where supported with graceful fallback;
|
||||
strict enforcement is a separate test mode.
|
||||
Strict cache creation may use half of the configured request timeout, instead
|
||||
of the five-second optional-optimization allowance. The overall deadline still
|
||||
applies; a creation HTTP status is retained for diagnosing a rejected cache.
|
||||
@@ -145,3 +150,8 @@ for this inspection; it is not a new quality or cost benchmark.
|
||||
- [Thinking configuration](https://ai.google.dev/gemini-api/docs/generate-content/thinking)
|
||||
- [Model-specific pricing](https://ai.google.dev/gemini-api/docs/pricing)
|
||||
- [Interactions session API, for a separate future adapter](https://ai.google.dev/gemini-api/docs/interactions-overview)
|
||||
|
||||
The regular local Gemini configurations were switched to the native adapter for
|
||||
this policy. Small prefixes still respect the configured eligibility heuristic;
|
||||
API minimums are not bypassed by padding. Transient cache failures retain the
|
||||
full-input fallback so an optimization outage does not stop normal analysis.
|
||||
Reference in new issue
Block a user