Add configurable LLM gateway with ordered failover and durable retries

docs/llm-gateway-v1.md

@@ -0,0 +1,139 @@
+# LLMGateway v1
+
+This milestone implements architecture section 13: configuration, provider transport,
+ordered failover, bounded retries, schema validation, metadata and lifecycle checkpoints.
+It does not implement Case Interpretation, hypotheses or Investigation Tools. The
+existing CaseBuilder remains deterministic and does not start an LLM request on its own.
+
+## Configuration
+
+Copy `config/llm.example.toml` to `config/llm.local.toml` (ignored by Git). Provider array
+order is selection order; disabled providers are skipped. URL, model and adapter are
+independent. The bundled `openai_compatible` adapter uses a base URL ending in `/v1`
+and appends `/chat/completions`. An own server or Colab-hosted service exposing that
+protocol needs only another configuration entry. Other protocols can implement the
+async `LLMProvider` interface and register an adapter with `LLMGateway(..., adapters=...)`.
+There is no provider-specific behavior in the runner or case-processing modules.
+
+Keys are referenced by `api_key_env`; actual values come from process environment
+variables. Inline credentials and URL credentials are not supported. Neither resolved
+keys, response error bodies nor endpoint URLs are included in logs/checkpoints.
+Nonlocal HTTP requires explicit `allow_insecure_http = true`; HTTPS verifies certificates.
+Redirects are rejected and ambient proxy settings are not inherited.
+
+`privacy_profile` determines the allowed destination classes:
+
+| Profile | Eligible providers |
+| --- | --- |
+| `local_only` (default) | loopback/local |
+| `self_hosted` | local and explicitly trusted self-hosted |
+| `cloud_allowed` | local, self-hosted and cloud |
+
+The gateway preserves configured order within the eligible set. A request may narrow
+that policy but cannot broaden it. PromptBuilder defaults to the configured policy.
+To enable cloud failover, enable the cloud entry and set `privacy_profile = "cloud_allowed"`.
+Missing credentials or having no eligible provider is a configuration error.
+
+`response_format` may be `json_schema`, `json_object` or `none`, accommodating different
+server capabilities. JSON is always required and validated locally against the caller's
+schema. `token_parameter` selects `max_completion_tokens` or `max_tokens`. No unsupported
+parameter retry or hidden downgrade occurs. The Chat Completions transport follows the
+[official OpenAI API reference](https://developers.openai.com/api/reference/resources/chat/subresources/completions/methods/create).
+
+## Failover and retry semantics
+
+One attempt is one complete round through eligible providers. Each round starts with
+the first provider, immediately proceeds to the next on connection failures, timeout,
+HTTP 408/429 or HTTP 5xx, and stops on the first validated success. Authentication errors,
+other HTTP configuration failures, refusals, truncated model output, invalid JSON and
+schema violations are controlled errors; they are not logged as unreachable servers.
+Adapters execute one call and never retry independently.
+
+Operational defaults:
+
+| Setting | Default |
+| --- | --- |
+| `max_attempts` | 10 full rounds |
+| `retry_delay_seconds` | 60 between failed rounds |
+| `blocked_retry_seconds` | 3600 before an explicit restart of an exhausted operation |
+| provider `timeout_seconds` | 30 |
+| `max_request_bytes` / `max_response_bytes` | 1 MiB each |
+
+These are configurable safety defaults, not immutable package-format constraints.
+For N eligible providers, an unsuccessful operation makes at most 10 × N calls,
+excluding a possible replay of the last unconfirmed round after a crash.
+
+`LLMGateway.complete_round()` is an async transport operation. `CaseLLMRunner.run()`
+executes at most one due round per dispatch. After an unsuccessful round it persists
+`RETRY_PENDING` with a due timestamp and returns immediately. A scheduler can dispatch
+other cases while waiting and invoke the same operation again when due. Calling early
+returns `DEFERRED`, with no request and no new package generation. This milestone adds
+the runner API, not a continuously running worker daemon or a domain-phase controller.
+
+After the configured number of rounds, the runner commits `BLOCKED`, reason
+`LLM_UNAVAILABLE`, destination `inbox`, a retry timestamp and `automatic_retry: false`.
+The managed package remains in cases; routing is metadata. The InboxProcessor respects
+that flag, including after a blocked package is received by another service. An exhausted
+operation does not silently start another ten-round campaign. An explicit restart needs
+resumption through CheckpointStore after the due date and a new stable operation_id.
+Other BLOCKED cases retain their existing automatic retry behavior.
+
+## Durable state and integration
+
+Each operation_id is stable within an analysis revision. Checkpoints contain request
+and configuration fingerprints, round count, phase, due time and immutable artifact
+references. A changed request/configuration under the same operation_id is rejected;
+use a new operation_id for deliberately changed configuration or input.
+
+Under `llm/operations/<hashed revision and operation id>/`, the package contains the
+exact credential-free request and one immutable outcome file per completed round.
+Outcome files contain provider/model/prompt version, available token usage, elapsed
+time and earlier failed providers in that round. Optional configured input/output rates
+produce an estimated cost in `cost_currency`; missing usage/rates yield null, never a
+guessed zero. Failed/unconfirmed calls may incur costs that cannot be measured locally.
+Persisted files are covered by package integrity. Generated LLM artifacts do not become
+raw CaseBuilder inputs. The original package/files and package_id remain unchanged;
+package_generation increases per checkpoint, analysis_revision does not.
+
+A confirmed success or controlled error is reused after restart without another model
+call. Dispatches for the same case use a nonblocking operation lock; they report LockBusy
+instead of waiting. The external request never holds the global worker lock. Concurrent
+cases remain independent. Checkpoint commit uses the expected generation to reject stale
+results prepared while another component modified the case. A crash may repeat the last
+unconfirmed remote call; remote execution and local ZIP publication cannot be atomic.
+
+The runner records failure logs in `runtime/logs/llm.jsonl` (rotating at 1 MiB with three
+backups). An unreachable-server failure during BUILDING_CASE explicitly identifies
+Casebuilding; another phase is logged by its actual name. The gateway logs only provider
+IDs and safe error codes, not prompts, raw provider error bodies or credentials.
+
+The generic versioned prompt `structured_json_v1.txt` prepares JSON transport and treats
+input as untrusted data. It does not interpret a case or provide tools. Caller-supplied
+output schemas cannot resolve external references; local JSON Schema and timestamp
+validation runs before accepting any output.
+
+The stdlib HTTP adapter runs blocking I/O in an executor. Async deadlines release the
+awaiting case; cancellation cannot forcibly terminate an OS DNS lookup or already
+running remote inference. A late result is discarded. Providers must respect transport
+timeouts; a future native async adapter can implement the same interface without
+changing configuration/lifecycle contracts.
+
+Example integration after an active phase has been established:
+
+```python
+config = GatewayConfig.load("config/llm.local.toml")
+gateway = LLMGateway(config)
+runner = CaseLLMRunner(store, gateway)
+request = PromptBuilder().build(
+    task="Transport test",
+    data={"observation": "A supplied observation"},
+    output_schema={"type": "object", "properties": {"answer": {"type": "string"}},
+                   "required": ["answer"], "additionalProperties": False},
+)
+result = await runner.run(case_id, package_id, operation_id="transport-test-v1", request=request)
+# SUCCESS / RETRY_PENDING / DEFERRED / BLOCKED / ERROR
+```
+
+Tests use deterministic fake adapters and an actual local HTTP server. No cloud
+credentials, paid calls or real models are required. The next domain milestone is
+CaseInterpreter, whose versioned prompt and output schema will use this gateway.