Add bounded contextual entity resolution before interpretation
This commit is contained in:
1 parent
87581e5cd2
commit
4fa7fb6e74
21 files changed
+1166
-14
No files matched your search
@@ -0,0 +1,124 @@
|
||||
# Contextual entity resolution
|
||||
|
||||
Entity resolution belongs to **Case Interpretation**, the first semantic phase.
|
||||
The deterministic CaseBuilder and its completed `case.json` remain unchanged.
|
||||
This step identifies entities needed to understand a report; it does not inspect
|
||||
spawn availability, generate hypotheses, diagnose a cause or propose changes.
|
||||
|
||||
## Sequence
|
||||
|
||||
1. A versioned model prompt proposes grounded name searches, with contextual
|
||||
roles and references to readable case sources. Search terms must occur in
|
||||
the cited text; invented names, unknown catalogs and unread references fail
|
||||
validation. The model cannot choose tables, columns, credentials or endpoints.
|
||||
2. The controller validates the entire plan. The existing read-only project
|
||||
research service executes parameterized, literal `contains` name queries
|
||||
against operator-configured catalogs. Each result is captured in the package,
|
||||
including its exact query, database snapshot binding and coverage limits.
|
||||
3. A separate model assessment compares the returned candidates with the
|
||||
original context. It returns `selected`, `ambiguous` or `no_match` for each
|
||||
search. A selection must refer to an actual candidate from that search and
|
||||
cite both its capture and the original mention. Truncated searches cannot
|
||||
produce a unique selection. Similar names alone are insufficient.
|
||||
4. The controller adds captured lookup facts and selected entities to the frozen
|
||||
interpretation context. The final interpretation uses the resolved prompt.
|
||||
Selected entities must be retained with mention citations; unresolved
|
||||
identities must be retained as unknowns. These constraints participate in the
|
||||
existing interpretation correction loop.
|
||||
|
||||
Normally this adds **two model calls** before the final interpretation. It is a
|
||||
quality feature, not a token reduction. Invalid replies and provider retries can
|
||||
add calls, within the persisted policies. No API request is needed to replay
|
||||
completed resolution and interpretation artifacts.
|
||||
|
||||
## Configuration
|
||||
|
||||
`config/project.example.toml` demonstrates the optional `[entity_resolution]`
|
||||
block. It is disabled when absent. The current Vcore example enables creature
|
||||
name and localized creature name catalogs; the core mechanism is independent of
|
||||
WoW and supports other entity types and database tables.
|
||||
|
||||
- `max_searches`: maximum proposed searches (default 4).
|
||||
- `max_candidates`: maximum rows per search (default 8).
|
||||
- `max_invalid_responses`: maximum invalid replies **per semantic stage**
|
||||
(default 3).
|
||||
- `catalogs`: operator-selected entity type, table, identity column, name column
|
||||
and returned data columns. Tables must also be in the project DB allowlist.
|
||||
|
||||
These are configurable operational safety defaults, not package format limits.
|
||||
No arbitrary SQL, regex, joins, model-supplied table names, network browsing,
|
||||
runtime spawn counts or unlimited research loop is exposed here. Additional
|
||||
catalogs can cover localized names or other entity domains. A miss is scoped to
|
||||
the chosen catalog, name filter and configured snapshot; it never proves that
|
||||
an entity is absent from the runtime world.
|
||||
|
||||
## Provenance and lifecycle
|
||||
|
||||
Artifacts live below
|
||||
`analysis/interpretations/NNNN/entity-resolution/`: frozen input and policy,
|
||||
accepted plan, content-addressed query captures, candidate index, assessment,
|
||||
and attempt-specific rejected response records. The usual LLM artifacts retain
|
||||
provider, model, token counts, timing and validated requests/responses for both
|
||||
semantic stages. Every accepted stage is checkpointed.
|
||||
|
||||
Existing canonical source aliases stay unchanged. New `F` aliases expose captured
|
||||
technical lookup results; new `N` aliases expose selected entities with exact
|
||||
capture row provenance, mention references and an assessment reference. New `D`
|
||||
aliases expose unresolved identities. Their `/entity_resolution/` pointers refer
|
||||
to these separate artifacts, not invented fields inside the canonical case.
|
||||
|
||||
A DB-backed ID/name is a technical fact. Its assigned role is explicitly labeled
|
||||
`model_contextual_selection_not_causation`: semantic association is not a proof
|
||||
of runtime behavior or of a defect. An existing quest giver remains relevant and
|
||||
can still be a contributor; it is not equated with the missing target creature.
|
||||
Not seeing a creature is not automatically proof of missing spawn records.
|
||||
|
||||
Restart reuses accepted plans, captures and assessments. A database outage blocks
|
||||
the case as `RESEARCH_UNAVAILABLE` without losing the plan. Invalid response
|
||||
exhaustion blocks as `ENTITY_RESOLUTION_INVALID_RESPONSES`. Neither changes
|
||||
hypothesis budgets or creates hypotheses. Blocked cases do not hold a worker;
|
||||
resumption is explicit, after the recorded retry timestamp (60-second safety
|
||||
delay). Exhausted response policies require review and a new analysis revision.
|
||||
Changing catalogs or snapshot bindings during an unfinished stage is rejected.
|
||||
Completed results can be verified in a different workspace without a live DB.
|
||||
|
||||
Existing completed/started interpretations are not silently enriched or rewritten.
|
||||
They retain their original prompts and contexts; use a fresh case or a new
|
||||
analysis revision to test this feature. `--stop-after interpretation` includes
|
||||
resolution and still prevents hypothesis generation and investigation.
|
||||
|
||||
## Tests
|
||||
|
||||
Offline tests cover unobserved affected subjects, competing candidates, unknowns,
|
||||
grounding, limits, invalid response corrections/exhaustion, scoped empty results,
|
||||
DB outages, provider retries, changed bindings, append-only preservation,
|
||||
portable replay and the trial phase stop.
|
||||
|
||||
Opt-in tests use `VBA_TEST_PROJECT_CONFIG` for the local SELECT-only snapshot.
|
||||
With `VBA_TEST_CASE_PACKAGE` pointing to a built but not yet interpreted package,
|
||||
an additional end-to-end test uses the real case and database, with **fixture
|
||||
model replies**. It proves query/provenance/lifecycle integration, not the quality
|
||||
of a live model's semantic decisions. For the known Wolpertinger example, the
|
||||
fixture proposes the name found in the quest text; the real DB supplies the ID.
|
||||
The implementation contains no case-specific Creature-ID or diagnosis.
|
||||
|
||||
## Prepared development trial
|
||||
|
||||
A fresh copy of BUG-000020 is prepared in
|
||||
`service-data/trials/entity-resolution-20261015`, package ID
|
||||
`trial-entity-resolution-BUG-000020-20261015`. Only import, admission and
|
||||
deterministic case construction have run (generation 4, no model calls).
|
||||
|
||||
Run from the repository root in the PowerShell session containing the API key:
|
||||
|
||||
```powershell
|
||||
.\.venv\Scripts\python.exe -m viano_bug_analysis.trial run --workspace service-data/trials/entity-resolution-20261015 --llm-config config/llm.timeout300.local.toml --project-config config/project.local.toml --case-id BUG-000020 --package-id trial-entity-resolution-BUG-000020-20261015 --stop-after interpretation --max-dispatches 6
|
||||
```
|
||||
|
||||
This makes billable model requests but stops before hypotheses/investigation.
|
||||
If a provider retry is pending, repeat the same command after its timestamp.
|
||||
After completion, inspect the resolution plan, captured candidates, assessment
|
||||
and final interpretation together. Success means the model itself proposes a
|
||||
context-grounded search, distinguishes Goldark from the affected creature, and
|
||||
retains the selected identity and its provenance. A fixture test alone does not
|
||||
establish this live-model quality.
|
||||
Reference in new issue
Block a user