Precision-first reuse of previous answers and retrieval results
Repeated and semantically similar requests create unnecessary retrieval and model work, but reusing the wrong answer can be worse than doing the work again.
Semantic similarity proposes candidates. Entity, time, scope, and freshness constraints determine whether reuse is safe.
Check deterministic exact reuse first, then retrieve semantically similar prior results and progressively filter them before deciding how — or whether — they can be reused.
Nearest-neighbor semantic search measures natural language proximity, but has zero inherent comprehension of corporate boundaries, dates, or freshness:
Identical semantic phrasing (e.g., 'What was Q3 EBITDA margin?') refers to completely different organizations or tickers. Semantic search alone cannot distinguish company ownership.
A perfectly accurate historical answer for FY2023 is factually incorrect when the user requests FY2024 results, despite 98% vector semantic similarity between the two query strings.
Related intelligence from an external or confidential folder cannot be reused in an unprivileged or public session. Similarity ignores tenant, folder, and document access boundaries.
When an underlying financial report or policy document is revised, previously cached answers become stale. Vector proximity remains high even when underlying data changes.
A semantic cache cannot rely solely on embedding proximity. Candidate prior results pass through a progressive filtering funnel that eliminates mismatched entities, periods, and stale data:
Historical queries, retrieved evidence passages & cached answer artifacts
OpenSearch BM25 keyword matching combined with dense vector retrieval
Strict entity resolution; eliminates cross-company and ticker mismatch
Verifies fiscal year, quarter, and reporting period boundaries
Enforces collection, folder, and tenant authorization constraints
Invalidates stale entries against document updates and market clock
High-confidence intelligence ready for direct answer, context, or subtask reuse
The architecture establishes an exact O(1) hash check upfront, cascading into OpenSearch hybrid retrieval and multi-constraint evaluation when deterministic misses occur:
A raw incoming question is decomposed into multiple parallel representations before any lookup or retrieval occurs:
Deterministic normalized hash (lowercased, entity-canonicalized, whitespace-stripped) for O(1) exact cache matching.
Tokenized keywords, extracted financial identifiers, acronyms, and product terms for exact term matching in OpenSearch.
Dense semantic embedding representing high-level user intent and conceptual inquiry for approximate nearest neighbor retrieval.
Extracted named entities (company, ticker), temporal intervals (FY/quarter, calendar dates), metric types, and scope boundaries.
Neither exact hashing nor semantic similarity is sufficient on its own. The system layers them sequentially to balance instant execution speed with linguistic flexibility.
Brittle under natural-language phrasing variations and minor syntactic reorderings.
Produces probabilistic candidates that require rigorous multi-constraint validation.
In enterprise and financial systems, returning the right metric for the wrong company is a catastrophic failure. Semantic similarity alone cannot guarantee entity boundaries.
The first design rule makes entity resolution a hard gate: if the requested entity does not match the candidate’s entity metadata exactly, reuse is rejected immediately — regardless of how high the vector similarity score may be.
A semantically identical question asked today may require a completely different answer if underlying financial filings, market prices, or enterprise policies have changed.
The second design rule integrates document modification timestamps and TTL invalidation clocks directly into candidate acceptance. Stale intelligence is purged before prompt assembly.
A prior result does not have to be an exact answer replacement to provide immense value. Reusable artifacts can act as direct answers, supporting evidence context, or subtask accelerators.
Natural Language Inference (NLI) provides rigorous semantic entailment and contradiction detection, but introduces neural inference latency. The system balances deep checking against fast metadata filtering:
Handles the vast majority of production requests through deterministic metadata gating and calibrated embedding thresholds.
Invoked selectively on high-stakes, ambiguous queries to verify that the cached answer logically entails the new query premise without contradiction.
Cross-encoder inference adds latency; unsuitable as an unconditional fast path.
A matched candidate is not simply an all-or-nothing cache hit. The system decides the exact mechanism by which prior intelligence serves the active request:
Returned directly to the user with zero model invocation when entity, period, intent, and freshness match completely.
Passed into the active generation prompt as authoritative context passages, reducing retrieval workload for the LLM.
Supplied to an orchestrator agent solving a multi-step query, satisfying one subtask without re-executing its tool call.
Rejected due to entity mismatch, stale document version, or boundary violation; triggers fresh grounded pipeline.
In complex agentic reasoning pipelines, reuse is not limited to answering the top-level user prompt in one shot. The cache satisfies intermediate subproblems without re-executing tool calls:
Extract Company X FY24 Margin
Fetch Company Y Fresh Filing
Historical Peer Average
Optimizing solely for cache hit rate encourages unsafe loose matching. In production systems, evaluation prioritizes precision, entity fidelity, and temporal correctness:
When the system accepts a candidate for reuse, how frequently was that decision completely appropriate? Ensures users never receive false-match shortcuts.
Strictly tracks whether reused intelligence ever crossed a corporate, ticker, or tenant boundary. In financial AI, serving Company A's data for Company B's question is a critical failure.
Measures whether candidates that have undergone source file updates, revised earnings statements, or expired TTL clocks are properly invalidated.
Retrieval Quality: Precision@K, Recall@K, MRR, nDCG on candidate retrieval.
Avoided Compute: Redundant database scans avoided, model calls bypassed, tokens saved, latency reduction.
A caching layer must never introduce a single point of failure. The system treats the cache as an accelerant with seamless fallbacks:
| Failure Mode | System Safeguard Response | Architectural Rationale |
|---|---|---|
| ✕Wrong entity match | Reject via hard entity constraints | Company/ticker mismatch fails hard equality check; candidate is discarded immediately. |
| ✕Stale / expired result | Reject through freshness validation | TTL expiry or underlying file modification invalidates the cached entry before prompt assembly. |
| ✕Semantic false positive | Filter / validate / trigger fresh pipeline | Low score separation or NLI contradiction triggers immediate fallback to fresh retrieval. |
| ✕Over-normalization risk | Retain raw + normalized representations | Original raw query string is preserved alongside canonical tokens to prevent lost nuances. |
| ✕Reuse layer unavailable / timeout | Bypass cache → direct fresh generation | Cache outage degrades gracefully to standard retrieval without throwing user-facing errors. |
| Architectural Decision | Primary Benefit | Engineering Cost / Compromise |
|---|---|---|
| Exact canonical fast path | Instant zero-cost return for identical normalized requests | Misses conceptual paraphrases with different wording |
| Hybrid semantic retrieval (OpenSearch) | Matches complex linguistic variations and subtask patterns | Introduces vector/BM25 retrieval overhead on exact misses |
| Hard entity / temporal constraints | Completely eliminates cross-company and stale-period hallucinations | Reduces raw cache utilization by discarding out-of-boundary matches |
| Freshness & TTL validation | Guarantees responses reflect active documents and current market state | Requires invalidation lifecycle management across document updates |
| Selective NLI cross-encoder validation | Deep logical support checking and contradiction detection | Inference latency overhead; only viable on high-stakes ambiguous paths |
| Multiple operational reuse modes | Allows partial context and subtask acceleration in agent workflows | More complex orchestrator integration than a simple binary cache |
| Safe fresh fallback architecture | System never breaks when reuse confidence is weak or cache is down | Pays full retrieval/generation cost when uncertainty is detected |
My work focused on designing the precision-first retrieval and reuse framework:
The core lesson was that semantic caching is fundamentally a precision-retrieval safety problem, not an approximate memory shortcut.
Key engineering competencies proven by this architecture:
