Flagship AI System · Regulated Production Design

Retrieval & Ranking

Adaptive Retrieval & Specialist Routing

Confidence-gated routing for overlapping retrieval paths with a safe default

Query RoutingBM25Semantic RetrievalEvaluation

RoleSenior AI/ML Engineer

DomainEnterprise AI / Fintech

StatusProduction Design

ContextEnterprise AI Systems

EXECUTIVE BRIEF :: AT A GLANCE

01.PROBLEM

Overlapping backend capabilities and imbalanced traffic make forced routing unreliable.

02.CORE INSIGHT

A specialist route should be a high-confidence override, not a mandatory classification.

03.STRATEGY

Combine deterministic, lexical, and semantic evidence — then fall back safely when confidence is weak.

SYSTEM LIMITATIONS

Why Not a Standard Multi-Class Classifier?

The available data was not shaped like a clean textbook classification problem. In production, three fundamental properties made naive multi-class routing brittle:

01. Structural Overlap

Overlapping Capabilities

Multiple backend paths could sometimes answer the same request reasonably well. Some labels were not mutually exclusive in the strict statistical sense, making forced single-class routing artificial.

Fuzzy domain boundaries

02. Class Distribution

Imbalanced Traffic

A small number of broad routes accounted for most examples, while specialist routes represented a smaller long-tail. Global accuracy becomes misleading when dominant routes mask minority specialist failures.

Long-tail specialist volume

03. Operational SLA

Latency Sensitivity

Routing happens before the useful answer begins. Adding expensive inference just to resolve every ambiguous edge case hurts the user experience more than routing conservatively to a broad default.

Critical request path

SYSTEM ARCHITECTURE

Precision-First Routing Cascade

The architecture combines deterministic signals, BM25 lexical matching, and dense semantic embeddings into a low-latency cascade. Ambiguity is handled safely by the default path, while specialist routes require verifiable evidence.

ARCHITECTURE :: ADAPTIVE ROUTING CASCADE

CONFIDENCE-GATED

Per-Route PrecisionReliable Overrides

Macro F1Class Balance

Confusion MatrixLeakage Analysis

Wrong-Specialist RateError Minimization

1. Inexpensive Normalization

Extracts structured identifiers, regex patterns, and canonical entity tokens upfront.

2. Evidence Fusion

Combines exact BM25 keyword overlap with semantic similarity to route intent embeddings.

3. Confidence Gating

Overrides the broad default path only when absolute confidence and route separation are proven.

SYSTEM DESIGN PRINCIPLES

Key Engineering Decisions

Architecting for asymmetric misrouting penalties led to three foundational design rules:

Deterministic Before Probabilistic

Zero-Model Fast Path

Some requests contain patterns that can be recognized reliably without invoking a semantic model. Those queries should take the inexpensive path first.

Examples include strongly structured expressions, known query forms, or terminology with clear routing semantics. This gives the system two distinct advantages:

Lower routing overhead for easy, high-frequency deterministic cases;
Predictable behavior where semantic reasoning adds little value.

Semantic routing is reserved exclusively for requests where natural-language interpretation is actually required.

Specialist Precision Over Specialist Recall

Asymmetric Penalty Control

A specialist route should only be selected when the evidence is overwhelmingly clear. Conceptually:

HIGH CONFIDENCE + SEPARATION→ SPECIALIST OVERRIDE

WEAK EVIDENCE OR AMBIGUITY→ SAFE DEFAULT PATH

This intentionally sacrifices some specialist recall. A query that could have been handled by a specialist may occasionally remain on the broad path. That is an acceptable trade-off when the broad route is safe and a wrong specialist selection carries a severe latency and accuracy penalty.

Design Principle: Be conservative about specialization and aggressive about safe fallback.

Confidence Is More Than the Highest Score

Margin Calibration

The highest route score by itself is not enough. When multiple routes exhibit high but virtually identical probabilities, the router is effectively uncertain.

SCORE SEPARATION :: CONCEPTUAL DYNAMICSILLUSTRATIVE EXAMPLE

Scenario A: Ambiguous CompetitionΔ = 0.02 (Small)

Route A (Specialist)0.84

Route B (Competing)0.82

Decision:→ SAFE DEFAULT PATH

High score alone is insufficient; overlapping probabilities indicate route uncertainty.

Scenario B: Clear SeparationΔ = 0.51 (Large)

Route A (Specialist)0.92

Route B (Competing)0.41

Decision:→ SPECIALIST ROUTE OVERRIDE

Strong absolute confidence with high margin of victory justifies specialist dispatch.

The routing decision therefore evaluates both absolute confidence and separation margin from competing routes before granting an override.

EVALUATION DESIGN

Why Accuracy Was Not Enough

CORE EVALUATION CHALLENGE

Dominant routes can make aggregate global accuracy look healthy while critical minority specialist failures remain completely hidden.

Under severe traffic imbalance, a naive baseline that always routes to the default path achieves a deceptively high global accuracy score despite a 0% success rate on specialized intents. Operational evaluation must evaluate behavior per route.

01. Reliability

Per-Route Precision

When the router selects this specific route, how trustworthy is that decision? Essential for ensuring specialist overrides are strictly authoritative.

Minimizes false overrides

02. Balance

Macro-Averaged F1

Gives every route equal mathematical weight in the aggregate score regardless of query frequency, preventing dominant classes from masking minority failures.

Equalized category scoring

03. Error Topology

Confusion Matrix Analysis

Shows exactly which adjacent route boundaries produce leakage, rather than collapsing multi-dimensional routing errors into a single scalar number.

Boundary leakage diagnosis

04. Primary Governance MetricKEY PENALTY

Wrong-Specialist-Route Rate

Tracks instances where the router overconfidently forced a narrow path when remaining on the broad safe default would have provided valid context.

Safety-critical optimization target

ARCHITECTURAL CONSTRAINT

Why Not Use an LLM as the Router?

Latency OverheadCompute CostNon-DeterminismDebug Complexity

An LLM can classify intents, but placing a heavy generative model in the critical request routing path introduces overheads that defeat the primary objectives of low-latency retrieval.

The superior architecture uses the simplest reliable mechanism first (deterministic regex → BM25 lexical → semantic embeddings) and escalates only when required. A generative model adds high value during reasoning and context synthesis, without needing to become the bottleneck router for every incoming request.

SYSTEM COMPROMISES

Engineering Trade-Offs

Architectural Decision	Primary Benefit	Engineering Cost / Compromise
Deterministic fast paths	Fast, predictable handling of clear structured patterns	Limited coverage of natural-language phrasing variations
Lexical routing signals (BM25)	Exact keyword and domain-specific terminology precision	Vulnerable to synonyms and semantic paraphrases
Semantic routing signals	Captures natural-language intent and vocabulary variation	Introduces boundary uncertainty near decision thresholds
Confidence-gated specialists	Prevents high-penalty incorrect specialist overrides	Some valid specialist requests safely remain on default
Safe default path	Guarantees robust, fault-tolerant handling of ambiguity	May not always provide maximum narrow domain depth
Per-route evaluation	Exposes long-tail failures hidden by aggregate numbers	More complex harness than tracking a single scalar accuracy

Core Trade-Off Stance:Operational safety over theoretical routing completeness

OWNERSHIP & CORE COMPETENCIES

Contribution & Technical Proof

MY DIRECT CONTRIBUTION

My work focused on the routing architecture and the reasoning behind how the system should make, calibrate, and evaluate routing decisions:

✓Structuring the routing flow around deterministic, lexical, and semantic signals
✓Designing conservative specialist-routing and fallback behavior
✓Analyzing the operational effect of overlapping routes and imbalanced data
✓Moving evaluation away from aggregate accuracy toward per-route metrics, macro evaluation, confusion analysis, and specialist-route reliability
✓Keeping the routing layer lightweight enough that it did not become a latency bottleneck

The core takeaway was recognizing that the correct abstraction is selective routing under uncertainty, rather than forced classification.

WHAT THIS DEMONSTRATES

Key engineering competencies proven by this architecture:

Query understandingHybrid lexical + semantic reasoningLow-latency system designConfidence calibrationSafe fallback architectureImbalanced classification evaluationRetrieval orchestrationProduction AI reliability