The Three-Layer Architecture Behind Prompt Optimizer's Routing

If you're building AI-powered applications, you've probably noticed your API bills climbing faster than your user growth. With frontier models like Claude Opus 4.5 ($5/$25 per 1M tokens) and GPT-5.2 Pro ($21/$168 per 1M tokens), even moderate usage can cost thousands per month.

Sending every request to a flagship model is rarely necessary — plenty of requests are simple enough for a cheaper model or don't need an LLM call at all. Here's how we built a middleware layer that classifies each request and routes it accordingly, while maintaining high accuracy in task classification. A note on cost: the routing and caching layers reduce unnecessary spend. The prompt optimization layer itself is a different story — it usually adds tokens rather than removing them, because it's adding structure the original prompt was missing. More on that below.

The Cost Problem

Let's look at a typical developer workflow:

// Common pattern: Send everything to the flagship model
const response = await anthropic.messages.create({
 model:"claude-opus-4.5",
 max_tokens: 4096,
 messages: [{
  role:"user",
  content:"Summarize this customer email..." // Simple task
 }]
});

Cost for 100 requests/day: ~$180/month

The issue? You're using a $25/M output token model for a task that Claude Haiku ($5/M) could handle equally well.

The Three-Layer Architecture

We built Prompt Optimizer API as a transparent middleware layer that sits between your application and LLM providers. It operates on three levels:

Layer 1: Intelligent Caching

The first layer identifies duplicate or near-duplicate requests:

// Prompt Optimizer API automatically detects duplicates
const cachedResponse = await cache.lookup(
 hashPrompt(userMessage, { ignoreMinorVariations: true })
);
if (cachedResponse && cachedResponse.age < MAX_CACHE_AGE) {
 return cachedResponse; // Zero cost
}

How it works:

• Semantic hashing of prompts (not just string matching)
• TTL-based invalidation for time-sensitive content
• Automatic cache warming for common patterns

Cache hit rate depends entirely on how repetitive your traffic is — FAQ-style, high-repetition workloads benefit most; novel requests won't hit cache at all. A cache hit is zero-cost by definition, but we don't publish an aggregate hit-rate number because it varies too widely by use case to be meaningful.

Layer 2: Tiered Model Routing

The core innovation is context detection. We trained a lightweight classifier that routes requests to the optimal model tier:

interface RoutingDecision {
 complexity:'simple' |'moderate' |'complex';
 recommendedModel: string;
 confidenceScore: number;
}
const decision = await classifier.analyze(prompt);
const modelMap = {
 simple:'claude-haiku-4.5',   // $1/$5 per 1M
 moderate:'claude-sonnet-4.5',  // $3/$15 per 1M
 complex:'claude-opus-4.5'    // $5/$25 per 1M
};
const response = await llm.generate({
 model: modelMap[decision.complexity],
 prompt: prompt
});

Classification criteria:

• Token count and structural complexity
• Presence of reasoning keywords ("analyze","evaluate","design")
• Code generation vs. text generation
• Domain specificity (legal, medical, general)

In a 360-prompt production sample, roughly a quarter of requests resolved on the rules tier alone — no LLM call at all — and about three-quarters used the lighter hybrid tier instead of a full frontier-model call. Actual savings depend on how much of your traffic is genuinely simple; we don't publish a blanket dollar figure because it varies by workload.

Layer 3: Prompt Optimization

For requests that must go to flagship models, we optimize the prompt itself for clarity and structure. Be clear-eyed about what this layer does to token count: in most cases it adds tokens, because it's adding structure (persona, constraints, success criteria) that a bare prompt didn't have. The example below — trimming redundant instructions — is the exception, not the rule:

// Before optimization
const verbosePrompt = `
Please analyze this code and tell me what it does.
I need you to be very detailed and thorough.
Make sure you explain every part carefully.
${codeSnippet}
`;
// After optimization (automatic)
const optimizedPrompt = `Analyze this code:
${codeSnippet}`;

Optimization techniques:

• Instruction compression: Remove redundant phrasing
• Context pruning: Strip unnecessary metadata
• Format standardization: Use efficient prompt templates
• Token-aware truncation: Smart context window management

In a 360-prompt production audit, measured with a real tokenizer, 354 of 360 prompts expanded after optimization — only 1 got smaller. Every context category expanded on average, including code generation. The exception is prompts that are already verbose or contain repeated context (like long chat histories); those are the cases where this layer can genuinely shrink token count. Don't adopt this layer expecting a token-count win — adopt it for output quality and consistency.

Where the Cost Impact Actually Comes From

We don't publish a blanket savings percentage because the real number depends entirely on your traffic: how repetitive it is (caching), how much of it is genuinely simple (tiered routing), and how verbose your baseline prompts already are (optimization). Caching and tiered routing are the layers that reliably cut cost — they either skip the call entirely or send it to a cheaper model. The optimization layer is about output quality; measured in production, it more often increases token count than decreases it.

Integration Guide

Option 1: Drop-in Replacement (Simplest)

Replace your LLM SDK initialization:

// Before
import Anthropic from'@anthropic-ai/sdk';
const anthropic = new Anthropic({
 apiKey: process.env.ANTHROPIC_API_KEY
});
// After (with Prompt Optimizer)
import { PromptOptimizer } from'@promptoptimizer/sdk';
const anthropic = new PromptOptimizer({
 apiKey: process.env.PROMPT_OPTIMIZER_KEY,
 provider:'anthropic',
 fallbackKey: process.env.ANTHROPIC_API_KEY
});
// Same API surface - zero code changes needed
const response = await anthropic.messages.create({
 model:"claude-opus-4.5", // May be downgraded automatically
 messages: [{ role:"user", content:"..." }]
});

Option 2: API Gateway Pattern (Enterprise)

Deploy as a reverse proxy:

# docker-compose.yml
services:
 prompt-optimizer:
  image: promptoptimizer/gateway:latest
  environment:
   - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
   - OPENAI_API_KEY=${OPENAI_API_KEY}
   - CACHE_BACKEND=redis
   - CACHE_TTL=3600
  ports:
   -"8080:8080"
 redis:
  image: redis:7-alpine
  volumes:
   - cache-data:/data
volumes:
 cache-data:

Monitoring and Observability

The system exposes metrics for routing and cache tracking — figures below are illustrative of the shape, not a specific customer's results:

// Built-in analytics
const stats = await optimizer.getStats();
console.log(stats);
/*
{
 totalRequests: 10000,
 cacheHitRate: 0.12,
 routingBreakdown: {
  rules: 0.26,   // no LLM call
  hybrid: 0.73,  // rules + targeted LLM call
  fullLlm: 0.01  // full frontier-model rewrite
 }
}
*/

Conclusion

Treating LLM API routing as a systems problem rather than a prompt engineering problem gets you:

• Zero-cost caching for duplicate requests
• Fewer full frontier-model calls, via rules and hybrid tiers
• High-accuracy task classification
• <20ms latency overhead