How do you route AI models by cost, quality and latency?

Key facts

Routing inputs	Task type, input length, user tier and validation result
Latency metrics	Time to first token for streaming plus total completion time
Quality gates	Schemas, tests, citations and judge rubrics
Cost metric	Cost per successful task, not cost per request
Catalogue	30+ models behind one OpenAI-compatible endpoint
Policy	Keep routing rules in configuration and version them
Free plan	plugsky-micro and plugsky-lite, no card required
Product status	Live

TL;DR

• Score the request first, then choose the cheapest model that passes.
• Rules cover most routing; add a classifier only when rules cannot decide.
• Latency means time to first token for streaming, not one average.
• Escalation needs an objective trigger and a hard cap.
• Review routing outcomes monthly — untested routers rot.

How it works, step by step

1. Define quality gates and latency budgets per workload before choosing models.
2. Collect routing signals you already have: task type, input length, tenant tier, validator output.
3. Start with explicit rules and a small model ladder for each workload.
4. Add a classifier only for the requests your rules cannot separate.
5. Shadow-test the router against the current single-model path before switching traffic.
6. Track escalation rate, latency percentiles and cost per successful task.
7. Re-tune thresholds after model-card changes and review them on a schedule.

Try it yourself

Open the AI workload router simulator →

Signals you can score

Three dimensions decide the route: cost, quality and latency. Each is estimated from signals available before the request runs, then corrected by what happens after.

• Difficulty: task type, input length, number of retrieved passages, structured-output requirement.
• Quality bar: whether a wrong answer is embarrassing, recoverable or costly.
• Latency budget: interactive chat, autocomplete and batch jobs have very different limits.
• Outcome signals: validation result, retry count, user feedback — the data that tunes thresholds.

Implementing the router

Begin with a ladder, not a matrix: define two or three tiers per workload and move up only on failure. Rules such as input length or task type handle most traffic; a small classifier earns its place when easy and hard requests look identical on the surface.

Keep the router stateless and the policy in configuration. Every path needs a fallback — if the chosen model errors or times out, retry on the next tier before surfacing a failure. Fast first hops such as plugsky-lite keep interactive paths responsive, while stronger tiers handle escalations.

Testing and tuning

Run the router in shadow mode against your current path and compare quality, latency and cost on the same traffic. Only then switch real users. Once live, watch the escalation rate: a sudden rise usually means a prompt or validator change, not a model problem.

Report latency as percentiles — p50 hides the p95 that users complain about — and keep time to first token separate from total completion time. Routing is a living policy; schedule a monthly review, and re-run your evaluation set whenever a model card changes. See model routing for the wider implementation patterns.

Honest comparison

Routing signal	How to detect	Cheap action	Strong action
Short, simple request	Input length and task type	Free or small tier	Escalate only on validation failure
Structured output	Schema requirement	Small tier with JSON mode	Stronger tier on schema failure
Long input	Token count above threshold	Retrieval instead of full context	Long-context tier
Low-confidence answer	Validator or judge score	Retry once on the same tier	Escalate a tier
High-stakes request	Workload or tenant flag	Draft on the cheap tier	Top tier plus human review