The standard advice when building an LLM-powered product is: pick a model, stick with it, tune your prompts. That advice made sense in 2023. In 2025, running a single provider in production at any meaningful scale is a mistake — not because the models aren't good enough, but because different tasks have different price/performance profiles, and you're leaving a lot on the table by treating them the same.
We run three models in production. Here's the decision behind each one.
Groq Llama 3.3: the workhorse
Llama 3.3 70B on Groq handles the bulk of the pipeline: intent classification, entity extraction, and response generation for tier-1 support. Two properties made it the default route. First, speed — Groq's inference is fast enough that we stream completions token-by-token into the chat UI and the user perceives the response as instant. Our median response time is 1.2 seconds end-to-end, and most of that budget is retrieval, not inference. Second, cost — at support-ticket volume, the per-call economics of an open-weight model on Groq are a fraction of frontier-model pricing for quality that's more than sufficient when the task is well-scoped.
The discipline that makes this work: Llama only gets tasks with clear instructions and grounded context. Classification with a fixed label set, response generation with retrieved knowledge-base passages attached. We don't ask it open-ended questions and hope.
Claude: judgment and nuance
Claude handles the tasks that require genuine reasoning under ambiguity: escalation decisions, sentiment-aware handling of frustrated users, and nuanced intent disambiguation when the classifier's confidence is low. The key property isn't raw capability — it's calibration. When a ticket is genuinely ambiguous, we want a model that says "I'm not sure whether this is a billing question or a technical question" rather than one that picks one and runs with it at 95% confidence.
The tradeoff: it's the most expensive call in our pipeline. We route to it selectively — it sees roughly one in ten requests, and only the ones where the cost of a confident wrong answer is high.
Ollama: fast, free, and deterministic
We run a local Ollama instance for everything that's high-volume, low-stakes, and deterministic. FAQ lookups, template slot-filling, response formatting, simple yes/no classification on clean input. These calls have sub-100ms latency and zero API cost.
The important constraint: Ollama only gets tasks with narrow input space and well-validated outputs. We don't trust a small local model with anything ambiguous. The routing layer enforces this — if the local model's confidence score is below threshold, the task is re-run through Llama 3.3 on Groq.
The routing layer
The gateway works like this: every LLM call goes through a central dispatcher that takes a task type, input, and latency budget. The dispatcher has a routing table — a set of rules that maps task × confidence × budget to a provider. High-volume deterministic tasks go to Ollama. Classification and response generation go to Llama 3.3 on Groq. Anything requiring judgment, or where confidence stays below threshold after two attempts, escalates to Claude.
The practical effect at production volume: the majority of calls resolve on Ollama or Groq, and only the hard tail reaches the frontier model. The infrastructure cost for the LLM layer is a fraction of what it would be if everything went through a frontier model by default.
The single-model approach is simpler to build. The multi-provider approach is significantly cheaper to run and more resilient — when one provider has an outage or a degraded API, the others absorb the load. At scale, that resilience is worth the routing complexity.