Self-Hosting LLMs vs an API Gateway: When Each Makes Sense (Podcast EP13)

Key takeaways

Count the full cost of self-hosting: data-center GPUs, one deployment per model, load balancing, failover, monitoring and the engineers who keep it running.

Self-host when regulation forbids sending data to third-party APIs, when you have very high volume on a single model, when you must serve a proprietary fine-tune, or when you need very low latency in one region.

Use an API when you need several different models, have normal traffic levels, and want new models without a deployment cycle.

Consider a hybrid: self-host one or two high-volume models such as a classifier or embedding model, and send everything else through an API.

Without a dedicated ML ops team, start with an API, find product-market fit, and only self-host the workloads whose economics clearly justify it.

Show notes

The appeal and the reality of self-hosting#

On paper, self-hosting looks ideal. Download an open-weight model such as Llama 4, DeepSeek V3 or Qwen 3, run it on your own GPU, and you get inference with no per-token bill, full control, no provider rate limits and complete data privacy.

The reality arrives with the hardware list. A model that can compete with leading commercial models needs data-center GPUs such as A100s or H100s, which are expensive to buy and, at the time of recording (April 2026), cost several dollars per hour to rent. A single rented H100 running around the clock was estimated at roughly three thousand dollars a month.

And that covers one model on one GPU. Offering separate coding, reasoning and writing models multiplies the bill, and you still have to handle updates, load balancing, failover, monitoring and the engineering time behind all of it. Those hidden costs rarely appear in a back-of-the-envelope GPU calculation, yet they often decide whether a self-hosted deployment is actually cheaper than paying per token.

When self-hosting is the right call#

There are clear cases where owning the stack makes sense:

Data sovereignty. Healthcare, finance and government teams may face rules that prohibit sending data to third-party APIs.
Extreme volume on one model. At millions of requests per day on a single model, owning hardware eventually beats paying per token. The episode put the break-even in the range of fifty to a hundred thousand requests per day, depending heavily on token lengths.
Proprietary fine-tunes. A model fine-tuned on your own data often has to be deployed by you, although hosted fine-tuning services such as Fireworks and Together are narrowing that gap.
Latency-critical apps. If you need extremely fast time-to-first-token for users in one region, co-locating the model with your application removes network hops.

Why most teams are better off with an API#

For the majority of developers and startups, an API gateway is the more practical choice, for four reasons.

Model diversity. No single self-hosted model beats every commercial model at every task. At the time of recording, the episode highlighted GPT-5 for instruction-following, Claude Opus 4 for long-context reasoning and Gemini 3 Pro for multimodal work, with other models stronger at math and code. A gateway gives you all of them at once, and you can pick per task instead of forcing one model to do everything.

Cost at normal scale. At moderate traffic, per-token spend is typically a small fraction of the fixed monthly cost of even one rented GPU, so the API stays cheaper until you reach large, single-model volume.

No ops burden. No GPU monitoring, driver updates or out-of-memory crashes at 3 AM; your team works on the product.

Fast access to new models. API users typically get new releases within hours, while self-hosters wait for the open-source community to catch up and then run their own deployment cycle. In a field that moves this quickly, being a week behind can cost you a competitive edge.

The hybrid approach#

The strongest teams often combine both. They self-host one or two fine-tuned models for their core, high-volume task, such as classification or embeddings, and send everything else - complex reasoning, content generation, code assistance, multimodal work - to an API.

This captures the savings of self-hosting where volume is concentrated while keeping the flexibility of many models elsewhere. Because OpenAI-compatible gateways such as Crazyrouter expose the same interface your self-hosted model can, your application talks to both in one way and you mix and match by model name. If a self-hosted workload later stops paying for itself, you can move it back to the API without rewriting the calling code.

A four-question decision framework#

Do regulations prohibit external APIs? If yes, self-host.
Do you make more than about a hundred thousand requests per day to a single model? If yes, run a real cost comparison for self-hosting that specific model.
Do you need multiple models for different tasks? If yes, use a gateway; self-hosting five models is a heavy operational load.
Do you have a dedicated ML ops team? If not, do not self-host. Startups regularly lose months to GPU drivers and memory tuning instead of building their product.

For most developers the sequence is: start on an API, validate the product, find product-market fit, then selectively move your highest-volume workloads in-house once the numbers justify it. No single answer fits everyone, but working through these four questions honestly usually makes the right path obvious, and it keeps you from committing GPU budget and engineering months before you know which workloads will actually be large.

Questions from this episode

Is it cheaper to self-host an LLM or use an API?

At normal traffic levels an API is usually cheaper, because you pay only per token while a rented data-center GPU costs thousands of dollars a month before engineering time. Self-hosting starts to pay off only at very high, sustained volume on a single model. Run the comparison for your specific model and token lengths before switching.

When should a company self-host its own AI model?

Self-host when regulations forbid sending data to third parties, when one model handles very high daily volume, when you need to serve a proprietary fine-tuned model, or when you need very low latency in a single region. Outside those cases, the operational burden usually outweighs the savings.

Can I use self-hosted models and API models together?

Yes, and many teams do. A common hybrid self-hosts one or two high-volume models, such as a classifier or embedding model, and routes reasoning, generation and multimodal tasks to an API. Using OpenAI-compatible interfaces on both sides keeps your application code the same.

Self-Hosting vs API Gateway — The Real Cost of Running Your Own AI Models