AI Picker
Cost · Research Note

Evaluating the real API cost of a switching LLM harness before scaling

A model-switching harness is sold on a promise: route each task to the cheapest model that can handle it, save money. That promise is half-true. What it leaves out is that switching itself is a cost center — duplicate context sends, retry billing, prompt maint

AI Picker rule: Core judgment, caveats, internal links, and fact boundaries all survived the style pass; no forbidden topics or unverified claims introduced.

Core judgment

A model-switching harness is sold on a promise: route each task to the cheapest model that can handle it, save money. That promise is half-true. What it leaves out is that switching itself is a cost center — duplicate context sends, retry billing, prompt maintenance labor, proxy latency. Those costs can exceed the per-token savings the harness advertises on its homepage.

The cheapest model is not the cheapest workflow. Before wiring a switching layer into production, measure what switching actually costs — not what each model costs on its pricing page.

Concrete scenario

A five-person engineering team picks a harness that routes between a cheaper model for boilerplate and a stronger model for complex reasoning. The harness homepage shows a comparison: "50–70% savings vs. using Model A for everything." They sign up.

Week one looks great. Per-token costs are low. The cheaper model handles most requests.

By week two, the invoice is higher than their old single-model setup. Three things happened that nobody budgeted for:

  1. Context re-sends. Every time the harness escalated a task from the cheap model to the strong model mid-session, it re-sent the full conversation history to the new provider. Input-token charges stacked up. On a long conversation switched twice, they paid for the same context three times.
  1. Retry billing. When the cheap model returned a bad result or hit a rate limit, the harness retried — sometimes on the strong model, sometimes on the same model. Every retry was a billable API call, including the ones that produced no usable output.
  1. Prompt maintenance. Prompts tuned for one model degraded on the other. The team spent hours writing and maintaining parallel prompt variants. That labor cost never appeared on the API invoice, but it showed up in sprint velocity.

None of these costs were visible on the harness pricing page. They were all visible on the invoice.

Non-obvious insight

Most cost comparisons for multi-model setups compare per-token prices across models. That is the comparison the harness wants you to make, because it is the comparison it always wins.

The comparison that matters is total billed tokens per completed task, including re-sends, retries, and failed calls. A task that costs 2,000 tokens on a single strong model might cost 6,000 tokens on a switching setup — 2,000 on the cheap model (failed or rejected), 2,000 re-sent context to the strong model, 2,000 for the strong model's response. The cheap model was cheaper per token. The workflow was more expensive.

The harness landing page will not show you this number. It is a number the harness produces, not a number the provider charges.

The decision

You are deciding whether to commit to a tool or workflow that lets you swap between LLM models — and whether to pay for it, upgrade it, or cap spend before usage grows.

This is a cost-lane note. The question is not which harness is best. It is: before you wire a model-switching layer into production or daily team use, what costs will show up on the invoice that the harness landing page does not mention?

Evidence to verify

This note does not cite specific prices, rate limits, or model benchmarks. Those shift too fast and must be checked against official pricing pages at the time you evaluate.

Before committing to any harness, verify these from official sources:

  1. Per-token pricing for every model you plan to switch between. Check each provider's official pricing page. Do not rely on the harness's advertised savings percentage.
  2. Whether the harness adds its own markup, proxy fee, or subscription tier. Some routing layers charge on top of provider API costs. Find this in their pricing or docs, not their homepage headline.
  3. Context window differences across models. A prompt sized for one model's context limit may get truncated, rejected, or charged differently on another. Check official model docs for each.
  4. Tokenizer differences. The same text can consume different token counts on different models. This directly affects cost. Check each provider's tokenization docs.
  5. Rate limits and concurrency caps per provider plan. A harness that fans out requests across models still hits each provider's limits independently. Check official rate-limit docs.
  6. Retry and fallback behavior. When a model fails or times out, does the harness retry on the same model, switch models, or bill you for the failed attempt? Check the harness's own docs.

Market voice (community discussion) suggests people routinely underestimate items 3–6. Treat that as a prompt to investigate, not as proof.

The four costs to measure in a pilot

Before scaling any model-switching setup, run a two-week trial and log these:

1. Duplicate context cost. Every model switch mid-task re-sends the conversation or system prompt to the new model. Each re-send is a full input-token charge. If your workflow switches frequently within a session, context re-sends can dominate the bill — sometimes exceeding what you saved by picking a cheaper model.

2. Retry tax. Model failures, rate-limit hits, and timeouts trigger retries. If the harness retries on a more expensive model, or retries the same large prompt multiple times, you pay for every attempt — including failed ones. Track the ratio of successful responses to total API calls.

3. Prompt drift cost. Prompts that work well on one model often need adjustment for another. Maintaining multiple prompt variants is labor, and degraded output quality when a prompt optimized for Model A runs unchanged on Model B means more retries, more human review, more editing.

4. Switching overhead in latency. Routing decisions add latency. Model switching mid-session adds more. If the harness routes through its own proxy, that is an additional network hop on every call. For interactive use, latency is a cost measured in user time and abandoned sessions.

Log total tokens billed, total API calls (including retries), total failed calls, and hours spent adjusting prompts. Compare against using a single well-chosen model for the same period. If the switching setup is not clearly cheaper after that comparison, the switching layer is a cost, not a savings.

So what

Almost any setup can switch models with a config change. Whether the switching layer earns its keep after you count the costs it creates is a different question.

If you make fewer than a few hundred API calls per week, a switching layer is almost never worth it — fixed costs and switching overhead will dominate. Pick the best single model for your task.

At higher volume, the switching layer has to beat a single-model baseline by a wide enough margin to justify prompt-maintenance labor and the duplicate-context tax. A two-week pilot with full billing logs is the only way to know. If the harness vendor cannot or will not help you measure duplicate-context cost and retry rate, that silence is itself a signal.

Demand the comparison the harness does not volunteer: total billed tokens per completed task.

When I would not trust this

  • I would not trust this note if the harness I am evaluating publishes transparent, audited cost reports from real customers showing net savings after retries, context re-sends, and prompt maintenance. If that evidence exists and is verifiable, the switching layer may genuinely save money and this checklist's skepticism is overstated.
  • I would not use this checklist for a single-model setup. If you are not switching models, duplicate-context and prompt-drift costs do not apply. Use a standard API cost evaluation instead.
  • The two-week pilot rule breaks down at very low volume. If you make fewer than a few hundred API calls per week, fixed costs (seats, subscriptions) dominate and per-token switching overhead will not be measurable. Pick the model that works and revisit when volume grows.
  • I would not treat market voice from a single low-engagement thread as proof of anything. The HN thread that prompted this note is a question, not evidence. It tells me the topic is worth a checklist, not that any specific harness is good or bad.