Provider APIs

What it is

A provider API is a hosted endpoint offering access to models the provider operates. You send a request with an API key, the provider runs inference on its own infrastructure, and you pay per token. The weights are not published and the model is not something you possess.

This is how most agentic systems reach their strongest capability, and there is nothing wrong with that. The architectural question is not whether to use one. It is whether your system would still function, at acceptable quality, if this particular one became unavailable, more expensive, or restricted.

What you are actually buying

It helps to separate the components, because they are frequently bundled in discussion and are not equally replaceable.

• Capability. Access to models at the frontier of what is currently possible. This is the part with no close substitute for the hardest tasks.
• Operations. Someone else runs the accelerators, handles scaling, applies updates, and absorbs demand spikes. This is worth more than it looks on a spreadsheet.
• Availability. Capacity when you need it, without procurement lead times.
• Feature velocity. Caching, structured output, tool calling, and longer contexts arrive without any work from you.
• A support relationship. Someone to escalate to when behavior changes, which matters more in production than during evaluation.

The first of these is why teams use provider APIs. The rest are why they keep using them even when an open-weight alternative would be adequate.

The capability gap, honestly

Statements about how far ahead frontier models are age badly, so it is more useful to describe the shape of the gap than its size.

For classification, extraction, formatting, and routine summarization, the gap is small or absent. Good open-weight models handle these well, and the difference between tiers is rarely the limiting factor in output quality.

For multi-step planning, for reasoning that requires holding several constraints at once, for long-context work where relevant detail is buried, and for code that must be correct rather than plausible, the gap is larger and it shows up in agentic systems more than in single-turn use. An agent that plans badly does not produce a slightly worse answer, it takes a wrong path and spends five steps on it.

This asymmetry is the practical argument for tiering rather than choosing. The calls where the gap is small are also the frequent ones, and the calls where the gap is large are the rare, expensive ones. Routing accordingly gets most of the capability for a fraction of the cost, which is a better outcome than either extreme.

Features worth using

Provider APIs offer capabilities beyond generating text, and three of them change agentic system design materially.

Native tool calling

Passing tool schemas and receiving a structured call back is far more reliable than instructing a model to produce a particular format. This is the normal path now, and building on prompt-formatted output instead is choosing a worse mechanism.

Prompt caching

Reusing a previously processed prefix at reduced cost and latency. Agentic loops resend a largely identical prefix on every step, so this matters more here than almost anywhere else. It imposes one design rule worth following regardless of provider: put stable content first and volatile content last.

Streaming

Returning tokens as they are produced. For agents this is less about perceived speed in a chat window and more about being able to detect a bad direction early and cancel, rather than paying for a complete response you will discard.

A fourth capability worth watching is long context. Larger windows let you put more in, and attention over very long inputs is uneven, so the discipline of selecting the right context rather than supplying all of it remains correct even when the window would allow otherwise.

How pricing actually behaves

Per-token pricing looks simple and behaves unintuitively in agentic systems, which is why cost surprises are so common. Three properties explain most of them.

Cost scales with steps, not with requests

A user asks one question. The agent takes eleven steps. Each step resends the accumulated context. The cost of that single question is not eleven times the first call, it is closer to the sum of a growing series, because context accumulates as the task proceeds. Estimating from a single-call price produces a number that is wrong by a large factor.

The distribution has a long tail

Most tasks resolve quickly. Some do not. A task that occasionally loops, retries, or explores a dead end costs many times the median. Because the tail is where the money is, average cost per task is a misleading planning number and the ninety-fifth percentile is the one to budget against.

Input dominates output

Intuition says generation is the expensive part. In agentic work the input side is usually larger, sometimes by an order of magnitude, because every step carries the history. This inverts the usual optimization advice: trimming context, summarizing older steps, and scoping tools per task all reduce cost far more than constraining response length does.

Two mechanisms exist specifically to blunt this, and both are worth adopting early. Prefix caching makes the repeated portion of context much cheaper, provided the stable part comes first. Batch or asynchronous processing, where a provider offers it at reduced rates, suits any workload that does not need an answer immediately, which describes more agentic work than teams initially assume.

Where lock-in actually accumulates

Teams worry about lock-in in the wrong place. The API key is not the problem, and neither is the client library. Both are trivially replaceable. Lock-in accumulates in four quieter forms.

Prompts tuned to one model.Phrasing that exploits a particular model's behavior, refined over months. Moving means re-tuning, and nobody schedules that.

Dependence on provider-specific features. Building a workflow around a capability only one provider offers means the workflow does not move even if the model call does.

Evaluation baselines that only exist implicitly. Without a recorded evaluation set, nobody can tell whether an alternative is adequate, so the alternative is never seriously considered.

Assumed capability. Designs that only work because the model is very strong. An agent given vague instructions and thirty tools works on a frontier model and falls apart on anything else. The design, not the integration, is what pins you.

Staying portable while using one

None of the above is an argument against provider APIs. It is an argument for a handful of habits that cost very little while you are staying.

1. Route every call through one internal function. The single highest-value habit, and it costs an afternoon.
2. Express model choice as named roles. Planner, extractor, classifier. Roles resolve to models in configuration.
3. Define tool schemas once, translate per provider. Rather than writing them in one vendor's dialect.
4. Structure prompts rather than tuning phrasing. Clear separation of instructions, facts, tools, and task transfers between families. Clever phrasing does not.
5. Keep an evaluation set. Twenty real tasks with known answers. Without it, portability is a theory you cannot test.
6. Run a second family occasionally. Even for a small share of traffic. Portability that is never exercised quietly stops working.
7. Record model and version per call. So that a change in behavior can be attributed rather than argued about.

Where it fits in an agentic system

The natural home for a frontier provider API in an agentic architecture is narrower than most systems actually use it for.

It belongs on planning, where a wrong decision costs several wasted steps. It belongs on final synthesis, because that is the output a person judges. It belongs on genuinely hard reasoning and on code that must be correct. It belongs wherever an error is more expensive than the token cost difference.

It does not belong on the classification call that runs forty times per task, on extraction from a well-structured document, or on deciding whether a tool result looks like an error. Those are the calls that dominate volume, and routing them to the most expensive model available is the most common source of avoidable spend in agentic systems.

There is also a reliability argument for not concentrating everything on one provider. An agent that fails at step seven loses the work of the first six. A fallback in a different model family, exercised occasionally so it is known to work, turns a provider incident into degraded quality rather than an outage.

Operational realities

• Rate limits. Agentic workloads are bursty in a way that request-response applications are not, because one user request expands into many calls. Limits are hit earlier than the request volume suggests.
• Latency variance. Tail latency matters more when calls are chained. A ninety-fifth percentile that is four times the median is a different experience across ten sequential steps than across one.
• Model deprecation. Hosted models are retired with notice. A system with the model identifier in configuration handles this in an hour; one with it scattered through code does not.
• Behavior drift under a stable name. Providers update models. A stable identifier does not always mean stable behavior, which is why pinning specific versions where offered, and re-running the evaluation set on a schedule, both matter.
• Cost attribution. Provider billing tells you what was spent. It does not tell you which feature spent it. That mapping has to come from your own per-call records.

Data handling questions to settle

These questions are best answered once, written down, and enforced in code rather than remembered.

• What is retained, and for how long. Providers differ, and enterprise terms often differ from default terms.
• Whether content is used for training. Usually not on business tiers, and worth confirming rather than assuming.
• Where processing happens. Region matters for some regulatory regimes, and not every provider offers a choice.
• What your agents can actually send. This is the one teams skip. An agent reads tool results, and tool results contain whatever the underlying system holds. The classification of data an agent may encounter is broader than the classification of data anyone deliberately sends.
• Where the enforcement lives. A policy that exists only in a document is not enforcement. The check belongs in the routing layer, before dispatch.

Choosing between providers

Provider comparisons usually turn into benchmark arguments, which is the least useful axis. A more practical comparison weighs six things, in roughly this order for agentic use.

• Performance on your own evaluation set. Twenty real tasks with known answers, scored the same way for every candidate. This dominates everything else, and it is the step most often skipped.
• Tool-calling and structured-output reliability. Agentic systems live on this. A model that reasons slightly better and adheres to schemas slightly worse is usually the wrong trade, because schema failures are loud, frequent, and require retries.
• Behavior at your real context lengths. Not the advertised maximum. The length your tasks actually reach, with material buried in the middle.
• Cost at your actual mix. Weighted by how many calls of each kind you make, and including cached input if you can use it.
• Tail latency. The ninety-fifth percentile, multiplied by your typical step count, is what a user experiences.
• Terms and region. Retention, training use, and processing location, checked against the data your agents can actually encounter rather than the data you intend to send.

Worth adding: choose a fallback from a different provider family rather than a second model from the same one. Same-family alternatives tend to share infrastructure and share incidents, so a fallback that fails alongside the primary is not a fallback.

Gotchas worth knowing

• Input tokens dominate. In agentic loops the input side is usually the larger cost, because context is resent every step. Optimizing output length is the wrong lever.
• Cache invalidation by accident. Placing anything volatile early in a prompt, such as a timestamp, defeats prefix caching entirely and silently.
• Retries multiply cost. An automatic retry on a long-context call doubles the spend for that step. Retries need limits.
• Streaming complicates tool calls. Handling partial tool call structures correctly is a common source of subtle bugs.
• Parameter defaults differ between providers. Sampling settings that were implicit in one provider become explicit differences in another, which is a frequent cause of a model appearing worse after a migration.
• Free evaluation credits distort decisions. A model that is cheap during a trial and expensive at volume should be evaluated at volume pricing.

How it scores on openness

• Replaceable. Depends entirely on your architecture rather than on the provider. With a routing abstraction, strong. Without one, weak.
• Inspectable. Weak for the model. You can observe inputs, outputs, and costs, and not what produced them.
• Portable. Partial. Prompts and tool definitions move if written for portability. The model does not.
• Bounded. Not applicable at this layer. Authority is enforced by the harness.
• Grounded. Not applicable. Facts come from the data layer.
• Auditable. Partial. You can record what was sent and returned. You cannot reproduce a past result if the model behind the name has changed.

What it is not

A provider API is not an agent. It returns text and structured tool calls. The loop, the state, the limits, and the accountability belong to the harness.

A provider API is not a policy boundary. Instructions in a system prompt describing what the model must not do are a request, not enforcement, and they fail under adversarial input.

A provider API is not inherently incompatible with an open architecture. Using proprietary capability inside a system whose data, standards, and execution layers remain open is a reasonable position. The failure is not using one. It is building so that you could not stop.