Open-weight models

What it is

An open-weight model is one whose trained parameters have been published for download. You can fetch the files, load them into an inference engine, and run the model on hardware you control, without asking anyone.

That is the whole claim, and it is narrower than the surrounding conversation usually implies. Open weights say nothing about how the model was trained, what data was used, whether the training code is available, or what the license permits you to do with the output.

The property still matters enormously, because it is the one that determines whether a model is something you have or something you rent.

Open weights is not open source

This confusion causes more bad decisions than any other in this layer, so it is worth stating flatly.

Open source, applied to software, means the source is available under a license permitting use, modification, and redistribution, including by competitors. Applied to a model, the equivalent would mean publishing the training code, the data pipeline, and enough detail to reproduce the artifact.

Open weights means the finished artifact is downloadable. The recipe is not included. You can use the model and you cannot rebuild it, audit what went into it, or verify claims about its training.

Both are legitimate positions and they buy different things. Open weights buy operational freedom: run it where you like, keep it as long as you like, adapt it. Open source buys understanding and reproducibility. Most models described as open in general conversation are open-weight, and a smaller number are open in the fuller sense.

Licenses actually vary

Downloadable does not mean permitted. Model licenses range from genuinely permissive to restrictive in ways that only surface during legal review, which is usually late.

Categories worth recognizing:

• Standard permissive software licenses. Apache 2.0 or MIT applied to the weights. The simplest case, with no model-specific restrictions.
• Custom vendor licenses. Often permissive in practice but with added conditions: usage thresholds above which separate terms apply, naming or attribution requirements, or restrictions on using outputs to train competing models.
• Research-only licenses. Explicitly prohibiting commercial deployment. Common for models released to accompany a paper.
• Use-restriction licenses. Permitting most uses while prohibiting specific applications. Whether these qualify as open is contested; what matters operationally is that they are conditions you must actually check against your use.

The practical advice is unglamorous. Read the license before building on the model, not before shipping. Record which license applies to each model in use, because the answer changes between versions of the same model family more often than people expect.

What you actually download

A model repository typically contains a handful of things, and knowing what each is removes most of the confusion of a first attempt.

• Weight files. The parameters themselves, usually in safetensors format, often split across several files. This is the bulk of the download, ranging from a few gigabytes to hundreds.
• Configuration. Architecture details the inference engine needs: layer counts, dimensions, attention configuration, context length.
• Tokenizer files. How text becomes tokens. Using a mismatched tokenizer produces output that is subtly wrong rather than obviously broken, which makes it a nasty failure.
• A chat template. How conversation turns are formatted into the exact string the model was trained on. Getting this wrong is the most common cause of a locally run model performing far worse than expected.
• A model card. Documentation of intended use, training approach at some level of detail, evaluation results, and known limitations. Quality varies widely.

The safetensors format is worth a note. It replaced Python pickle-based weight files, which could execute arbitrary code on load. Downloading weights in a pickle format from an untrusted source is running untrusted code. Safetensors cannot do that by design, which is why it became the norm.

Quantization and what it costs

Weights are trained at higher precision than they need to be run at. Quantization reduces the precision of each parameter, shrinking memory requirements substantially in exchange for some quality.

The arithmetic is what makes it interesting. A model with seventy billion parameters at sixteen-bit precision needs roughly one hundred and forty gigabytes of memory for weights alone, before context. The same model quantized to four bits needs roughly thirty-five. The first requires multiple datacenter accelerators. The second fits on hardware a team might actually have.

The cost is real and non-linear. Moving from sixteen to eight bits is usually close to free in quality terms. Eight to four is noticeable on hard tasks and often acceptable. Below four, degradation becomes obvious, particularly on reasoning and code, and particularly on long inputs.

A useful rule of thumb when hardware is the constraint: a larger model at four bits generally beats a smaller model at sixteen bits, up to the point where quantization artifacts start affecting the specific task. Since that point differs by task, this is a question to settle with your own evaluation set rather than with published numbers.

What hardware you actually need

The most common reason a first attempt at self-hosting goes badly is that sizing was done on weight size alone. Three things consume memory, and only the first is obvious.

The weights are fixed: parameter count multiplied by bytes per parameter. This is the number everyone calculates.

The key-value cache grows with sequence length and with the number of concurrent requests. Every token already processed in an active request holds cached state. In agentic workloads, where contexts are long and requests overlap, this frequently exceeds the weights themselves. A model that loads comfortably and then fails under three concurrent long-context requests is almost always hitting this.

Activation memory during a forward pass is smaller and non-trivial, and serving engines manage it for you.

The practical implication is to size for concurrency and context, not for the model. A useful starting estimate is to budget the weight size, then add capacity for the number of simultaneous requests you expect multiplied by your typical context length, then leave headroom. Serving engines expose settings to cap concurrency and context so that the failure mode is queuing rather than an out-of-memory crash, and setting those deliberately is worth doing before the first real load rather than after.

On the accelerator question, the honest summary is that consumer hardware runs small and mid-sized quantized models well enough for development and for the high-frequency small calls described later on this page. Serving a large model to real traffic is datacenter-class work. Teams that conflate the two end up disappointed by a workstation experiment and conclude that open weights are not viable, when what they actually learned is that one machine does not serve production traffic.

Running one

The tooling has improved enough that a first run takes minutes rather than a weekend, and the options divide along a clear line.

Workstation runtimes such as Ollama and llama.cpp are built for one user at a time. They handle download, quantization format, and serving behind a simple API. This is the right starting point for development, for privacy-sensitive local work, and for finding out whether a model is good enough before committing infrastructure.

Serving engines such as vLLM are built for concurrency. They batch requests, manage memory across many simultaneous sequences, and reach throughput that workstation runtimes do not attempt. This is what you deploy when a model has to serve real traffic, and it is a meaningfully different operational commitment.

Hosted open weights means someone else runs the serving engine and sells inference. You keep the ability to move, since the weights are downloadable and other providers offer the same model, and you avoid operating accelerators. For most teams this is the sensible middle position, and it is frequently overlooked because the conversation tends to be framed as local versus proprietary.

When open weights are the right call

• Data that cannot leave. Regulated records, material under legal hold, code with contractual restrictions. Here it is the only compliant option, not an optimization.
• Domain specialization. Fine-tuning on a narrow domain can make a mid-sized model outperform a much larger general one for that domain. This requires weights you can modify.
• Reproducibility. A pinned checkpoint behaves identically in two years. A hosted endpoint behind a stable name does not, which matters when a decision must be defensible after the fact.
• Longevity. Hosted models get deprecated. Weights on your storage cannot be withdrawn.
• Steady high volume. Beyond some sustained request rate, owning capacity is cheaper than renting it. The crossover is higher than most teams assume and it is real.
• Latency floors. Local inference removes network round trips, which matters for high-frequency small calls inside an agent loop.

When they are not

The honest counter-case matters as much, because reaching for open weights reflexively produces systems that are principled and worse.

Low or bursty volume leaves expensive capacity idle. Small teams without appetite for operating accelerators acquire a permanent operational burden. Work at the edge of what any model can do runs into the capability gap, which is still real for the hardest reasoning and long-context tasks even as it narrows. And rapid iteration is simply faster against a hosted endpoint, where trying a new model is a configuration change rather than a deployment.

The architectural answer is not to choose globally. It is to make the choice per workload, which is exactly what a routing layer provides.

Where they fit in an agentic system

Agentic workloads have a shape that suits open weights unusually well in some places and poorly in others.

High-frequency small calls

Classification, routing, extraction, and formatting happen constantly in an agent loop. These are the calls where a small local model is fast, free, private, and entirely adequate. Sending them to a frontier endpoint is the most common source of unnecessary cost in agentic systems.

Bulk processing

Summarizing ten thousand documents is exactly the workload where owning capacity beats renting it, and where the quality bar is usually reachable by a mid-sized model.

Sensitive tool results

An agent that reads customer records or internal code produces context containing that material. If policy restricts where such content may go, the model handling those steps has to run inside the boundary, regardless of what handles the rest.

Planning and hard reasoning

This is where open weights are least likely to be the right choice today. Multi-step planning is where model quality shows most clearly and where a weaker model produces expensive mistakes rather than merely worse prose.

Development

A local model removes cost anxiety from iteration and forces prompts to stay portable, both of which improve the system even when production runs on something else.

Evaluating one honestly

Public benchmarks narrow a shortlist and do not choose between finalists. Benchmark contamination is real, and general capability is a poor predictor of performance on a specific agentic workload.

What works is a small evaluation set built from your own tasks, run against every candidate including the incumbent, scoring quality, cost, and latency together. For open weights specifically, add three checks that hosted models rarely need:

• Chat template correctness. Verify against the model card. A wrong template is the most common cause of a model appearing much weaker than reported.
• Structured output reliability. Test tool-call and schema adherence explicitly, since this varies more between open models than general capability does.
• Behavior at your real context lengths. A model advertising a long context may degrade well before the limit, and agentic contexts are long.

Adaptation and fine-tuning

The ability to modify a model is the capability that open weights uniquely provide, and it is both more accessible and less often necessary than the discussion around it suggests.

Full fine-tuning updates every parameter and requires substantial hardware. Parameter-efficient methods, of which low-rank adaptation is the most common, train a small number of additional parameters while leaving the base model frozen. The resulting adapter is small enough to store cheaply and to swap at serving time, which makes it practical to maintain several task-specific adaptations of one base model.

What fine-tuning is good at is teaching form: a house output format, a domain vocabulary, a consistent tone, a specific classification task where you have labelled examples. What it is poor at is teaching facts. Knowledge introduced by fine-tuning is unattributable, undated, and impossible to update without retraining, which are exactly the properties a grounded architecture is trying to avoid. Facts belong in retrieval, where they can be pointed at and corrected.

Before reaching for it, the ordering that saves the most effort is: fix the context first, then the tools, then the prompt structure, then consider adaptation. Most problems people plan to fine-tune away turn out to be context assembly problems, and the diagnostic is cheap: if a stronger model handles the task correctly with the same context, the issue is capability and adaptation might help; if every model fails the same way, the context is wrong and no amount of training will fix it.

When fine-tuning is warranted, evaluate on two axes rather than one. A model tuned narrowly will improve on the target task and can quietly lose general capability, which shows up later as brittleness on the edges of the workload rather than as an obvious regression.

Gotchas worth knowing

• Memory is not just weights. Context consumes memory per concurrent request, and it grows with sequence length. Sizing on weight size alone leads to failures under concurrency.
• Benchmark numbers rarely reproduce. Different quantization, template, and sampling settings produce different results from the same weights.
• Parameter count is not capability. Training quality and data matter more than size, and comparisons across families on size alone are unreliable.
• Fine-tuning is easy to do badly. A small dataset can degrade general capability while improving a narrow task. Evaluate on both.
• Model names are reused. The same family name across releases can mean substantially different behavior. Pin versions.
• Distribution is a supply chain. Weights come from a repository. Verify the source and prefer safetensors over pickle formats.

How they score on openness

• Replaceable. Strong, provided your integration goes through a routing abstraction rather than a provider-specific client.
• Inspectable. Partial. You can examine the artifact and run experiments on it. You usually cannot inspect what produced it.
• Portable. Strong. The files run anywhere you have hardware.
• Bounded. Not applicable. A model has no authority model; boundaries are enforced by the harness.
• Grounded. Indirect. Open weights do not improve grounding; supplied facts do.
• Auditable. Strong for reproducibility. A pinned checkpoint plus recorded parameters means a past result can be re-derived.

What they are not

Open weights are not a privacy guarantee by themselves. Running a model locally keeps data local. It does nothing about what the surrounding tools send elsewhere, which is usually where leakage actually happens.

Open weights are not free. Hardware, operations, and engineering time replace per-token pricing. Sometimes that trade is favorable and sometimes it is not, and it should be calculated rather than assumed.

Open weights are not automatically safer or less safe. Guardrails vary by model and by deployment, and a model you run yourself is one you are responsible for configuring, which is a different position from a safer one.