Cloudflare Unweight is actually a Storage Problem

Cloudflare published a research paper recently about a tool they built called Unweight.

It compresses LLM weights by 22%.

Losslessly.
No quality degradation.
Bit-exact outputs.

The AI press latched onto the compression number and ran with it.

But that’s the wrong story, and misses the forest for the trees.

The actual story is buried three paragraphs into the technical writeup, and it’s a story us infrastructure veterans have been living for 30 years. If you’ve ever sized a storage array for a latency-sensitive workload, tuned a CDN for transfer cost, or argued with a vendor about cache hit ratios, you already understand what Cloudflare did.

You just haven’t seen it applied to a GPU yet.

Grab a drink and let me walk you through it.

---

The Oldest Problem in Computing

Let’s establish a baseline here… on an H100, the tensor cores can process data nearly 600 times faster than memory can deliver it.

Read that again.
Six. hundred. times. faster.

The compute is sitting there idle, waiting for data to show up. The bottleneck isn’t the math. The bottleneck is the memory bus.

This is not a new problem. This is the oldest problem in systems design.

Compute and storage have never run at the same speed. The entire history of storage architecture is a history of tricks designed to close that gap. Read-ahead caching. Write coalescing. Tiering. Prefetch algorithms. Inline deduplication to reduce what crosses the bus to begin with. Every one of these techniques is an answer to the same question…

How do you keep fast compute fed when storage is slow?

A quick teaching moment, because I want to make sure we’re all on the same page before I dig into what Cloudflare did. There are two kinds of memory on a modern GPU, and the difference matters. The big one is HBM, short for High Bandwidth Memory. It’s the 80GB or 141GB or 192GB number you see on the GPU spec sheet. It’s fast by every conventional measure (3.35 TB/s on an H100, which is roughly 100 times faster than your laptop’s RAM), but it’s still slow compared to the tensor cores it feeds. The other kind is on-chip shared memory, often called SMEM. It’s tiny (around 228KB per streaming multiprocessor on a Hopper GPU), it’s screamingly fast, and it sits right next to the compute units. Think of HBM as the main array of NVMe drives in your storage system, and SMEM as the cache on the storage controller. Same hierarchy. Different physical scale.

Now, every time a GPU generates a single token from an LLM, it has to read every model weight from HBM.

Every weight.
Every token.

If the weights are 140GB and your bandwidth is 3.35 TB/s, you can do the napkin math on how often the tensor cores are sitting idle. Most token generation is bandwidth-bound, not compute-bound. The math units are bored. The bus is overworked.

So, Cloudflare asked a simple question… If the bus is the bottleneck, can we put fewer bytes on it without losing any accuracy?

Their answer turns on a specific quirk of how model weights are stored. Modern LLMs use a 16-bit number format called BF16, short for Brain Float 16. Each BF16 number has three parts:

1. a sign bit (positive or negative)
2. a mantissa (the precision part, the digits)
3. and an exponent (the magnitude part, how big or small the number is)

The sign and mantissa look like random data. They vary unpredictably across the millions or billions of weights in a model, and you can’t compress random data. But the exponent tells a different story.

Here’s the cool part. Out of 256 possible exponent values, the top 16 cover more than 99 percent of the weights in any given layer. Information theory says you only need about 2.6 bits to represent that distribution. The format gives it 8. That gap, between 2.6 bits of actual information and 8 bits of allocated space, is pure redundancy.

If you’ve worked with storage for any length of time, you just smiled. Are you smiling? Do you know why yet? Sound familiar?

That’s the same kind of statistical redundancy that inline deduplication exploits on an all-flash array.

You don’t compress random data. You compress the patterns that show up over and over again. The weights have a pattern in their exponents.

Cloudflare found it and squeezed it, and the squeezing is done with Huffman coding.

Quick side-note: Huffman is a classical compression technique from 1952. It assigns short codes to common values and longer codes to rare ones. Same trick your favorite storage array uses for compression at rest. Same trick gzip uses on text files. It works because most real-world data has uneven distributions, and uneven distributions are compressible.

Cloudflare applies this Huffman coding selectively. Only to the MLP weights, which are the gate, up, and down projections inside each transformer layer. These three matrices are about two-thirds of a model’s parameters and they dominate memory traffic during token generation. The attention weights, the embeddings, and the layer norms stay uncompressed.

This is a surgical move, not blanket compression.
Compress where the bandwidth pain is.
Leave the rest alone.

The result is roughly 30 percent compression on the exponent stream, which translates to roughly 20% reduction in total model size. On a Llama 3.1 8B model, that’s about 13 percent of total memory saved. On a 70B model, you’re saving 18 to 28 GB depending on configuration.

At fleet scale, the cumulative bandwidth savings compound dramatically.

But the compression ratio isn’t the clever part. The clever part is where the decompression happens.

Cloudflare wrote a custom CUDA kernel that pulls the compressed weights from HBM into the on-chip shared memory, decompresses them there, and feeds the reconstructed weights directly into the tensor cores. The decompression never round-trips back through HBM. It happens in the fast cache, right next to the compute, in the time the bus would otherwise be sitting idle anyway.

This is read-ahead cache logic applied to GPU memory. The architecture is the same. The implementation layer is different. If you’ve ever tuned the read-ahead settings on a storage controller for a specific workload profile (sequential streaming versus random small-block IO, say), you’ve already done the conceptual work to understand what Cloudflare built. You’re paying compute to save bandwidth.

In essence, you’re trading something you have for something you don’t.

---

Why Cloudflare Built This

Cloudflare is not an AI lab. They didn’t build Unweight to publish a paper or win a benchmark. They built it because they run a CDN that ships LLM models to dozens of GPU points-of-presence around the world, and the model distribution problem was eating their lunch before the inference problem did.

Quick side-note, because the term gets tossed around a lot. A CDN is a Content Delivery Network. The original use case was caching static web content close to users so a viewer in Tokyo wasn’t pulling images from a server in Virginia. Cloudflare runs one of the largest in the world, with edge nodes in hundreds of cities.

Now imagine extending that model to AI inference. Instead of caching images, you cache models. Instead of serving HTML, you serve tokens. The user in Tokyo gets their model response from a GPU 50 milliseconds away, not from a hyperscaler datacenter on a different continent. That’s the bet Cloudflare is making with Workers AI and the broader Agents platform they rolled out this month.

Now here’s where the math gets interesting. If your model is 140GB and you need a copy of it on every edge GPU in your fleet, every gigabyte of compression is a real number on your network egress bill, right? Every minute of transfer time matters when you’re spinning up a new POP or rotating to a new model version. Compressed model bundles are 22% smaller. That’s the same percentage you save on memory bandwidth, but it manifests as faster time-to-deploy and lower transfer costs.

Cloudflare felt this pain on their CDN bill before they felt it on their HBM utilization graphs. The Unweight blog post talks about inference performance because that’s the sexy story, but the model distribution savings are arguably the bigger operational win.

They built a CDN solution to a problem the AI research community frames as a GPU problem.

This is what makes Cloudflare a credible voice on production inference, and it’s why their engineering choices are worth paying attention to. They’re not building toy benchmarks. They’re solving the problems that hit them at 3am on a Tuesday when traffic spikes and a new model needs to be everywhere five minutes ago. The most practical inference optimizations are going to keep coming from operators who run production infrastructure, not from researchers who train models. The operators are the ones who feel the bottlenecks in their utilization metrics, their egress bills, and their power draw.

We need to be paying more attention to what Cloudflare ships. Not because they’re going to be your inference vendor (most of you have hyperscaler relationships you’re not changing), but because the patterns they pioneer at edge scale tend to filter back into datacenter-scale designs eighteen months later more often than not.

---

Should You Care or Not?

Cloudflare is admirably forthcoming about the tradeoffs, which I appreciate. I’ve always said I’m transparent to a fault, and I love seeing it from companies I do business with.

Unweight is not a free lunch.

The decompression work has to happen somewhere, and right now it costs about 30% of throughput at typical batch sizes. That overhead narrows at larger batch sizes (larger batches mean more compute work to overlap with the decompression, so the cost gets hidden) and is expected to keep narrowing as Cloudflare optimizes the kernels. Three known optimization paths are already in flight, and the down projection (about a third of the compressible weights) hasn’t been compressed yet. So the headline numbers represent today, not the ceiling.

Should you turn it on? You all already know the answer…it’s the most famous and popular answer of any IT conundrum.

Everybody, all together now…

1…
2…
3…

“It Depends…”

If your inference is bandwidth-bound (most token generation is), and you have compute headroom (you usually do), the trade is favorable. You’re spending compute you weren’t using to save bandwidth you were running out of. This is exactly the same kind of decision you make when you turn on inline compression on a flash array. It costs you a little CPU. It saves you a lot of capacity. The economics are almost always positive when one resource is constrained and another is idle.

If you’re running small-batch, latency-sensitive inference (single-user chatbot, low-concurrency RAG, anything where time-to-first-token is the metric you live and die by), the 30% to 41% overhead at batch size 1 is probably not worth it yet. Wait for the next round of kernel optimizations. The math will get better.

If you’re running large-batch throughput inference (batch processing, document analysis, anything where tokens-per-second-per-dollar is the metric), turn it on. The overhead shrinks at batch size and the bandwidth savings compound.

If you’re hosting many small models on shared GPUs (Workers AI style multi-tenant inference platforms, internal model gardens, agent-per-team architectures), this is where Unweight gets really interesting. Twenty percent smaller weights means roughly 20% more models per GPU. That’s a unit economics change, not just a latency tweak. If you’re trying to run a fleet of 30 specialized models on the same hardware budget, that’s the difference between the CFO approving the build and the CFO suggesting you talk to a hyperscaler instead.

One gotcha I need to make sure you’re aware of… Unweight is currently a Hopper-only optimization. The kernel designs are specific to the H100 and H200 SM architecture, the WGMMA instruction set, and the 228KB SMEM budget per SM. Blackwell adaptation (B200 and B300) is on the roadmap but not yet shipping.

For most of you, that’s probably fine. Hell, most cloud inference capacity is still on Hopper, and Hopper is going to be the workhorse for at least another product cycle. But factor it in if you’re sketching a hardware refresh plan that crosses the Hopper-to-Blackwell boundary.

This is what the Pareto frontier conversation is really about, by the way…

Another quick side-note… NVIDIA talks constantly about the “Pareto frontier” of inference, which is just a fancy name for the curve of tradeoffs between throughput, latency, and energy efficiency. Different workloads sit at different points on the curve. The whole point of NVIDIA’s stack (TensorRT-LLM for compilation, vLLM for memory management, Dynamo for disaggregation) is to give you knobs that move you along that curve. Unweight is a new knob. It trades decompression compute for memory bandwidth.

In environments where bandwidth is the scarce resource, it’s a good knob to have. In environments where it’s not, leave it off.

The point isn’t that everyone should use it.
The point is that you should know it’s there.

---

The Bigger Picture

Still with me? I know this is a lot to digest, and I’m trying to frame it in relatable terms. So, step back from the specifics for a minute, because there’s a pattern worth naming.

Unweight didn’t ship in isolation. It came out of Cloudflare’s Agents Week 2026, alongside an inference gateway across a dozen-plus model providers, a persistent agent memory service, edge sandboxes, a feature flag service, and a half-dozen other primitives that all point in the same direction. The important takeaway is that Cloudflare is rebuilding cloud primitives for AI workloads.

Inference at the edge.
Models as cached objects.
Agents as the new request type.

They’re not adding AI features to a CDN. They’re rebuilding the CDN for an AI-shaped traffic pattern.

If that sounds familiar, it should. It’s the same shape as what happened to storage 20 years ago, when distributed file systems started to outpace traditional SAN architectures and we had to rethink everything from cache coherency to replication topology. AI inference is becoming a memory hierarchy problem the way storage became a network problem. The vocabulary is different. The shape is the same.

The irony for cloud architects watching this play out… the same data reduction instincts that produced Unweight at the GPU layer have been baked into enterprise storage arrays for two decades.

Inline deduplication.
Compression at rest.
Hot-tier caching.
Cold-tier offload.

They’re already running natively in your hyperscaler of choice as first-party services. FSx for ONTAP on AWS, Azure NetApp Files, and Google Cloud NetApp Volumes are how those instincts arrived at the storage tier under modern AI workloads. Cloudflare just brought the same thinking to the HBM tier above them.

The companies that figure out AI infrastructure first are going to be the ones with people who can map storage instincts onto memory hierarchies, CDN instincts onto inference distribution, and capacity planning instincts onto fleet-level GPU economics.

That’s not the AI research community. That’s you.

Your storage instincts are directly applicable to inference optimization. Your CDN intuition tells you exactly why Cloudflare built Unweight before anyone else did. Your capacity planning math works the same whether the unit is IOPS or tokens-per-second.

The vocabulary is different.
Tensor cores instead of storage controllers.
HBM instead of NVMe.
BF16 exponent bytes instead of block-level deduplication.
SMEM instead of controller cache.

But the problem statement is the same one you’ve been solving for decades.

The next time someone tells you that AI infrastructure requires entirely new thinking, point them at Cloudflare’s Unweight paper (PDF). Then tell them it’s a read-ahead cache problem with a CDN distribution wrapper. Watch them nod slowly as the pieces connect.

You weren’t behind. You were early. You just hadn’t seen the new vocabulary yet.

/Nick

---

Frequently Asked Questions

What is Cloudflare Unweight and how does it work?

Unweight is a lossless inference-time compression system for LLM weights on NVIDIA Hopper GPUs. It reduces total model size by roughly 20% by Huffman-coding the redundant exponent bytes in BF16 weight tensors. Decompression happens in fast on-chip shared memory and feeds reconstructed weights directly to the tensor cores, avoiding a round-trip through high-bandwidth memory. The result is less data crossing the GPU’s memory bus during inference, with no impact on output quality.

Does Unweight affect model output quality?

No. Unweight is bit-exact lossless compression, meaning the decompressed weights are numerically identical to the originals. This is fundamentally different from quantization techniques like FP8 or INT4, which trade precision for size and can affect response quality in unpredictable ways. If you need guaranteed output equivalence (regulated industries, A/B-tested production endpoints, anything where reproducibility matters), Unweight gives you that guarantee. Quantization does not.

When should infrastructure teams actually enable Unweight?

Three scenarios make sense today. First, large-batch throughput inference, where the 30 percent throughput overhead shrinks at scale and the bandwidth savings compound. Second, multi-model serving on shared GPUs, where 20 percent smaller weights means meaningfully more models per GPU, which is a unit economics improvement rather than a latency tweak. Third, edge or distributed inference deployments, where compressed model bundles also reduce transfer times and egress costs. Skip it for small-batch, latency-sensitive workloads until the kernel optimizations land.

Does Unweight work on Blackwell GPUs?

Not yet. The current kernels are specific to Hopper architecture (H100 and H200), including the WGMMA instruction set and the 228KB shared memory budget per streaming multiprocessor. Adaptation to Blackwell (B200 and B300) is on the roadmap but not yet shipping. Since most cloud inference capacity is still running on Hopper hardware (and will be through at least the next product cycle), this is workable for most teams today. Factor it into longer hardware refresh plans.

Is Unweight the same as quantization?

No, and the difference is important. Quantization is lossy compression that reduces the precision of weights (16-bit floats become 8-bit or 4-bit integers). It saves more space but changes outputs in ways that can be hard to predict. Unweight is lossless compression that exploits statistical redundancy in the exponent field of BF16 weights without changing precision. Outputs are bit-identical to the uncompressed model. The two techniques are complementary, not alternatives. You can quantize a model and still apply Unweight on top, or use Unweight to keep BF16 fidelity while still saving bandwidth.