Google TurboQuant in August 2026: Where It Actually Runs

The August 2026 status report on Google's KV cache compression algorithm: the full technical grounding, every place it actually runs, the exact flags to use, and our own July claims graded against what happened next.

One hundred thirty-four days after Google announced TurboQuant, it ships in exactly two production systems: vLLM for LLM serving and Qdrant 1.18 for vector search. Everything else is forks, side doors, and abandoned pull requests.

On March 24, 2026, Google Research published TurboQuant, a training-free, data-oblivious vector quantization algorithm that compresses the Key-Value (KV) cache of large language models to roughly 3.5 effective bits per value, claiming at least 6x memory reduction and up to 8x attention-logit speedup on NVIDIA H100 GPUs. The internet called it real-life Pied Piper, and memory chip stocks cratered within 48 hours: SK Hynix dropped 6%, Samsung fell 5%, Micron slid 3.4% - CNBC. Then the actual engineering verdicts started arriving: a three-week vLLM merge, a locked llama.cpp rejection, a hyperscaler reference architecture, and an accuracy study that redrew the safety map.

We published the first version of this guide in the hype window, three days after the announcement, and refreshed it at the 100-day mark in July. This August update is different in kind, not just in date. We re-verified every load-bearing claim against the live source this week, added the two adoption facts the July version missed entirely (Qdrant shipped it, and MLX became the local side door), rewrote the market section because its triumphant ending aged badly in four weeks, and graded our own July calls in a second scorecard. Where we were wrong, the entry says wrong. That accountability loop is the point of this page: aggregators can summarize the paper, but they cannot tell you which of their own predictions failed, and the failed ones are where the useful information lives.

The technical foundations have not changed, so this guide keeps them in full: how TurboQuant works stage by stage, why the KV cache is the real bottleneck, the honest accuracy picture at 70B and 200B+ scale, the quantization landscape it sits inside, the AWS enterprise architecture built on it, and the hardware shift under it. On top of that grounding, the August layer covers the verdict table of everywhere it runs, the exact vLLM flags, the keys-versus-values plot twist, the Qdrant surface, the MLX saga, the market's second reversal, and what all of it means for long-running AI agent workloads, where KV cache pressure is most extreme.

Contents

1. Where TurboQuant Actually Runs: The August 2026 Verdict Table
2. Why the KV Cache Is Still the Real LLM Bottleneck
3. TurboQuant: What Google Actually Built
4. How TurboQuant Works (The Two-Stage Algorithm)
5. From Announcement to August: The Adoption Timeline
6. The Accuracy Reality Check: What "Zero Loss" Survived at Scale
7. Running TurboQuant in Production: The vLLM Playbook
8. The Keys-vs-Values Plot Twist
9. Qdrant 1.18: The Second Shipped Surface
10. Local Inference: llama.cpp Said No, MLX Opened a Side Door
11. The LLM Quantization Landscape in Mid-2026
12. The Successor Wave: TurboQuant as the New Baseline
13. Enterprise Adoption: AWS's Official TurboQuant Architecture
14. The Market Verdict, Act Three: SK Hynix After the Melt-Up
15. The Hardware Ground Shifted: Blackwell Ultra and Native FP4
16. Agent Workloads: Compounding Error Over Long Horizons
17. The August 2026 Decision Cheat Sheet
18. Grading Our Own Calls: Two Scorecards
19. What Comes Next

1. Where TurboQuant Actually Runs: The August 2026 Verdict Table

If you came here to decide whether TurboQuant is usable for your stack this month, this section is the answer, and it comes first on purpose. The long-form background (how the two-stage algorithm works, why the KV cache dominates long-context memory, what the theory guarantees) has not changed since spring, so it is kept in full in Sections 2 through 4 for readers who need the grounding, and the original paper remains the best deep reference - arXiv. What changed is the deployment map, and the deployment map is what nobody else keeps current.

The one-paragraph technical recap for new readers: TurboQuant compresses the KV cache, the per-token memory that grows linearly with context length and concurrency, not the model weights. It composes a randomized-rotation quantizer (PolarQuant) with a 1-bit error correction stage (QJL) to get near-optimal compression with unbiased attention inner products, needs no calibration data, and is provably within about 2.7x of the information-theoretic limit - Tom's Hardware. The KV cache it targets is a real budget line: a 70B-class model at a 1 million token context needs roughly 328 GB for the uncompressed cache against about 140 GB for the weights - InfoQ.

Here is where that math actually runs as of the first week of August 2026, with a verdict per row. Every status in this table was verified against the linked source this week, not recalled from our July version.

Three things jump out of this table when you compare it against any March-era coverage still ranking in search. First, the binary "adopted or rejected" frame is wrong: TurboQuant now lives on two shipped surfaces (LLM serving and vector search) plus one official cloud reference architecture, the AWS design that lifts Llama 3.1 405B context capacity from roughly 82K to 400K+ tokens on P5en instances - AWS. Second, the local-inference story is no longer a flat rejection; Section 10 traces the MLX side door that opened after our July version went to press. Third, Google still has not released code 134 days in, which means every benchmark number in this article comes from independent implementations, a fact that cuts both ways: verification is decentralized, but no single "official" number exists to hide behind.

The rest of this guide walks each verdict in depth. The foundations come first for readers new to the topic, then the practitioner material in the order you need it: the accuracy picture, how to run it, the new finding that complicates the preset choice, the vector-search surface, the local saga, the landscape and successor research, the enterprise and market and hardware context, and the operator's view from running agent fleets against exactly this memory wall.

2. Why the KV Cache Is Still the Real LLM Bottleneck

Most conversations about LLM efficiency focus on model weights. A 70 billion parameter model in FP16 takes roughly 140 GB of VRAM just to load. Compressing those weights to 4 bits cuts that to around 35 GB, fitting the model on a single high-end GPU instead of a multi-GPU cluster. This is the problem that methods like GPTQ, AWQ, and GGUF solve, and they solve it well.

But weights are a fixed cost. You load them once and they stay in memory for the duration of the session. The KV cache, by contrast, grows with every token processed. In transformer architectures, the attention mechanism stores key and value vectors for all previously seen tokens so the model can reference earlier context when generating new tokens. For a short conversation, this is manageable. For long document analysis, a multi-turn agent session, or a RAG pipeline processing thousands of retrieved chunks, the KV cache becomes the dominant memory consumer.

The numbers are stark, and they have only grown since March. For a 70B parameter model serving 512 concurrent users, the KV cache alone can consume 512 GB of memory, nearly four times the memory required for the model weights themselves - VentureBeat. InfoQ's April analysis made the single-user version of the same point: a Llama 70B-class model holding a 1 million token context needs roughly 328 GB for the uncompressed KV cache against about 140 GB for the weights - InfoQ. The cache is not a rounding error. At long context, it is the budget.

To understand why the cache grows so fast, consider the mechanics. Each transformer layer stores a key vector and a value vector for every token in the sequence. A model with 80 layers processing a 128,000-token context at FP16 precision stores 80 times 128,000 times 2 (key plus value) tensors, each with the model's hidden dimension. The total scales linearly with sequence length, linearly with layer count, and linearly with batch size. There is no compression, no pruning, and no sharing in a standard implementation: every token gets its own cache entry in every layer for every user.

The bottleneck is especially acute for three use cases that define the current AI deployment landscape. First, long-context inference: frontier models now routinely support context windows of 200,000 tokens or more, but the KV cache for these sequences is enormous. Second, agentic AI workflows: autonomous agents that plan, execute, observe, and iterate accumulate massive KV caches across multi-turn sessions, a dynamic we mapped in detail in our guide to long-running coding agents. Third, high-concurrency serving: cloud providers running thousands of simultaneous inference requests must allocate KV cache memory per request, and the total scales with both sequence length and batch size.

Prior to TurboQuant, the most cited KV cache compression method was KIVI, published at ICML 2024, which achieved approximately 2.6x compression using asymmetric per-channel and per-token quantization. Other approaches like SnapKV and PyramidKV used token pruning (discarding less important cached tokens) rather than compressing them. These methods helped, but none achieved the compression ratios that would fundamentally change the economics of long-context serving.

The reason KV cache compression is harder than weight compression comes down to a fundamental property of the attention mechanism. When the model computes attention scores, it takes the inner product between query vectors and key vectors. Even a small systematic bias introduced by quantization can corrupt these inner products, causing the model to attend to the wrong tokens. Weight quantization methods do not face this constraint as severely, because weights are multiplied with activations in feed-forward layers where bias affects magnitude more than relative ordering. KV cache quantization must preserve not just the values themselves but the relationships between them as measured by inner products.

The fundamental trade-off in KV cache management, before TurboQuant, was between memory and quality. You could truncate the cache (losing old context), prune it (selectively discarding tokens the model deems less important), or quantize it (reducing precision). Truncation and pruning are lossy in uncontrolled ways: you never know for certain which tokens will matter later in the generation. Quantization preserves all tokens at lower precision, but prior methods either degraded quality noticeably or achieved only modest compression ratios. This is precisely the problem TurboQuant was designed to solve, and, as the past 134 days demonstrated, it solved part of it.

3. TurboQuant: What Google Actually Built

TurboQuant is a vector quantization algorithm, not a model training technique and not a weight compressor. It operates exclusively on the KV cache during inference, compressing key and value vectors as they are written to the cache and decompressing them when the attention mechanism reads them back. The entire process happens online, meaning it processes each vector as it arrives without needing access to the full dataset or any calibration data.

The paper, titled "TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate," was first posted to arXiv in April 2025 and published as a conference paper at ICLR 2026. It was presented as a poster on Saturday, April 25, 2026, from 11:15 AM to 1:45 PM in Pavilion 4, booth #4709, in Rio de Janeiro - ICLR. The presenting authors were Amir Zandieh and Vahab Mirrokni (Google Fellow and VP) from Google Research, Majid Daliri from New York University, and Majid Hadian from Google DeepMind. The technique has since accumulated enough ecosystem weight to earn its own Wikipedia article, which documents the two variants, TurboQuant-mse and TurboQuant-prod.

The Google Research announcement is dated March 24, 2026, one day earlier than most early coverage reported (our own March version of this guide said March 25, an error we corrected in July). The blog post claims 3-bit KV cache quantization, at least 6x memory reduction, and up to 8x attention-logit speedup on H100. Notably, it links the three underlying papers but no code repository, and as of August 2026 Google still has not released an official implementation. The Q2 2026 reference implementation the community expected never materialized. Everything running in production today is community-built or framework-native.

The headline numbers bear repeating because they are unusual. TurboQuant compresses KV cache entries to an effective 3.5 bits per value (3 bits for the primary quantization plus 1 bit for error correction), achieving at least 6x memory reduction compared to standard 16-bit or 32-bit representations. On NVIDIA H100 GPUs, the compressed representation enables up to 8x speedup specifically for attention logit computation - Tom's Hardware. The paper reported zero measurable accuracy loss at 3.5-bit precision on models up to 8B parameters, a claim we examine much more carefully in Section 6, because larger-scale testing since May has drawn a sharper line between where it holds and where it does not.

What makes these numbers credible rather than marketing is the theoretical backing. The paper proves that TurboQuant operates within a factor of sqrt(3pi/2), approximately 2.7x, of the information-theoretic lower bound established by Shannon source coding theory. In plain terms: no online quantization algorithm, regardless of how clever, can achieve more than about 2.7x better compression at the same error rate. TurboQuant is provably close to the theoretical maximum.

Three properties distinguish TurboQuant from every prior KV cache compression method. It is training-free, requiring no gradient updates, no fine-tuning, and no model-specific adaptation. It is data-oblivious, meaning it does not need a calibration dataset or any statistical profiling of the model's activation distributions. And it is online, processing each vector independently as it arrives without buffering or look-ahead. These properties make it deployable without per-model setup cost, which is a significant practical advantage over methods that require hours of calibration.

Our March version claimed TurboQuant "could be inserted into any existing serving framework as a drop-in compression layer." The integration record since proved that claim too strong. The merged vLLM implementation explicitly excludes hybrid Mamba-plus-attention architectures, which bit users trying to run recent hybrid models through it - vLLM PR #38479. SGLang's fused implementation shows prefill dropping to 64% of bf16 throughput at 4096-token inputs even while decode runs at near parity - SGLang Issue #23134. Drop-in for standard attention stacks, yes. Drop-in for everything, no.

4. How TurboQuant Works (The Two-Stage Algorithm)

TurboQuant combines two independently published algorithms into a unified system. The first stage, called PolarQuant, handles the primary compression. The second stage, called QJL (Quantized Johnson-Lindenstrauss), corrects the systematic bias that the first stage introduces. Together they achieve what neither can alone: high compression with unbiased inner product estimates.

The core problem with naive quantization of KV cache vectors is that different dimensions have different value distributions. Some dimensions cluster tightly around zero while others spread across a wide range. Traditional quantizers handle this by computing per-block normalization constants (scaling factors) that rescale each block of values before quantization. But these normalization constants themselves consume memory, eroding the compression benefit at low bit-widths.

Stage 1: PolarQuant (MSE-Optimal Quantization)

PolarQuant eliminates this overhead through a geometric insight. Before quantization, it applies a random orthogonal rotation to each input vector. Random rotation sounds like it should destroy information, but in high-dimensional spaces the opposite happens: rotation transforms arbitrary distributions into predictable ones. After rotation, each coordinate's distribution converges to a Beta distribution (which approaches Gaussian in high dimensions), making the values concentrated and statistically uniform across dimensions.

Because the rotated values follow a known distribution, PolarQuant can use a fixed, precomputed codebook rather than computing per-block scaling factors. The codebook is a set of Lloyd-Max optimal scalar quantizers computed once for the Beta distribution at each target bit-width. During inference, each value is simply mapped to its nearest codebook entry with no additional metadata needed. This is the key innovation: by exploiting the geometry of high-dimensional spaces, PolarQuant achieves MSE-optimal quantization without the memory overhead of normalization constants. The rotation step also neutralizes the activation outlier problem that breaks naive low-bit quantization, by spreading outlier energy uniformly across all dimensions, the same geometric insight behind QuIP# and SpinQuant but applied online to cache vectors.

Stage 2: QJL Error Correction (Unbiased Inner Products)

PolarQuant by itself is excellent at minimizing mean squared error, but MSE optimality does not guarantee unbiased inner products, and inner products are what the attention mechanism computes. Even an MSE-optimal quantizer can introduce systematic bias in query-key dot products, causing the model to systematically over-weight or under-weight certain attention connections.

QJL addresses this with a residual correction that costs exactly 1 bit per dimension. After PolarQuant compresses a vector, QJL computes the residual error, applies the Johnson-Lindenstrauss Transform to project that residual, and reduces each projected element to a single sign bit. The mathematical proof relies on the fact that the sign of a random projection preserves enough directional information to cancel the quantization bias while adding only 1 bit of overhead per dimension. QJL was separately published at AAAI 2025; PolarQuant appeared at AISTATS 2026.

The Guarantees, and What They Do Not Cover

When both stages operate together, the total effective bit-width is approximately 3.5 bits, and the system achieves two simultaneous guarantees prior methods could not combine. The MSE distortion is bounded by sqrt(3pi/2) times 1/4^b for b-bit quantization: concretely 0.03 at 3-bit and 0.009 at 4-bit for unit-norm vectors. The inner product estimate is unbiased, with variance that shrinks as vector dimension grows, which is why larger head dimensions tolerate TurboQuant better. And the lower-bound theorem proves any conceivable randomized quantizer must incur at least 1/4^b MSE, so TurboQuant sits within a constant 2.7x factor of perfect across all bit-widths - arXiv.

It is worth being precise about what these theorems guarantee, because the gap between theory and deployment is exactly where the drama of the past four months happened. The theorems bound per-vector distortion. They do not bound end-to-end task accuracy after billions of slightly-distorted attention operations interact with each other across 80 layers and hundreds of thousands of decode steps, and they say nothing about how errors compound across steps in multi-turn agentic loops. The paper was honest about this scope. The headlines were not. Section 6 and Section 16 cover what large-scale empirical testing found in the gap.

TurboQuant did not emerge from nowhere; it is the capstone of a three-paper research arc. QJL (AAAI 2025) proved the 1-bit debiasing trick but needed a stronger primary quantizer. PolarQuant (AISTATS 2026) delivered the metadata-free MSE-optimal quantizer but inherited the inner-product bias that all MSE-optimal methods share. TurboQuant composes them so that each stage covers the other's weakness, and the near-optimality proof relies on the combined analysis. This construction is why the accuracy claims were credible enough for a three-week production merge: the system is not an empirical trick that might not transfer, but a composition of independently peer-reviewed components whose guarantees are inherited rather than hoped for.

The term data-oblivious deserves one more note because it drives the deployment story. A data-oblivious algorithm makes decisions based only on mathematical properties (dimensionality, bit-width, the post-rotation distribution), never on the actual data values. In contrast to GPTQ (which needs Hessian statistics from calibration data) or AWQ (which profiles activation magnitudes), TurboQuant works identically on any model without modification. That property is why vLLM could ship it as a generic --kv-cache-dtype option rather than a per-model artifact, and it is why the same math transfers to embedding search, where TurboQuant indexes a 1,536-dimensional vector in 0.0013 seconds versus 37 to 3,957 seconds for Product Quantization and RaBitQ style methods that require offline codebook training. Section 9 shows what happened when a vector database took that transfer seriously.

5. From Announcement to August: The Adoption Timeline

No other TurboQuant guide we are aware of has this section, because most were written in the two weeks after the announcement and never touched again. The value of a running retrospective is that it separates what the ecosystem actually adopted from what commentators predicted it would adopt. The two lists are very different.

The timeline below is the factual spine of this entire refresh. Every event on it changed at least one claim from an earlier version of this guide, and each gets a deeper treatment in the sections that follow.

Before walking the timeline, it is worth recalling the atmosphere it started in, because the initial reception shaped which claims got scrutinized and which got amplified. The internet's first response was to compare TurboQuant to Pied Piper from HBO's Silicon Valley, and the Google Research blog post drew over 7.7 million views in its first 48 hours - TechCrunch. Community implementations appeared on GitHub before Google's own market close, and the most useful early skepticism came from the Hacker News thread, where GPU researchers noted that the paper "conveniently avoids reporting inference wall-clock time" and questioned how the rotation step would behave on tensor cores - Hacker News. Those two criticisms aged remarkably well: the wall-clock question is exactly what the vLLM study later quantified as a real throughput deficit, and kernel efficiency is exactly what SGLang's fused rewrite had to solve. The hype was wrong in the ways hype usually is, but the technical skeptics of week one turned out to be previewing the findings of month three.

The compressed version of the story: integration won in the cloud and lost at the edge. On April 15, three weeks after the announcement and ten days before the ICLR poster, vLLM merged TurboQuant upstream, shipping four presets with 2.6x to 4.9x KV compression. Our March version reported only "a feature request (Issue #38171)" with "preliminary integration work underway." The actual merge happened faster than almost anyone predicted, which was the single biggest status change of the spring.

The academic milestone followed on schedule. The ICLR 2026 poster session ran on April 25 in Rio de Janeiro exactly as planned - ICLR. What did not follow was Google's code. The research.google blog still links only the three papers, and the community expectation of an official Q2 2026 reference implementation (which we repeated in March) simply failed to happen. In practice this mattered less than expected, because vLLM's merged implementation became the de facto reference.

May brought the reckoning on accuracy, and the quietest big event of the whole arc. On May 11, the vLLM team published the first comprehensive TurboQuant study at real production scale, testing Llama-3.3-70B-Instruct, Qwen3-30B-A3B variants, and the 200B+ parameter MiniMax-M2.7 - vLLM blog. The findings redrew the safety map: conservative presets held up nearly everywhere, while aggressive presets showed roughly 20-point accuracy drops on hard math and coding reasoning. Two days later, on May 13, Qdrant shipped TurboQuant in Qdrant 1.18 for vector search, an event most TurboQuant coverage, ours included, failed to register for months; Section 9 makes the correction in full.

June resolved the local-inference question in the negative, at least upstream. On June 2, llama.cpp closed its TurboQuant PR unmerged, with the maintainer writing that "so far no one has presented evidence that would justify the opportunity cost of adding TurboQuant support," and locked the thread the next day - llama.cpp PR #21089. One day earlier, on June 1, AWS had published an official Machine Learning blog architecture pairing TurboQuant with GPUDirect Storage on FSx for Lustre - AWS. The juxtaposition is the whole story in miniature: hyperscalers embraced TurboQuant the same week the local-first ecosystem declined it. July then added the market's third act (SK Hynix's Nasdaq listing and reversal, covered in Section 14) and the quiet persistence signal: vLLM v0.26.0 shipped July 27 with the TurboQuant presets intact in the current release line.

The timeline reframes what kind of technology TurboQuant turned out to be. It is not a universal standard, and it is not vaporware. It is a production optimization for memory-bound serving, with real, measurable, preset-dependent trade-offs, officially supported in the most popular open-source serving engine and a major vector database, adopted in hyperscaler reference architectures, rejected where its costs outweighed its benefits, and already serving as the baseline that newer research must beat. Each of those clauses gets its own section below.

6. The Accuracy Reality Check: What "Zero Loss" Survived at Scale

The original TurboQuant paper evaluated models up to roughly 8 billion parameters (Llama-3.1-8B-Instruct being the largest), across LongBench, Needle-in-a-Haystack, ZeroSCROLLS, RULER, and L-Eval - Google Research. At 3.5-bit quantization, LongBench scored 50.06 versus 50.16 for the FP16 baseline, a difference within noise. Our March version flagged the obvious caveat: nothing above 8B had been tested, and empirical confirmation on frontier-scale models was absent. That caveat has now been resolved, and it resolved partly against the headline claim.

The vLLM team's comprehensive study, published May 11, 2026 on vLLM 0.20.2, tested Llama-3.3-70B-Instruct, Qwen3-30B-A3B-Instruct-2507, Qwen3-30B-A3B-Thinking-2507, and the 200B+ parameter MiniMax-M2.7. The results split cleanly by preset aggressiveness. Conservative configurations stayed near baseline on general tasks. Aggressive configurations showed approximately 20-point accuracy drops on hard math and coding reasoning benchmarks. At 256k context on Qwen3-30B, aggressive presets scored 31-33% AUC versus 45% for the BF16 baseline. The failure modes concentrate exactly where errors have the most room to compound: long chains of reasoning and very long contexts.

The official per-preset perplexity numbers from the vLLM API documentation quantify the gradient. The k8v4 preset costs +1.17% perplexity. The 4bit_nc preset costs +2.71%. The k3v4_nc preset jumps to +10.63%, and the fully aggressive 3bit_nc preset costs +20.59%. Those four numbers are the most useful thing to memorize in this entire guide: they tell you that the marginal-safety line sits at roughly 4 bits for keys, and that the 3-bit configuration Google's headline was built on is a genuine quality trade, not a free lunch.

The merged PR's own evaluation on Qwen3-4B makes the same point on task accuracy. GSM8K math accuracy degrades monotonically with preset aggressiveness, while Needle-in-a-Haystack stays at 100% on every preset - vLLM PR #38479. Retrieval is robust to KV quantization. Multi-step reasoning is not. That asymmetry, not the headline compression ratio, is the practical takeaway most March coverage missed entirely.

Was the paper wrong, then? No, and the distinction matters for how you weigh future compression claims. The paper reported zero loss at the configurations and scales it tested, with theoretical bounds that were always about per-vector distortion rather than end-to-end task accuracy. What failed was the extrapolation, made by headlines and by early guides including ours, that per-vector near-optimality implies task-level immunity at any scale, any context length, and any preset. The honest 2026 formulation is: TurboQuant is effectively lossless at conservative presets like k8v4, measurably lossy at aggressive presets, and the loss concentrates in reasoning-heavy and very-long-context workloads.

Independent analyses converged on the same shape from different angles. Deep Infra's deployment writeup found that quality degradation begins below roughly 3B parameters at 3-bit, and that values need tighter precision than keys in the sense that 2-bit values degrade significantly, which is exactly why the shipped presets are asymmetric configurations like k3v4 rather than symmetric ones. The Kaitchup's independent review from March 27 confirmed 2.5 to 3.5-bit KV compression works on LongBench and NIAH, while flagging that Google's own evaluation leaned on older models and that community code was still early and fragmented at the time. And Two Minute Papers' estimate via InfoQ put realistic real-world gains at 30-40% memory and speed for typical workloads, with the 6x headline applying to KV-dominated corner cases.

There is also a throughput asterisk that the March hype skipped. The vLLM study measured TurboQuant variants running 20-34% below FP8 and BF16 baseline throughput on their test matrix, with up to 60-68% latency overhead in the worst configurations, in exchange for 2.4x-3.7x KV capacity gains versus FP8's 2x. The compensating result: under burst load, BF16 time-to-first-token exploded to roughly 17 seconds while TurboQuant configurations prevented the blowup entirely, because compressed caches kept requests from queuing for memory. TurboQuant is not a speed upgrade. It is a capacity and stability upgrade that sometimes costs speed, and whether that trade wins depends on whether your bottleneck is compute or memory.

7. Running TurboQuant in Production: The vLLM Playbook

In March, running TurboQuant meant patching together community GitHub repos. As of August 2026, it means passing one flag to stock vLLM, and the presets merged in April persist into the current release line: vLLM v0.26.0 shipped July 27, 2026, and the TurboQuant preset documentation is live in the current API reference - vLLM docs. This section is the concrete playbook, because the difference between the four presets is the difference between free capacity and a silent 20-point reasoning regression.

The merged implementation exposes TurboQuant through the existing --kv-cache-dtype argument, with the default configuration using 3-bit keys and 4-bit values at head dimension 128, built on Hadamard rotation plus Lloyd-Max scalar quantization. Starting a server with the conservative preset looks like this:

# Conservative preset: 2.6x KV compression, +1.17% perplexity
vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --kv-cache-dtype turboquant_k8v4

# Aggressive preset: 4.9x KV compression, +20.59% perplexity (reasoning risk)
vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --kv-cache-dtype turboquant_3bit_nc

The four presets form a clean ladder of compression against quality, and picking one should be a deliberate decision rather than a default. The table below consolidates the numbers from the merged PR and the official API docs into one view, all re-checked against the live documentation this week.

The production-safe recommendation that falls out of every study so far is k8v4 as the default. It keeps 8-bit keys (protecting the inner products that attention accuracy depends on), compresses values to 4 bits, sustains 79-100% of baseline throughput, and costs a nearly invisible +1.17% perplexity - vLLM PR #38479. At fleet scale, 2.6x cache capacity is a GPU-count change rather than a tuning tweak. Reach for the aggressive presets only when context capacity is the binding constraint and the workload is retrieval-shaped rather than reasoning-shaped, and validate on your own evaluation set before and after.

Two traps still catch practitioners, and both were re-verified live this week. The first is the docs-lag trap: vLLM's user-facing quantized-KV-cache feature page still lists only auto, fp8, fp8_e4m3, and fp8_e5m2, with no mention of the TurboQuant presets - vLLM feature docs. Four months after the merge, teams reading the wrong docs page still conclude the feature does not exist. It exists; the feature docs are behind the API reference. The second is the hybrid architecture exclusion: the merged implementation does not support hybrid Mamba-plus-attention models, so recent hybrid designs cannot use these presets at all, and vLLM will reject the configuration, though the community testing on exactly those architectures produced this refresh's most interesting new finding, covered next.

Beyond vLLM, the serving-framework picture is uneven. SGLang has the most technically interesting implementation proposal: a fused-Triton rewrite that reads packed 4-bit KV directly during attention with no dequantization buffer, hitting 93-105% of bf16 decode throughput (against roughly 1% in the naive first implementation), GSM8K at 79.3% versus bf16's 79.2%, MMLU parity at 68.3%, and 3.88x compression on H200 with Llama-3.1-8B - SGLang Issue #23134. The catch: prefill throughput degrades to 64% of bf16 at 4096-token inputs, and the effort stalled, leaving SGLang support in community-patch territory rather than a stock flag. The fused-kernel numbers matter anyway, because they establish that the early community figure of roughly 1.2x full-pipeline speedup from naive Triton kernels was an implementation artifact, not an algorithmic ceiling.

For teams doing capacity planning, the k8v4 arithmetic works like this. A 70B-class deployment whose KV cache consumes 512 GB at high concurrency drops to roughly 197 GB at 2.6x, and to roughly 104 GB at 4.9x if the aggressive preset's quality trade is acceptable. Combined with FP8 weights, the k8v4 preset is the rare optimization that changes GPU count rather than utilization percentages, which is why it cleared vLLM's bar for an upstream merge in three weeks while so many quantization papers never leave the research-repo stage.

The economic sensitivity of that arithmetic depends entirely on what fraction of your GPU memory the cache occupies, and it pays to compute this before touching any flag. For short-context, high-throughput chat serving with sub-1K token conversations, weights dominate memory and the cache is a small slice, so TurboQuant's compression saves little and its throughput cost can make it a net negative. For long-context processing (legal review, code-repository understanding, document analysis at 100K+ tokens), the cache can equal or exceed weight memory, and 2.6x cache compression translates almost directly into GPU count reduction. For high-concurrency serving, the cache scales with active session count, and the 512 GB scenario above is routine rather than extreme. The one-sentence heuristic: measure your KV-cache share of total GPU memory first, and if it is under 20%, TurboQuant is not your lever, no matter how good the paper is.

8. The Keys-vs-Values Plot Twist

Every KV quantization guide written this spring, ours included, repeated the same design rule: protect the keys. Keys feed the attention inner products that decide which tokens the model looks at, so key precision was assumed to be the load-bearing wall, and the shipped preset names encode that assumption (k8v4 keeps 8-bit keys while compressing values to 4 bits). The official vllm-metal documentation states the doctrine outright: "K precision is load-bearing in a way V precision is not" - vllm-metal docs.

August's most technically interesting development is a body of community evidence pointing the opposite direction, at least for some architectures. Later testing in the merged vLLM PR thread, run on Nemotron-Cascade-2 (a hybrid Mamba model), found that value precision dominates reasoning quality: reasoning accuracy came in at 71.4% with 2-bit values, 85.7% with 4-bit values, and a full 100% with FP8 values, restoring baseline at 2x total KV compression even with keys held at 3 bits - vLLM PR #38479. The same tester's summary line deserves to be quoted because it names a failure mode benchmarks routinely miss: "cosine similarity can pass >0.99 while reasoning still breaks."

So which doctrine is right? The honest answer, and the one we are carrying into our own deployments, is that the asymmetry is architecture-dependent. A follow-up observation in the same thread noted that standard transformers tolerate value compression better while hybrid Mamba designs need value preservation, because information flows differently through their residual streams. The two results are in genuine tension: the preset ladder was designed around key protection and validated on standard transformers, while the strongest counter-evidence comes from a hybrid architecture that the merged implementation does not even officially support. Neither result invalidates the other; they map different regions of the same terrain.

The practical takeaway is not "flip your presets," it is stop trusting proxy metrics. Per-vector distortion bounds, cosine similarity, and even perplexity can all look clean while multi-step reasoning quietly degrades, which is the same lesson the compounding-error data in Section 16 teaches from a different angle. If your model is a standard transformer, k8v4's key-protective asymmetry remains the evidence-backed default. If you are working near the frontier of hybrid architectures, treat every published KV-quantization result as untransferred until you have run your own end-to-end reasoning evaluation, because the asymmetry that holds for one information-flow pattern demonstrably inverts for another. This finding did not exist when our July version shipped, and it refines a claim we made then; the scorecard in Section 18 logs it.

9. Qdrant 1.18: The Second Shipped Surface

The single biggest omission in our July version was not a nuance, it was a shipped product. On May 13, 2026, Qdrant published the details of TurboQuant support landing in Qdrant 1.18, bringing the algorithm to an entirely different workload: compressing stored embedding vectors for similarity search - Qdrant. We had covered TurboQuant's embedding-search math since March (the indexing-speed advantage over trained-codebook methods documented in Section 4 was always part of its pitch) and still missed that a major vector database had turned the math into a GA feature. Our adoption scorecard is this article's core franchise, and it was missing a row. That is now fixed, and the row changes the story's shape: TurboQuant is not "a vLLM feature," it is an algorithm with two independent production surfaces.

The Qdrant implementation exposes four operating points, and the compression ladder is steeper than anything in the LLM-serving world because embedding search tolerates more distortion than autoregressive generation. The options run from 4-bit at 8x compression through 2-bit at 16x and 1.5-bit at roughly 21x, down to 1-bit at 32x compression of the original float32 vectors.

The accuracy numbers are the reason this matters beyond a changelog entry. In Qdrant's published benchmarks across ten datasets, TurboQuant at 2-bit beats classic Binary Quantization at the same 16x storage by 9-24 percentage points of recall, and at 1-bit it beats vanilla BQ by 9-21 points at identical 32x storage. Against Scalar Quantization, TurboQuant's 4-bit mode reaches comparable recall at half the storage, and on some datasets (dbpedia-openai-ada among them) exceeds SQ by up to 4.6 percentage points. In plain terms: the same rotation-plus-codebook trick that makes 3-bit KV caches viable makes 1-bit and 2-bit embedding storage viable, displacing the binary quantization that had been the default aggressive option in vector search for years.

For anyone running retrieval infrastructure, the compounding effect with the LLM-serving side is the real prize. A production RAG stack pays for memory twice: once for the vector index that finds the context and once for the KV cache that holds it during generation. As of August 2026, TurboQuant compresses both ends of that pipeline, in shipped software, with no calibration step on either side. We walked through the retrieval half of this architecture in our guide to enterprise AI search with RAG and vectors; the update since that piece is that the aggressive-compression tier of vector storage now has a strictly better default than binary quantization for most datasets. Teams evaluating Qdrant 1.18 should treat the 2-bit mode as the interesting one: 16x compression with recall that previously required 4x-8x is the kind of shift that changes index-sizing spreadsheets, not just benchmarks.

There is also a Haystack-shaped footnote that shows the ecosystem filling the gap between the two surfaces. deepset maintains an official tutorial, "Compress the KV Cache with TurboQuant and Haystack," last updated June 15, 2026, which wires the community turboquant-vllm plugin's CompressedDynamicCache into a TransformersChatGenerator running Qwen/Qwen3-4B-Thinking-2507, and measures the cache dropping from 1,639 MiB to 435 MiB on an RTX 4090 - Haystack. A framework tutorial is a lagging indicator of real demand: nobody writes maintained walkthroughs for features their users are not asking about.

10. Local Inference: llama.cpp Said No, MLX Opened a Side Door

Our March version predicted that TurboQuant KV compression would likely become a standard llama.cpp feature and would automatically reach millions of local LLM users through Ollama. That prediction was wrong, and the way it went wrong is more instructive than most of what went right. Our July version then told the story as a clean rejection: llama.cpp closed the PR, Ollama inherits llama.cpp's decision, end of story. The rejection half remains true and re-verified. But the fuller record shows a door left ajar rather than welded shut, and a second storyline in Ollama's own repository that our July version missed entirely.

The technical work was real. PR #21089 implemented two CPU TurboQuant KV cache types: tbq3_0 at roughly 3.0625 bits per element with 5.22x compression versus f16, and tbq4_0 at 3.94x compression with quality comparable to the existing q4_0 cache type. The implementation used the Randomized Hadamard Transform, which suits CPUs well because Hadamard matrices contain only +1 and -1 values, avoiding floating-point multiplication in the rotation step. By the numbers, it worked.

On June 2, 2026, the maintainer closed the PR unmerged with a one-sentence verdict that deserves quoting in full: "so far no one has presented evidence that would justify the opportunity cost of adding TurboQuant support." The conversation was locked on June 3. The tracking feature request, Issue #20977, remains open with 277+ upvotes and no committed path forward, which means stock Ollama still has no native TurboQuant as of August 2026.

The reasoning behind the rejection is worth engaging with rather than dismissing, because it is a first-principles argument about maintenance economics, not a judgment that the math is wrong. Every KV cache type in llama.cpp multiplies the testing matrix across every backend the project supports: x86, ARM, Metal, CUDA, Vulkan, and more. The project already ships q4_0 and q8_0 cache types that deliver much of the practical benefit for typical local context lengths. Local users run shorter contexts than cloud serving fleets, so the KV cache is less often their binding constraint, and the marginal win of 5.22x over the existing 4-bit option shrinks further after accounting for the quality gradient documented in Section 6. From the maintainer's chair, a permanent maintenance obligation for a marginal, quality-risky gain is a bad trade. The llama.cpp decision and the vLLM decision are both rational, because the two projects serve users with opposite bottlenecks.

The nuance in the closure matters for anyone hoping to revive it. The same closing comment continues: "Anyways, closing this particular PR since it conflicts with procedure," and a second maintainer spelled out the bar for reconsideration: a new PR showing demonstrable gains over the existing cache quantization types, with full benchmarks per the contribution guidelines. That is not "never"; it is "not on this evidence." Maintenance economics can be repriced by sufficiently good numbers, and nobody has submitted them in the two months since.

The missed storyline is that Ollama did not just wait for llama.cpp. Two TurboQuant PRs ran in Ollama's own repo. The first, PR #15505, was a full native implementation: nine presets across CUDA, ROCm, and Metal, opened April 11, approved by one reviewer, then closed on May 30, not on quality grounds but because a separate refactor (PR #16031) moved GGUF inference out to an external llama-server, which re-routes any TurboQuant support through upstream llama.cpp, the project that had just said no. The implementation also measured real costs: decode throughput declines of roughly 30-73% depending on model and context on the tested RX 9060 XT, against VRAM savings up to about 50% at 65K context. The second PR, #15090, is still open as of this writing: tq3/tq4 cache types that take a 256K-context qwen3.5:9b session on an RTX 3070 Ti from 16-17 GB down to 8-9 GB, with the author reporting token generation actually up 6.5-12% at long contexts in this implementation.

The live thread inside that open PR is the side door. Discussion participants report that Ollama's maintainers intend to lean on MLX's implementation rather than a native one, with one contributor stating flatly that "MLX IS the library that is making TurboQuant support on Ollama 0.19 happen," cross-platform including CUDA, though others note the feature had not shipped in a release at the time of the comments. Treat the version number as intent rather than commitment; the structural point is that the local path routes around llama.cpp through MLX, not through it. Meanwhile the Apple-side tooling is already real: the vLLM project's Metal port maintains an official TurboQuant docs page with q8_0 through 2-bit key and value options, a default configuration delivering 2.56x compression, and aggressive settings approaching 4.92x - vllm-metal.

The community layer above that is livelier but demands honest labeling. The most visible repo, turboquant-mlx, implements fused-Metal-kernel TurboQuant as a drop-in for mlx-lm's KV cache and headlines "4.6x compression at 98% FP16 speed"; its own benchmark table, though, shows the mixed K8+V4 configuration on Qwen 2.5 7B at 32K context saving 18% of memory while decoding at 25.84 versus 35.75 tokens per second, roughly 72% of FP16 speed on that config. Both numbers can be true on different configurations, but the gap between headline and table is exactly the pattern this whole saga trains you to check for. There is even a community-quantized model sitting on the Ollama model library for users who want to try the output of this pipeline without building anything.

On the llama.cpp side, the gap did not stay empty either. A community fork combining Multi-Token Prediction speculative decoding with TurboQuant KV cache (exposed as --cache-type-k turbo3 and --cache-type-v turbo4) fits Qwen3.6-35B-A3B at 256K context on a single 24GB RTX 4090, with perplexity within about 1% of q8_0 - Hugging Face discussion. That configuration, a 35B-class model with a quarter-million-token window on a consumer gaming GPU, is exactly the local-inference dream the March hype promised. It exists. It just lives in forks, with fork-grade stability and no Ollama integration. For the tinkerer audience of our open-source personal AI guide, the honest August summary is: stock Ollama still has no TurboQuant, the llama.cpp route is dead pending better evidence, and the MLX route is where the actual commits are happening. Our July claim that local users would simply never get it gets downgraded to half-right in Section 18.

The strategic lesson generalizes beyond this one algorithm. Upstream maintainers are the real gatekeepers of local AI capability, and their acceptance criteria are opportunity cost and maintenance burden, not benchmark deltas. A compression technique can be provably near-optimal, officially adopted by hyperscalers, and still fail to reach millions of local users because one maintainer correctly judged the integration cost. Anyone modeling technology diffusion in the LLM stack should treat framework governance as a first-class variable, which is a mistake we will not repeat in our own predictions.

11. The LLM Quantization Landscape in Mid-2026

To understand where TurboQuant fits, you need the broader map. LLM quantization in mid-2026 spans four distinct categories: weight quantization, activation quantization, KV cache quantization, and native low-precision training. Each solves a different problem, and they are not interchangeable. TurboQuant lives entirely in the third category and composes with the other three.

Weight quantization remains the most mature category. GPTQ uses second-order information to minimize per-layer quantization error and reaches roughly 712 tokens per second with Marlin kernels at around 90% quality retention - JarvisLabs. AWQ, from MIT's Han Lab, protects the less-than-1% of weights that activation magnitudes mark as salient, reaching 741 tokens per second at roughly 95% quality retention, with over 19 million downloads on Hugging Face - MIT HAN Lab. GGUF dominates local deployment through llama.cpp and Ollama, with Q4_K_M as the recommended sub-format at a perplexity cost of just +0.0535 over FP16 - Enclave AI. At the extreme end, AQLM holds the sub-3-bit Pareto frontier using multi-codebook additive quantization - Yandex Research.

A useful calibration point for all of these: Red Hat ran over 500,000 evaluations comparing quantized and full-precision models and found that at 4-bit, for models above 7B parameters, the 95% confidence intervals of quantized versus full-precision performance overlap - Red Hat. Weight quantization at 4 bits is a solved problem for reasonably sized models. The open frontiers are below 4 bits, below 7B parameters, and in the cache.

Activation quantization is the harder problem, because activations have dynamic distributions with severe outliers: a few dimensions carry values 100x larger than the median, wasting quantization range. SmoothQuant transfers difficulty from activations to weights to enable W8A8. FP8 is the de facto data-center standard on H100 and H200, with Llama-3.3-70B at FP8 showing 99%+ quality recovery alongside 30% latency reduction - Oracle. The frontier moved decisively in 2026 with NVFP4, NVIDIA's 4-bit floating-point format with micro-block scaling. On June 26, 2026, NVIDIA published nvidia/Qwen3.6-27B-NVFP4 on Hugging Face: roughly 2.5x smaller on disk and GPU memory than 16-bit while scoring MMLU Pro 86.3 versus 86.1 for FP8 and GPQA Diamond 85.5 versus 86.0, Apache 2.0 licensed, running on both Hopper and Blackwell - Hugging Face. Four-bit weights and activations with FP8-parity scores is no longer a research claim. It is a downloadable artifact.

In the KV cache category, TurboQuant's original competitive set (KIVI at 2.6x, NVIDIA's calibration-hungry KVTC at 20x, codebook-based CommVQ, reasoning-focused PM-KVQ, and mixture-based MoQAE) has been joined by a post-TurboQuant successor wave that benchmarks against TurboQuant itself, covered in Section 12. The category also matured in self-knowledge: the key-value asymmetry Deep Infra documented is now standard design input rather than tribal knowledge, visible directly in the preset names that shipped, even as the hybrid-architecture evidence in Section 8 shows the asymmetry is not universal.

Native low-precision training remains the radical fourth path, and its status finally moved after months of stasis. Microsoft's BitNet b1.58 2B4T, the ternary model whose weights take only values in {-1, 0, +1}, is still the largest downloadable natively-trained ternary model as of August 2026, and no ternary model larger than 2.4B has been released; community-converted 1.58-bit models up to the 8-10B range (Llama3-8B-1.58, the Falcon3 family) remain conversions rather than native runs. But the ecosystem around it shipped twice in one July week: 1-bit embedding models (0.6B and 270M, with up to 2.28x speedup over F16) landed July 20, and the VibeASR.cpp speech-recognition engine followed July 23 - Microsoft BitNet. Our July version called this corner "essentially unchanged since March, which is itself news," a sentence that expired within two weeks; Section 18 grades it. The 100B ternary question that would make post-training compression obsolete remains completely open. Until someone spends frontier-scale compute on a ternary run, quantization of conventionally trained models, TurboQuant included, remains the only game deployable today.

The composability story is the practical conclusion of the landscape tour. A mid-2026 production stack can run NVFP4 or AWQ weights, FP8 or NVFP4 activations, and TurboQuant k8v4 cache simultaneously, because the three compression layers target disjoint memory domains. On a 70B-class deployment, that compounds to weights dropping from 140 GB to roughly 35-56 GB and the cache dropping by 2.6x, which frequently converts a multi-node deployment into a single-node deployment. That compounding, not any single technique, is why serving economics improved faster in the past year than raw hardware alone can explain, a dynamic that also shows up in per-token pricing across every major model family we have benchmarked, from GPT-5.6 to Kimi K2.6.

12. The Successor Wave: TurboQuant as the New Baseline

The clearest sign that TurboQuant mattered is not any adoption statistic. It is that within three months, new KV compression papers stopped benchmarking against KIVI and started benchmarking against TurboQuant. In research dynamics, becoming the baseline is the durable form of winning: the newest thing gets surpassed, but the reference point shapes every paper that follows.

The successor wave attacks TurboQuant from several distinct angles, each targeting one of its known weaknesses. TurboAngle (submitted March 29, 2026, five days after the Google blog post) quantizes angles in the Fast Walsh-Hadamard domain rather than coordinates, claiming near-lossless quality at 3.28-3.67 bits per element with perplexity degradation of just +0.0014 on Mistral-7B - arXiv 2603.27467. If those numbers survive independent replication, TurboAngle attacks the exact quality-at-3-bits gap that Section 6 documented.

RateQuant takes the information-theory route further, applying rate-distortion theory to allocate mixed precision per attention head instead of uniformly. At an average of 2.5 bits on Qwen3-8B, it cuts perplexity from 49.3 (KIVI) to 14.9, and beats QuaRot by 6.6 perplexity points - arXiv 2605.06675. The insight is that attention heads differ enormously in quantization sensitivity, so a uniform bit-width, TurboQuant's simplifying assumption, leaves quality on the table at extreme compression.

Three more directions round out the wave. IsoQuant replaces random rotations with hardware-aligned SO(4) isoclinic rotations, aiming at the GPU-efficiency criticisms of arbitrary rotation matrices - arXiv. InnerQ claims comparable accuracy at lower latency, targeting the 20-34% throughput deficit the vLLM study measured - arXiv. And PolyKV proposes a shared, asymmetrically compressed KV pool for multi-agent inference, where many agents reading overlapping context should not each pay full cache cost - arXiv. A sixth thread applies RoPE-aware bit allocation, adapting precision to rotary position embedding structure - arXiv.

None of these has shipped in a major serving framework as of August 2026 (a status we re-checked this week and predicted correctly in July, as Section 18 records), and the llama.cpp episode is a reminder that papers are not products. But the direction of travel is legible: uniform-precision, rotation-based scalar quantization (TurboQuant's design point) is being succeeded by allocation-aware methods that spend bits where sensitivity analysis says they matter. For practitioners the implication is patience on the aggressive end: if your workload needs sub-4-bit KV compression and TurboQuant's 3bit_nc quality cost is unacceptable today, the successor wave suggests that gap closes within a research cycle or two, and vLLM's fast merge of TurboQuant established that the pipeline from paper to production flag can now run in weeks rather than years.

13. Enterprise Adoption: AWS's Official TurboQuant Architecture

In March, there was no enterprise adoption to report because none existed. That changed on June 1, 2026, when AWS published an official Machine Learning blog architecture pairing TurboQuant with NVIDIA GPUDirect Storage on Amazon FSx for Lustre - AWS. A hyperscaler publishing a reference architecture is a different species of signal from a GitHub star count: it means solution architects are fielding enough customer demand to justify a supported pattern.

The numbers in the AWS design are among the most dramatic in this entire guide. On P5en instances (8x H200 GPUs with 141 GB of HBM3e each), the architecture lifts Llama 3.1 405B context capacity from roughly 82K tokens to 400K+ tokens, a nearly 5x gain achieved by compressing the KV cache rather than adding hardware. The storage half of the design cuts 405B model load time from 10-20 minutes to 6.4 seconds, a 169x improvement, by streaming weights directly from FSx for Lustre into GPU memory via GPUDirect, bypassing the host CPU bounce buffer entirely.

The pairing logic deserves a moment of first-principles attention, because it is not obvious why a compression algorithm and a storage interconnect belong in one architecture. Both attack the same underlying constraint: HBM is the scarcest resource in the serving stack. GPUDirect ensures the expensive memory is not idle while models load; TurboQuant ensures the loaded memory holds several times more context. Together they change the operational model for very large models: 405B-class deployments stop being permanently-resident singletons (because loading is no longer a 20-minute event) and start supporting long-context, high-concurrency workloads on hardware that previously could not hold the cache. Elastic serving of frontier-scale open models is an architectural pattern that simply did not exist a year ago.

The unanswered enterprise question is the one everyone asks first: does Google itself run TurboQuant behind Gemini? As of August 2026 there is no public evidence either way. Google published the papers and the blog post, never released code, and has said nothing about internal deployment. The circumstantial case (why publish a production-grade efficiency technique and then not use it?) is plausible in both directions, since publishing research and hardening it into a serving fleet are different investments. We flag it as unknown rather than speculating, and we note that the absence of official code makes independent verification impossible. What is verifiable: AWS builds on it, vLLM ships it, Qdrant ships it, and the open ecosystem now has a fully reproducible path to the same capability class, which for enterprise buyers evaluating RAG and enterprise search stacks matters more than what happens inside Google's fleet.

14. The Market Verdict, Act Three: SK Hynix After the Melt-Up

Our March version framed the memory-stock question through the Jevons Paradox: the 1865 observation by William Stanley Jevons that when coal engines became more efficient, total coal consumption increased rather than decreased, because lower cost per unit made more use cases economically viable. We argued the March 26 sell-off was an overreaction and that cheaper inference would expand rather than shrink memory demand. That was a prediction then. On the demand side it is an empirical result now, and the receipts are unusually clean. But the market story gained a third act after our July version went to press, and the third act is the humbling one.

Recall the crash first. On March 26, the day after most coverage broke, SK Hynix fell 6%, Samsung fell roughly 5%, Micron dropped 3.4%, and SanDisk and Western Digital followed - investing.com. The market logic was that 6x less KV cache memory means less High Bandwidth Memory demand. The sell-off was amplified by SK Hynix's record $7.97 billion ASML order placed just two days earlier, which suddenly looked like capacity nobody would need - 247 Wall St.

The reversal was total. By May 11, 2026, SK Hynix traded at an all-time intraday high of 1.949 million won, up roughly 197% year to date, with a market capitalization above $900 billion - TradingKey. HBM manufacturers were fully booked, the memory supply-demand gap sat at a 15-year extreme, and the shortage was projected to persist until 2028. Our July version declared this reversal "as decisive as market verdicts get" and ended the story there, at the top.

Scoring the March hot takes is worth doing by name, because forecasting accountability is rare in this space. Morgan Stanley published "TurboQuant leads to more intense computing rather than dimming demand" within days of the crash and was right - Seeking Alpha. TrendForce, with Morgan Stanley and Lynx Equity Strategies, published the most technically grounded call on March 26 itself: TurboQuant does not touch model weights or training HBM usage, and memory and flash demand would not fall for 3-5 years due to supply constraints - TrendForce. The panic sellers were wrong within six weeks. The pattern matches the DeepSeek episode of January 2025 almost exactly: efficiency breakthrough, hardware sell-off, full reversal, new highs.

The mechanics of TrendForce's correct call reward a closer look, because it was not a lucky contrarian bet but a decomposition anyone could have replicated. Their March 26 note pointed out that TurboQuant compresses only the KV cache, leaving model weights and training memory untouched, and that HBM demand is driven primarily by weight storage, training runs, and raw bandwidth rather than by cache capacity. The 6x figure therefore applied to a fraction of a fraction of total memory demand, at a moment when supply was already sold out years forward. The sell-off, in other words, priced a weights-scale effect onto a cache-scale technology. When the fear trade met the order books, the order books won: fully-booked HBM lines do not become less booked because one memory sub-domain got denser.

The first-principles explanation for why efficiency gains keep failing to dent hardware demand is worth internalizing, because this exact drama will repeat with the next compression paper. Inference demand is not fixed. It is elastic against cost, and latent demand (longer contexts, more concurrent agents, more retries, more reasoning tokens) vastly exceeds deployed supply at current prices. When compression cuts the cost of a KV-token, providers do not buy less memory; they serve more context per dollar and sell it. TurboQuant did not reduce the world's appetite for HBM. It increased the amount of useful work each HBM gigabyte performs, which raises the return on buying more of it. That is Jevons, mechanism and all, and after 2025-2026 it should be the default prior for any "efficiency breakthrough kills hardware demand" headline.

Now the third act, from reporting we opened this week - Yahoo Finance. SK Hynix listed on the Nasdaq on July 10, 2026 at an IPO price of $149 per share. The ADR opened at $170, spiked to roughly $195, and then reversed about 34% to trade near $128, below the offering price, within weeks. Then came second-quarter earnings: operating income of $41.7 billion, up 557% year over year, and revenue of $54.6 billion, up 257%, both of which missed analyst estimates of $44.2 billion and $57.9 billion respectively. A company growing operating income sixfold and getting sold off for it is the purest possible expression of what had been priced in.

The live debate has moved from "will compression kill memory demand" (the March question, answered no) to boom-bust cyclicality (the August question, unresolved). The same report frames it directly: "memory chips remain largely commodity products. When supply tightens, prices soar. When additional capacity enters the market, pricing often falls much faster than demand." Investors who spent the spring repricing an HBM shortage are now asking whether today's AI memory shortage becomes tomorrow's oversupply, which is a valuation question, not a demand question, and the distinction is the whole lesson.

So how do we grade our own Jevons call now that the chart has a third act? The demand half was right and remains right: TurboQuant demonstrably did not reduce memory demand, and 557% operating-income growth in the quarter after a "memory-killing" algorithm shipped in production is about as direct a falsification of the March panic as markets provide. What our July framing got wrong was collapsing two different claims into one: "the sell-off was an overreaction" (a demand claim, correct) and "the stock going up proves it" (a valuation claim, never safe). A stock can round-trip 34% while the underlying demand thesis stays perfectly intact, because price embeds expectations, not just fundamentals. The refined prior we are logging for the next efficiency-breakthrough panic: use Jevons to reason about demand elasticity, and refuse to let any stock price, in either direction, stand in as proof of an infrastructure argument.

15. The Hardware Ground Shifted: Blackwell Ultra and Native FP4

The hardware baseline our March version compared against has moved substantially, which changes the question TurboQuant answers. In March, "Blackwell ramping in 2026" was a forward-looking clause. As of this summer, Blackwell Ultra GB300 NVL72 systems are in production at Microsoft Azure, CoreWeave, and OCI, and SemiAnalysis InferenceX data published by NVIDIA shows up to 35x lower cost per token and 50x higher throughput per megawatt versus Hopper, with 1.5x lower long-context cost per token versus GB200 NVL72 and 2x faster attention at 128K-input, 8K-output workloads - NVIDIA.

So does 3-bit KV cache still matter when the hardware itself now speaks native FP4 and long-context costs are collapsing at the silicon level? The answer is yes, for a structural reason: hardware generations and compression algorithms multiply rather than compete. NVFP4 is a weights-and-activations format. It makes the model's compute cheaper and its parameters smaller, as the Qwen3.6-27B-NVFP4 release demonstrates at FP8-parity quality. It does nothing about the KV cache, which still scales linearly with context length and concurrency regardless of what format the weights use. A GB300 rack running NVFP4 weights with TurboQuant k8v4 cache holds meaningfully more concurrent long-context sessions than the same rack without cache compression, and at hyperscaler fleet sizes that difference is procurement, not tuning.

The composition also runs in the other direction: cheaper long-context hardware expands the workloads for which KV compression matters. At Hopper-era costs, million-token contexts were exotic. At GB300 costs (35x lower per token), they become an ordinary product feature, and at a million tokens the uncompressed KV cache is a 300+ GB object even before concurrency. The cheaper the hardware makes long context, the more of the serving budget the cache becomes, and the more valuable each multiple of cache compression gets. This is the same Jevons logic from Section 14 operating one layer down the stack, and it is why specialized inference silicon efforts, like the ones we profiled in our Taalas chip guide, treat memory capacity per dollar as the headline metric rather than FLOPS.

For buyers, the mid-2026 stack question is no longer "which single optimization do I adopt" but which combination the serving framework supports on the hardware you can actually get. The reference configuration that keeps appearing across vendor materials is NVFP4 or FP8 weights, native low-precision attention compute, and a quantized KV cache in the 4-bit-class range, with TurboQuant presets as the shipped implementation in vLLM. On allocation-constrained hardware (and with the HBM shortage projected into 2028, most hardware is allocation-constrained), cache compression is frequently the only lever a team can pull without waiting on a delivery queue.

16. Agent Workloads: Compounding Error Over Long Horizons

The March version of this guide argued that autonomous agents are TurboQuant's strongest use case, because agent sessions accumulate the longest KV caches in production. That argument survives, but it now comes with quantified danger data that did not exist in March, and the danger data changes the recommendation from "compress aggressively" to "compress conservatively and validate over full horizons." This section is also the part of the guide that comes from operating rather than reading: O-mega runs fleets of autonomous agents whose working sessions routinely accumulate six-figure-token contexts, because a single user request fans out into parallel browser sessions, computer-use sessions, and delegation chains, each holding its own long context concurrently. That workload shape, not any benchmark, is why we have tracked KV compression since March.

A practitioner study published May 3, 2026 measured what happens when KV cache quantization errors compound across agentic steps, where each step's slightly-degraded output becomes the next step's input context - dasroot.net. On visual planning tasks, accuracy fell from 94.2% at full precision to 87.3% at 4-bit to 76.1% at 2-bit. On robotic manipulation, the fall was steeper: 91.5% to 83.8% to 69.4%. Single-turn benchmarks never see this failure mode, because it only emerges when errors have dozens of steps to accumulate. The study's recommendation is the right one: validate on long-horizon trajectories, not single-shot evals, before deploying aggressive KV quantization in agent systems. It is the same "proxy metrics lie" lesson the keys-versus-values episode taught in Section 8, arriving from a different direction.

The workload-class guidance falls out of combining this study with the vLLM accuracy data from Section 6. Chat workloads (short contexts, single-turn or few-turn) tolerate any preset; the cache is small anyway, so k8v4 is free insurance rather than a necessity. RAG and retrieval workloads are the aggressive-preset sweet spot, since needle-in-a-haystack retrieval held at 100% on every preset and contexts are long. Reasoning models and hard math or coding workloads should stay at k8v4 or uncompressed, given the roughly 20-point drops the vLLM study measured on aggressive presets. And multi-step agents inherit the strictest constraint of all, because compounding turns a per-step nick into a session-level wound: k8v4, plus long-horizon evaluation on your own trajectories before and after enabling it.

There is one agent-specific argument for TurboQuant that the March version could not make: burst stability. Agent fleets have spiky concurrency, because a single user request fans out into parallel sub-agents, tool calls, and retries. The vLLM study found that under burst load, BF16 time-to-first-token blew out to roughly 17 seconds while TurboQuant configurations prevented the blowup entirely, because compressed caches stopped requests from queuing on memory. For an orchestration platform, TTFT stability under fan-out is often worth more than raw throughput, since a 17-second stall in one sub-agent serializes an entire plan: every downstream step that consumes its output waits, and an orchestration layer built for parallelism degrades into a slow sequential one. This is the calculus behind platforms like O-mega, where a single workforce-level task can spawn browser sessions, computer-use sessions, and delegation chains that each hold long contexts concurrently, the exact usage profile where cache capacity and burst behavior, not single-stream speed, decide how many agents one GPU fleet can sustain. The same fan-out economics show up across every platform we tested in our computer-use agent benchmarks.

The deeper architectural point for agent builders is that KV compression and context engineering are complements, not substitutes. Compression raises the ceiling on how much context an agent can afford to keep; it does not decide what deserves keeping. An agent that summarizes aggressively because memory was scarce can, post-TurboQuant, afford to retain full tool outputs and intermediate reasoning across more steps, which measurably improves multi-step task completion, as we documented in our guide to making LLMs autonomous. Every summarization is a lossy bet about what the future of the session will need, and a 2.6x cache multiplier lets the same architecture make that bet later and less often. The teams getting the most from cache compression are not the ones maximizing the compression ratio. They are the ones spending the reclaimed memory on longer useful horizons at conservative presets.

17. The August 2026 Decision Cheat Sheet

Our March version closed its practical section with a speculative "who benefits" analysis. Speculation is now unnecessary: every deployment path has a verified status, re-checked against live sources this week. This section replaces guesses with the actual August 2026 availability matrix and an honest expected impact per scenario.

Read the table as a statement about where the ecosystem placed TurboQuant, not about where it could theoretically run. The math is portable; the production-grade support is concentrated in exactly two places, vLLM for serving and Qdrant for search, with AWS's architecture (Section 13) as the managed-infrastructure expression of the same stack. If your serving is vLLM-based, adopting k8v4 is a one-flag change with a well-characterized cost, and the main work is running your own before-and-after evaluation. If your serving is anything else, adopting TurboQuant today means carrying patches or forks, and the honest advice for most teams is to take FP8 KV cache where it is native and wait for the successor wave to land in your framework rather than fighting your tooling.

Two scenario-specific notes complete the sheet. Teams on hybrid Mamba-attention models are excluded from the vLLM presets entirely, full stop, until the implementation gains hybrid support, and per Section 8 they should treat all published preset guidance as untransferred anyway. And teams whose workload is reasoning-dominated (agentic coding, math-heavy pipelines, long chain-of-thought models) should treat every row of the table one notch more conservatively than written, per the compounding-error data in Section 16. The cheat sheet optimizes for the median workload. Your evaluation set outranks it.

18. Grading Our Own Calls: Two Scorecards

A refresh that silently rewrites its own past claims is marketing. A refresh that grades them is journalism, so here is the scorecard on what earlier versions of this guide predicted, claim by claim. Nobody else on this topic does this, and it is the part of this page no aggregator can copy. The March-to-July record: two right, two wrong, and the wrongs are more instructive.

The two misses share a root cause worth naming: both extrapolated from enthusiasm signals (an active discussion thread, a community expectation) rather than from the incentives of the people who actually decide. Google had no commercial reason to release code once the ecosystem was reimplementing the paper for free, and llama.cpp's maintainer had every reason to protect a sprawling backend matrix from a marginal feature. The two hits share the opposite property: they reasoned from structural incentives (vLLM's users are memory-constrained cloud operators who feel KV pressure daily; memory demand is elastic against cost). The meta-lesson: weight incentive analysis over sentiment analysis, and treat maintainer economics as load-bearing.

We also carry forward two smaller factual corrections from the July version for the record. The Google Research blog post is dated March 24, 2026, not March 25. And the "zero accuracy loss" framing we repeated from the paper's benchmarks should have carried its scale qualifier (models up to 8B, specific configurations) in the headline rather than in a caveats section. Both corrections are incorporated throughout this refresh.

Now the new franchise entry: the July 8 version graded against what four more weeks of verified evidence shows. The grading is stricter this round, because several July claims were subtler than the March ones.

Two of these deserve the longer post-mortem, because each encodes a repeatable mistake. The Qdrant miss is the worse one: the information was public for two months, in a first-party engineering blog, from a company we cover in our enterprise-search work. The failure mode was scope blindness, searching for "TurboQuant + LLM serving" news and never re-asking whether the algorithm had crossed into an adjacent workload. The fix is structural rather than motivational: our adoption scans now track the algorithm, not the use case. The BitNet miss is subtler; "unchanged, which is itself news" was true when written and is exactly the kind of sentence that rots silently, because nothing alerts you when a negative claim expires. The July artifacts do not change the headline fact that no natively trained ternary model beyond 2.4B exists, but an ecosystem shipping new artifacts twice in one week is not "unchanged," and we wrote "unchanged."

For readers keeping score across both cycles: March-to-July taught us to weight maintainer incentives over community enthusiasm (that lesson held; the llama.cpp door remains shut for exactly the predicted reasons). July-to-August adds two more rules to the ledger: negative claims need expiry dates, and adoption tracking must follow the algorithm across domains rather than staying inside the use case where you first met it. The SGLang entry gets an honorable mention for sharpening rather than changing: what we described in July as one stalled community effort is more precisely two abandoned attempts, the closed-as-inactive issue #21618 and a draft PR with 42 passing unit tests, Triton kernels, and no end-to-end benchmarks, which together say something specific: the hard part was never the quantization math, it was proving fleet-grade wins inside somebody else's serving engine.

19. What Comes Next

The formal milestones that defined "next" in March have all occurred: the ICLR poster happened, the framework integrations resolved (two merges, one rejection, one stall), the first at-scale accuracy studies published, and the market repriced twice. What remains genuinely open in August 2026 is a shorter, sharper list, and each item has a testable resolution we will grade in the next refresh.

The nearest-term question is whether the successor wave ships. TurboAngle's angle-domain quantization - arXiv and RateQuant's per-head bit allocation - arXiv both attack the aggressive-preset quality cliff documented in Section 6, and vLLM's three-week TurboQuant merge proved the paper-to-flag pipeline can run fast when the engineering arrives clean; zero of the successor papers have a framework merge as of this week. Close behind it: whether the MLX route reaches stock Ollama, which would make our "half-right" local grade fully wrong in the best possible way, and whether the keys-versus-values asymmetry gets a systematic study across architectures instead of scattered PR-thread evidence.

The medium-term question is hybrid architecture support. The vLLM exclusion of Mamba-plus-attention models is not a fundamental limitation of the math; it is an engineering gap in how compressed caches interact with state-space layers. As hybrid designs spread through the open-model ecosystem, whichever KV compression implementation first supports them inherits a growing share of deployments, and the value-precision evidence from Section 8 suggests the winning implementation will need hybrid-specific presets, not just hybrid compatibility.

The long-term question is unchanged, and it is still the biggest one: native low-precision training. BitNet remains capped at 2.4B natively-trained parameters, with efficient kernels, fresh July artifacts, and no frontier-scale ternary model - Microsoft BitNet. If someone funds a 100B ternary run and it holds quality, the entire post-training compression stack becomes less relevant, TurboQuant included. Until then, compression of conventionally trained models is where all deployable value lives, and models keep getting bigger faster than they get more efficient per parameter, as every recent open-weights flagship from DeepSeek V4 to GLM-5.2 demonstrates.

The decision framework, compressed to four sentences and current as of this week. If you serve LLMs on vLLM, enable turboquant_k8v4, treat it as a capacity and burst-stability upgrade rather than a speed upgrade, and validate on your own evaluation set, over full trajectories if your workload is agentic. If you run vector search at scale, benchmark Qdrant 1.18's 2-bit mode against your binary-quantized baseline, because a 9-24 point recall edge at equal storage is worth an afternoon of testing. If you run local models, stock tooling still has nothing official; watch the MLX thread rather than the llama.cpp one. And when the next compression breakthrough crashes a hardware stock, remember that this cycle taught both halves of the lesson: demand panic was wrong in March, and the price-as-proof victory lap was wrong in July.

The larger lesson of TurboQuant's first 134 days is about how infrastructure claims should be consumed. The paper was honest; the headline economy around it was not; the truth arrived through adversarial engineering: a serving team's comprehensive study, a maintainer's cost-benefit rejection, a vector database's quiet GA feature, a hyperscaler's reference architecture, and a market that repriced twice. Every layer of that verification machine worked. TurboQuant emerged from it smaller than the hype and considerably more real: an officially shipped, honestly bounded, provably near-optimal tool for the specific memory wall that long-context AI keeps hitting. That is what a successful research artifact looks like in 2026, and the KV cache, growing linearly with every token every agent processes, guarantees the problem it solves is not going anywhere.

---

Written by Yuma Heymans (@yumahey), founder of O-mega and co-founder of HeroHunt.ai. The KV cache numbers in this guide are not abstract to him: agent fleets that fan a single request into parallel long-context sessions hit this exact memory wall, which is why every claim here gets re-checked against live sources instead of press cycles.

This guide reflects the LLM compression landscape as of August 5, 2026, and supersedes our March and July analyses. Serving frameworks, presets, and market facts move fast: verify current availability against the linked primary sources before making infrastructure decisions.