← Back to blog

CacheClip: making RAG prefill 3.33× faster without giving up the answer

How a 135M-parameter helper model decides which KV cache entries the 14B model actually needs to recompute.

August 2026 · a walkthrough of arXiv:2510.10129

If you run a RAG system in production, prefill is where your latency budget goes to die. A single request carries four to sixteen thousand tokens of retrieved context, attention cost grows quadratically with that length, and the user sees nothing until the first token comes out.[1]

The obvious fix is to precompute the KV cache for every chunk in your corpus once, then glue the caches together at request time. The obvious fix does not work. Chunks encoded in isolation never attended to each other, and each one carries its own attention sink. Quality falls off a cliff on exactly the queries RAG exists to answer: the ones that need evidence from more than one document.

CacheClip is a framework from Intel that fixes both problems without touching the weights of the model you are serving. Its central move is unusual: it hands the question of "which tokens matter" to a separate 135M-parameter model running on the CPU, and that tiny model turns out to be a better predictor of the 14B model's deep-layer attention than the 14B model's own early layers.

This post is in eleven parts:

  1. The TTFT problem in RAG. Why long retrieved context is a systems problem, not a modeling one.
  2. Two failures, not one. Independent chunk caches break in two independent ways.
  3. The reuse landscape. Prefix caching, direct concat, calibration, selective recompute, and what each one costs.
  4. Why token selection is the hard part. And why CacheBlend's early-layer signal is the wrong signal.
  5. Small models know what big models look at. The empirical result the whole design rests on.
  6. How CacheClip works. Shared prefix, auxiliary-guided selection, sliding-window grouping, CPU/GPU split.
  7. Results. RULER and LongBench, plus the recomputation ratio and sequence length sweeps.
  8. Which piece does the work. The two components own different benchmarks.
  9. Where the 3.33× comes from. The latency breakdown at 16K tokens.
  10. When not to use this. Honest limits.
  11. Takeaways.

This is a systems post. It assumes you know what a KV cache is and roughly how a transformer prefill works. It does not assume you have read the paper. Every number below comes from the paper[1] and I flag the one place where the paper's own numbers disagree with each other.

1 · FULL ATTENTION PREFILL Q C1 C2 C3 C4 Primary LLM prefill compute K and V for all ~16K tokens 5.641 s TTFT baseline 2 · CACHECLIP Q C1 C2 C3 C4 KV caches precomputed offline reuse ~80% recompute ~20% 1.695 s 3.33× faster the tiny model picking that 20% runs on the head node CPU, so the GPU never sees the extra work
Figure 1: The whole pitch in one picture. At 16K input tokens on a single NVIDIA L20, full prefill costs 5.641 s. CacheClip reuses precomputed chunk caches and recomputes roughly a fifth of the tokens, reaching first token in 1.695 s.

1. The TTFT problem in RAG

RAG makes the prompt long on purpose. That is the whole mechanism: you prepend retrieved evidence so the model can answer from something other than its weights. The cost lands entirely on prefill.

  • Typical RAG request: 4K to 16K input tokens. That is the range the paper targets, and it matches most production retrieval setups.[1]
  • Attention is quadratic in sequence length. Going from 4K to 16K is 4× the tokens and roughly 16× the attention work.
  • The user waits for all of it. Time to first token is dominated by prefill, not decode.
  • And most of that work is repeated. A knowledge base is finite. The same chunks get retrieved over and over across different queries, and you pay full price for them every time.

That last point is the opening. If chunk 47 of your corpus gets retrieved a thousand times a day, computing its KV cache a thousand times is pure waste. Precompute it once, store it, load it on demand.

Key idea Precomputing chunk caches is a storage-for-compute trade. The trade is only worth it if the reconstructed cache behaves like a cache the model would have built itself. That equivalence is exactly what breaks, and the rest of this post is about repairing it cheaply.

2. Two failures, not one

Here is the part that took me a second read to appreciate. Concatenating independently encoded chunk caches produces two distinct defects with two distinct causes. Most prior work fixes one and leaves the other standing.

  1. Repeated attention sinks. Transformers dump disproportionate attention mass on the first few tokens of a sequence, a well documented effect.[4] Encode each chunk on its own and each chunk grows its own sink. Concatenate eight chunks and you have eight sinks in a sequence that should have one. The resulting attention distribution looks nothing like what the model saw during training.
  2. Missing inter-chunk attention. When chunk 3 was encoded, chunk 1 did not exist. No key or value in chunk 3 was ever influenced by chunk 1. The query tokens can still attend across everything at generation time, but that is a single shallow pass. It does not reconstruct the rich cross-document dependencies that full attention builds layer by layer.

These are orthogonal. One is a distributional problem in the first few positions. The other is a missing-information problem across the entire sequence. CacheClip's contribution is that it maps a separate, cheap mechanism onto each.

chunks encoded in isolation, then concatenated FAILURE A · repeated attention sinks FAILURE B · missing cross-chunk attention ■ C1 C2 C3 C1 C2 C3 three chunks, three sinks. The model expects one. Attention mass lands where training never put it. no key or value in C2 was ever influenced by C1. Multi-hop questions have nothing to hop across. MECHANISM · shared prefix + position IDs 1. prepend the same prefix to every chunk offline 2. at concat time keep only the first copy of it 3. renumber positions into one increasing range MECHANISM · selective recomputation 1. aux LLM scores chunk tokens against the query 2. top-k tokens grouped into contiguous windows 3. primary LLM recomputes only those, in context CacheClip = both, at once the ablations confirm the split: each mechanism dominates a different benchmark
Figure 2: Two independent defects, two independent repairs. Black blocks on the left mark the first tokens of each chunk, where the sink forms. Crossed dashed arcs on the right mark attention paths that were never computed. Reading this figure is the fastest way to understand why calibration alone or recomputation alone is not enough.

3. The reuse landscape

Four families of approach exist. Each one accepts a different loss.

PREFIX CACHING DIRECT CONCAT CALIBRATION SELECTIVE RECOMPUTE D1 D2 D3 D1 D2 D5 reuse stops at the first difference 3 chunks → 3 sinks, no cross arcs 1 sink, still no cross arcs 1 sink, cross paths restored Needs an identical prefix. In RAG the retrieved set and its order change every query, so reuse collapses to the first chunk or two. Fastest possible option. Pays for it with both failures at once: repeated sinks and zero cross-chunk attention. Shared prefix, keep one copy. Fixes the sinks and costs almost nothing. Cannot recover information that was never computed. Recompute a small subset in full context. Restores real cross-chunk paths. Only as good as the tokens you choose. That is the game. black block = attention sink · dashed arc with × = attention path that does not exist · green arc = path restored by recomputation red cell = token recomputed in full context · blue cell = cache entry reused as-is
Figure 3: The four reuse strategies drawn with a shared visual vocabulary. Reading left to right, each panel keeps more of full attention's behavior and costs more to run. CacheClip lives in the fourth panel and its entire contribution is choosing which cells go red.

A little more detail on each, since the trade-offs matter when you are picking one for a real deployment.

  • Prefix caching (vLLM[7], SGLang[8], RAGCache[6]). Reuses the cache when a new request shares a literal prefix with an old one. If the last request used [D1, D2, D3] and this one uses [D1, D2, D5], you reuse [D1, D2] and recompute everything after. Correct by construction, but the reuse condition is brutal in RAG: reorder the retrieved set and reuse evaporates. Caching the same chunk at multiple positions means storing multiple versions of it.
  • Direct concatenation (PromptCache[5]). Precompute each chunk independently, glue the caches, fix up positional embeddings. Enormous speedup, both quality failures intact.
  • Calibration (APE[3], Zhang et al.[12]). Prepend a shared prefix to every chunk at encode time, keep only the first copy at concat time. Removes the duplicate sinks. APE also rescales attention temperature to sharpen focus. Cheap and effective for failure A, useless for failure B.
  • Selective recomputation (CacheBlend[2], Cache-Craft[13]). Pick a subset of tokens, recompute their KV entries with the full concatenated context visible, overwrite. This genuinely restores cross-chunk paths. It is the right family, and everything now depends on the selection rule.
  • Finetuning (Block-Attention[9], TurboRAG[10], KVLink[11]). Teach the model to work with block-diagonal attention. Works well, but you are now retraining a 14B model, curating datasets that balance chunk configurations, and repeating the whole exercise whenever the base model changes.

4. Why token selection is the hard part

Selective recomputation only works because attention is sparse. That part is settled.

  • The top 10 to 20% of tokens carry the majority of attention weight.[14][15]
  • This holds across model scales and task types.
  • So if you can find those tokens, recomputing them recovers most of the value of full attention.
  • But sparsity patterns are query-dependent. Which tokens matter changes with the question being asked, so you cannot pick them offline.

Which puts you in an awkward position at prefill time.

Note: prefill is not decode During decode you already hold a full KV cache and can compute exact attention scores. Methods like H2O[14], SnapKV[15], Quest[22] and PyramidKV[23] all assume that. In a cache-reuse prefill you hold only local per-chunk caches, so exact scores over the concatenated context do not exist yet. The selector has to be query-aware and cheap and blind to full-context attention, all at once.

CacheBlend's answer was to use the primary model's own early layers: fully recompute layer 1, partially recompute layer 2, compare the resulting value matrix against the precomputed one, and treat the largest deviations as the tokens that matter.

The paper argues this is the wrong signal, for a reason that is well established in the interpretability literature.[1]

  • Shallow transformer layers mostly capture local syntax.
  • Deep layers carry long-range semantic dependencies and task-oriented reasoning.
  • Cross-chunk dependencies are exactly the long-range kind, so they live in deep layers.
  • Therefore the tokens that light up in layer 1 are systematically not the tokens that matter in layer 40.

This is a bias, not noise. You cannot fix it by increasing the recomputation budget, and the results section shows what that looks like in practice: CacheBlend's RULER average actually gets worse as you recompute more, down to 52.93 at 30% from 77.43 at 10%.

5. Small models know what big models look at

The paper's key empirical claim, and the thing the whole design rests on: a tiny model's last layer predicts a big model's last layer better than the big model's own first layer does.

The experiment:

  1. Four model pairs, one same-family and three cross-family with different architectures and different tokenizers.
  2. 200 samples from 2WikiMultihopQA[21] at each of 1K, 2K, 4K, 8K and 16K tokens.
  3. Extract the head-averaged last-token attention distribution from the first and last layer of each model.
  4. Project everything into a shared character space, spreading each token's attention weight uniformly across the characters it covers. This is what makes cross-tokenizer comparison fair. SmolLM2 emits about 1050 tokens where Qwen2.5 emits 1024 for the same text.
  5. Measure the Jaccard index of the top-20% highest-attention character positions.

Jaccard on top-20% positions is the right metric here because it measures exactly the thing the system needs: do the two distributions agree on where the important positions are. It does not care whether the distributions have the same shape.

Model pair 1K 2K 4K 8K 16K
A · Qwen0.5B → Qwen14B0.41 / 0.230.41 / 0.220.39 / 0.220.39 / 0.220.39 / 0.22
B · SmolLM → Qwen14B0.30 / 0.230.29 / 0.220.29 / 0.220.28 / 0.220.31 / 0.22
C · SmolLM → LLaMA8B0.31 / 0.130.31 / 0.130.29 / 0.120.28 / 0.120.28 / 0.11
D · SmolLM → Ministral8B0.32 / 0.190.32 / 0.170.31 / 0.170.29 / 0.160.34 / 0.16

Jaccard index of top-20% character positions. Each cell reads aux last layer vs primary last layer / primary first layer vs primary last layer. Higher is better on the left of each pair.

What to take from it:

  • All 16 configurations go the same way. The auxiliary model's last layer beats the primary model's own first layer, every time.
  • The margin is large on cross-family pairs. Pair C is 0.31 vs 0.13 at 1K. A 135M model from a different architecture family with a different tokenizer is more than twice as good a predictor of LLaMA-3.1-8B's last-layer focus as LLaMA's own layer 1.
  • It holds as context grows. The numbers barely move from 1K to 16K.
  • Same-family is better still. Pair A hits 0.41, and that shows up downstream: Qwen2.5-0.5B is the stronger auxiliary model in the end-to-end results.

The paper also reports KL divergence as a secondary metric, and there the picture is mixed: alignment is clean when the primary model is the same (pairs A and B) and reverses for pairs C and D. The explanation is reasonable. Different architectures produce distributions with different peakiness, which inflates KL without moving the peaks. Since the system only needs positions, not shapes, Jaccard is the metric that decides.

Key idea Transformers of wildly different sizes and families converge on a similar notion of "which tokens are important" in their final layers. Depth in the network matters more than parameter count for this particular judgment. That is a genuinely interesting result independent of CacheClip, and it is what makes a 135M model a legitimate oracle for a 14B one.

6. How CacheClip works

Two phases. Offline you build caches. Online you assemble and patch them.

OFFLINE · ONCE PER CHUNK shared prefix (the system prompt) text chunks C1 … Cn, 1000 tok, 50 overlap prepend prefix to every chunk primary LLM Qwen2.5-14B-Instruct auxiliary LLM SmolLM2-135M-Instruct primary KV store P+C1 · P+C2 · P+C3 … auxiliary KV store P+C1 · P+C2 · P+C3 … both stores are built once and shared across every request ONLINE · PER USER QUERY user query RAG retrieval → chunk ids TOKEN SELECTION · runs on CPU 1. load precomputed aux KV caches 2. batch [chunk_i + query], read last-layer attn 3. average over query rows → one score per token 4. top-k, then sliding-window grouping Intel Xeon 6554S, AMX-accelerated tokenizer remap aux ids → primary ids CACHE PREPARATION · GPU 1. concatenate retrieved chunk caches 2. drop every redundant prefix copy 3. rearrange position IDs into one range result: one sink, monotonic positions SELECTIVE RECOMPUTATION · GPU global KV cache, ~20% of entries overwritten ■ reuse precomputed ■ recompute in full context first token out selection overlaps with KV cache loading, so most of its latency is hidden
Figure 4: The full CacheClip pipeline, redrawn from Figure 4 of the paper. Red outlines are compute, blue outlines are memory. The two online branches run on different devices and overlap: the CPU is scoring tokens while the GPU is loading and stitching caches.

6.1 Shared prefix and position IDs

This is the cheap half. It costs essentially nothing at request time.

  1. Offline: prepend a fixed prefix to every chunk before encoding it. The default is the system prompt, which you were going to send anyway.
  2. Online: concatenate the retrieved caches and keep the prefix only from the first chunk. Every other copy is dropped.
  3. Result: the assembled sequence has exactly one attention sink, at position 0, which is what the model was trained on.
  4. Then fix the positions. Each chunk was encoded starting right after the prefix, so they all share the same position offsets. Dropping the duplicate prefixes leaves repeated position IDs. CacheClip renumbers so each chunk occupies a unique continuous range and the sequence is monotonic, matching full attention mode.
Note This step is borrowed, not invented. APE[3] and Zhang et al.[12] introduced the shared-prefix trick. CacheClip's contribution is combining it with a much better recomputation strategy, and the ablation shows the two are complementary rather than redundant.

6.2 Auxiliary-model-guided token selection

The expensive-sounding half, made cheap by the same caching trick applied twice.

aux KV cache prefix + chunk_i user query auxiliary LLM final layer only, all chunks in one batch attention from query tokens to chunk tokens columns = chunk tokens (N) rows = query tokens (|Q|) average down each column one importance score per chunk token top-k cut selected indices k = recomp% × N, in the aux tokenizer tokenizer remap aux token boundaries ≠ primary token boundaries aux primary → primary-side indices go to sliding-window grouping (Figure 6)
Figure 5: How a token gets selected. The auxiliary model only ever produces one thing the system cares about: a single importance score per chunk token. Everything upstream is about producing that vector cheaply, and everything downstream is about turning it into a safe set of indices for a different tokenizer.

Step by step:

  1. Precompute the auxiliary caches too. Every chunk gets a KV cache in the auxiliary model, offline, exactly as it does in the primary model. This is what makes online selection cheap.
  2. Batch chunk-plus-query pairs. Online, build [chunk_1 + query], [chunk_2 + query] and so on, and run them as one batch. Only the query tokens are actually computed. The chunk side is loaded from cache.
  3. Read the final layer. Take the attention matrix from the auxiliary model's last layer.
  4. Slice out what matters. Keep only attention from query tokens to chunk tokens. Attention among query tokens and attention to the shared prefix are discarded. What is left has shape [query_size, total_chunk_size].
  5. Collapse to a vector. Average over the query dimension to get one importance score per chunk token, shape [total_chunk_size].
  6. Take the top k. Where k is set by the recomputation ratio you configured.

The scoring step is just a column mean over the query rows of the last-layer attention matrix:

score(j) = (1 / |Q|) * sum over i in Q of  A_last[i, j]

  A_last   last-layer attention of the auxiliary model
  Q        the query token positions (rows)
  j        a chunk token position (column)
  N        total_chunk_size, the number of columns
  k        round(recomp_ratio * N)

Plain language: for each token in the retrieved context, ask "on average, how hard did the query look at you." Sort. Keep the top slice.

6.3 Sliding-window grouping

This is the component I would have skipped if I were designing the system, and it turns out to be the single biggest contributor on RULER. It exists because recomputing scattered individual tokens actively damages the cache.

  • Tokenizers split entities. A 7-digit number or a UUID is several tokens.
  • If you recompute token 3 of a 5-token UUID and leave the other four holding stale, locally-computed values, that entity's cache is now internally inconsistent.
  • The model reads a fragment that is half in-context and half out-of-context, and produces corrupted output.
  • Grouping forces recomputed tokens to arrive in contiguous runs, so entities stay whole.
STEP 1 · candidates from the auxiliary model's top-k red dot = candidate token. Window size w = 8, density threshold τ = 5. A window is only evaluated if it starts on a candidate. 6 candidates in 8 ≥ τ keep the whole window, including the gaps 3 candidates in 8 < τ isolated, drop them STEP 2 · the recomputation set ■ reuse precomputed KV ■ recompute in full context, as one contiguous run This is why grouping matters more than it looks. A 7-digit number or a UUID is several tokens. Recompute part of one and the cache holds a half-updated entity, which produces corrupted output. Grouping trades a little precision for entity-level consistency, and RULER rewards it heavily.
Figure 6: Sliding-window grouping with the paper's defaults, w = 8 and τ = 5. The dense cluster on the left passes the density test and is promoted wholesale, gaps included. The sparse candidates on the right fail it and are discarded rather than recomputed in isolation.

The rule, precisely:

  1. Scan the sequence with a window of size w (default 8) and step 1.
  2. Only evaluate a window whose first position is a candidate.
  3. Count candidates inside the window. That is the local density.
  4. If the count meets or exceeds τ (default 5), add every token in the window to the recomputation set, including the non-candidates.
  5. Candidates never covered by a passing window are treated as isolated and dropped entirely.

Point 5 is the counterintuitive one. The system throws away tokens the auxiliary model said were important, on the grounds that recomputing them alone would do more harm than good.

candidates = top_k(scores, k = recomp_ratio * N)
recompute  = empty set

for t in range(N - w + 1):
    if t not in candidates:
        continue                        # window must start on a candidate
    window  = range(t, t + w)
    density = count(j in window if j in candidates)
    if density >= tau:
        recompute |= window             # gaps included, entity stays whole

# candidates never covered by a passing window are dropped

primary_idx = remap_tokenizer(recompute, aux_tok, primary_tok)
primary_llm.recompute(primary_idx, position_ids = full_attention_positions)
global_kv.overwrite(primary_idx)

Two details in that last block are easy to miss. The indices have to be remapped between tokenizers, because the auxiliary and primary models generally do not agree on token boundaries. And the recomputation uses position IDs consistent with full attention mode, not the chunk-local ones the cache was built with. Otherwise you would be patching in entries that disagree with their neighbours about where they are in the sequence.

6.4 The CPU/GPU split

Adding a second model to an inference path usually means adding GPU memory pressure and contention. CacheClip sidesteps that by putting the auxiliary model somewhere the primary model is not.

  • The auxiliary model runs on the head node's CPU. Those cores are typically idle while the GPU does prefill.
  • No extra GPU memory, no extra GPU compute. Which also means no workload imbalance across cards in a multi-GPU deployment.
  • The FLOP count is small. With chunk-side KV caches precomputed, selection only processes the query tokens.
  • AMX helps. The paper runs on 5th-gen Intel Xeon 6554S and notes that AMX[20] accelerates the matrix ops involved.[19]
  • It overlaps. Token selection runs concurrently with the GPU loading the primary KV caches, hiding most of its 0.238 s.
  • The ratio is a runtime knob. You can dial recomputation up or down per request depending on whether that request cares more about latency or accuracy.

7. Results

Setup, so you know what the numbers mean:

  • Primary model: Qwen2.5-14B-Instruct. Auxiliary models: SmolLM2-135M-Instruct[18] and Qwen2.5-0.5B-Instruct.
  • Hardware: NVIDIA L20 GPUs, head node on 5th-gen Intel Xeon EMR 6554S.
  • Benchmarks: RULER's retrieval category[16] (eight needle-in-a-haystack variants, scored by average reference coverage) and LongBench[17] on multifieldqa_zh, 2wikimqa and hotpotqa.
  • Chunking: 1000 tokens with 50-token overlap. The retriever is bypassed and all chunks are used, so retriever quality does not confound the comparison.
  • Baselines: full attention, direct reuse, APE, CacheBlend.
100 80 60 40 10%40%70%90% 4K8K16K full attn 99.41 RULER average vs recomputation ratio Qwen2.5-14B, input 8192 tokens RULER average vs input length recomp% = 20% ■ CacheClip (Qwen2.5-0.5B aux) ■ CacheClip (SmolLM2-135M aux) ■ CacheBlend ■ APE -- full attention Left: CacheClip is already near its ceiling at 10 to 20% recomputation. CacheBlend gets worse before it gets better, then jumps at 90% when recomputation is broad enough to stop fragmenting multi-token entities.
Figure 7: Redrawn from Figure 6 of the paper, plotted from the appendix tables. Left, RULER average against recomputation budget at 8192 tokens. Right, RULER average against input length at a fixed 20% budget.

Reading the left chart:

  • CacheClip saturates immediately. The curve is nearly flat from 10% onward, which is the signature of a selector that finds the most valuable tokens first. That is attention sparsity paying off exactly as predicted.
  • CacheBlend is U-shaped. It scores 77.43 at 10%, drops to 52.93 at 30%, and does not recover until very high budgets. More recomputation making quality worse is a strong signal that the selection rule is not just imprecise but actively harmful, because partial recomputation fragments multi-token entities.
  • The 90% jump is not a win. Everything converges near full attention at 90% because you are barely reusing anything. It tells you nothing about the useful operating range.
  • Auxiliary model quality matters. Qwen2.5-0.5B (green) sits above SmolLM2-135M (blue) almost everywhere, matching its higher Jaccard alignment in Table 1.

Reading the right chart:

  • Going from 4K to 16K at a fixed 20% budget, APE drops 24.5 points and CacheBlend drops 21.0.
  • CacheClip drops 12.5 with SmolLM2-135M and 14.6 with Qwen2.5-0.5B.
  • At 16K the baselines converge around 58 while CacheClip holds 72.9 and 75.4.
  • This matters because longer contexts are where cache reuse is most valuable, and it is also where naive selection degrades fastest.

On LongBench, which is closer to real workloads than synthetic needle retrieval:

Method multifieldqa_zh 2wikimqa hotpotqa
Direct reuse49.4437.7044.63
APE59.7038.3445.29
CacheBlend, 20%57.3441.0844.11
CacheClip, 20%, SmolLM2-135M62.4341.4052.12
CacheClip, 20%, Qwen2.5-0.5B61.3746.0455.74
Full attention64.9354.3659.71
  • hotpotqa is the interesting column. It is multi-hop QA, so it needs cross-chunk reasoning, which is precisely what selective recomputation restores. CacheClip with SmolLM2 beats CacheBlend by 8.0 points and APE by 6.8 there.
  • With the Qwen2.5-0.5B auxiliary model, CacheClip reaches 91.1% of full-attention quality averaged across the three datasets.
  • Averaged, CacheClip with SmolLM2 leads CacheBlend by 4.5 and APE by 4.2 points.
  • The gap to full attention is still real. 2wikimqa at 46.04 against 54.36 is not close. This is a speed-quality trade, not a free lunch.
Note: one inconsistency in the source The paper's RULER discussion cites 96.00 on the multivalue task at 20% recomputation, sourced from its Figure 5. The appendix table for the same configuration (Qwen2.5-14B, 8192 tokens, 20%, SmolLM2-135M auxiliary) reports 72.00 on multivalue, and 83.50 with the Qwen2.5-0.5B auxiliary. The comparison against CacheBlend's 42.97 is consistent either way, and the qualitative conclusion holds, but I have plotted the appendix tables above rather than the figure because they are unambiguous about which configuration produced them.

8. Which piece does the work

The ablation is the most useful table in the paper, because it shows the two mechanisms are not interchangeable. Each one owns a different benchmark, for a reason that maps cleanly back to Figure 2.

RULER, 8192 tok, 20% recomp multivalue Average
Neither component25.2540.47
Shared prefix only26.2538.66
Grouping only74.5081.28
Both72.0081.50
Full attention96.2599.41
LongBench, 20% recomp 2wikimqa hotpotqa Average
Direct reuse, no recomputation37.7044.6343.92
Recompute only37.8447.6149.40
Recompute + shared prefix42.7048.9650.91
Recompute + grouping40.9047.4549.51
All three41.4052.1251.98
  • On RULER, grouping is everything. It moves the average from 40.47 to 81.28, a gain of 40.81 points. Shared prefix adds 0.22 on top. RULER's needles are numbers and UUIDs, so entity fragmentation is the dominant failure mode and grouping is the direct fix.
  • On LongBench, shared prefix is everything. It adds 1.51 points on average, concentrated in the multi-hop datasets (2wikimqa +4.86, hotpotqa +1.35). Grouping alone adds 0.11.
  • Neither is redundant. The two components address the two failures in Figure 2: grouping repairs local continuity, shared prefix repairs global attention distribution. Different benchmarks stress different failures.
  • Watch out for a trap in the RULER table. Shared prefix without grouping is slightly worse than neither (38.66 vs 40.47). Fixing the sink distribution does not help if your recomputation is busy corrupting entities.

9. Where the 3.33× comes from

Single NVIDIA L20, 16K input tokens, Qwen2.5-14B primary, SmolLM2-135M auxiliary on an Intel Xeon 6554S, 20% recomputation.

Method Token selection Recomputation Other Total Speedup
Full attentionn/an/an/a5.641 s1.00×
CacheClip0.238 s1.332 s0.125 s1.695 s3.33×
  • Recomputation dominates at 78.6% of total latency. That is where the recomputation ratio knob acts, and it is close to linear in the ratio.
  • Token selection costs 0.238 s and most of it is free. It runs on CPU and overlaps with the GPU loading primary KV caches.
  • Recomputation needs a custom kernel. The paper uses a Triton[24] Flash Attention kernel that accepts sparse query indices. Stock implementations assume dense contiguous queries, so an off-the-shelf reimplementation will not hit these numbers.
  • Storage is the cost you do not see in this table. You now maintain two KV cache stores, primary and auxiliary, for the entire corpus. The auxiliary one is small at 135M parameters, but the primary store is not.

10. When not to use this

Being honest about where this does not apply, based on what the paper reports and what it does not.

  • Quality is still below full attention. 85.2% of full-attention performance on NIAH and 91.1% on LongBench at 20% recomputation. If your application cannot absorb that, this is not your technique.
  • You need a real chunk cache store. Precomputed KV caches for a whole corpus are large, and you need two of them. If your corpus turns over quickly, precomputation amortizes badly.
  • You need idle CPU on the head node. The whole no-extra-GPU-cost argument rests on that. In a CPU-saturated serving setup the auxiliary model has to go somewhere else, and the accounting changes.
  • A custom sparse-query attention kernel is required. This is real engineering, not a config change.
  • The evaluation is single-model. Everything end-to-end uses Qwen2.5-14B-Instruct as the primary model. The attention-alignment study covers LLaMA-3.1-8B and Ministral-8B, but the full pipeline results do not.
  • The hyperparameters are not swept. Window size w=8 and threshold τ=5 are given as defaults with no sensitivity analysis. Given how much of the RULER gain comes from grouping, that is the ablation I most want to see.
  • No throughput or concurrency numbers. Latency is measured on a single request on a single GPU. How this behaves under batching and contention is open.

11. Takeaways

  1. Independent chunk caches fail in two ways, and you have to fix both. Repeated sinks and missing cross-chunk attention have different causes and different cures. Prior work generally addresses one.
  2. A 135M model is a better attention oracle for a 14B model than the 14B model's own first layer. This holds across four model pairs, five sequence lengths and 200 samples each, including across architectures and tokenizers. It is the most transferable idea in the paper.
  3. Selection quality beats selection budget. CacheClip hits its ceiling around 10 to 20% recomputation. CacheBlend needs 70% or more to reach comparable quality on cross-chunk tasks.
  4. Contiguity is not a detail. Grouping is worth 40 points on RULER. Recomputing scattered tokens splits multi-token entities and corrupts the cache, which is why CacheBlend gets worse as its budget grows.
  5. Putting the selector on the CPU is what makes it deployable. The auxiliary model adds no GPU memory, no GPU compute, no multi-card imbalance, and its latency hides behind cache loading.
  6. 3.33× faster prefill at roughly 85 to 91% of full-attention quality, tunable at runtime. Whether that trade is good depends entirely on your application, and the ratio knob means you do not have to decide once for all requests.
Worth watching The paper's own suggested next step is finetuning the auxiliary model specifically to predict the primary model's attention. Since auxiliary models are under 1B parameters, that training run is cheap compared to finetuning the primary model, and it targets the exact bottleneck: selection accuracy. It also generalizes beyond RAG to any workload that injects large text blocks into context, including agentic tool output.

References

  1. Yang, Leng, Zeng, Wu. CacheClip: Accelerating RAG with Effective KV Cache Reuse. Intel Corporation, 2026. https://arxiv.org/abs/2510.10129
  2. Yao et al. CacheBlend: Fast Large Language Model Serving with Cached Knowledge Fusion, 2024. https://arxiv.org/abs/2405.16444
  3. Yang, Chen, Chen. APE: Faster and Longer Context-Augmented Generation via Adaptive Parallel Encoding. ICLR 2025.
  4. Xiao et al. Efficient Streaming Language Models with Attention Sinks, 2023. https://arxiv.org/abs/2309.17453
  5. Gim et al. Prompt Cache: Modular Attention Reuse for Low-Latency Inference. MLSys 6, 2024.
  6. Jin et al. RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation, 2024. https://arxiv.org/abs/2404.12457
  7. Kwon et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023.
  8. Zheng et al. Efficiently Programming Large Language Models using SGLang, 2023. https://arxiv.org/abs/2312.07104
  9. Sun, Wang, Tian. Block-Attention for Efficient RAG, 2024. https://arxiv.org/abs/2409.15355
  10. Lu et al. TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Text, 2024. https://arxiv.org/abs/2410.07590
  11. Yang et al. KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse, 2025. https://arxiv.org/abs/2502.16002
  12. Zhang et al. Attention Entropy is a Key Factor: An Analysis of Parallel Context Encoding, 2024. https://arxiv.org/abs/2412.16545
  13. Agarwal et al. Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation. PACMMOD 3(3), 2025.
  14. Zhang et al. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models. NeurIPS 36, 2023.
  15. Li et al. SnapKV: LLM Knows What You Are Looking For Before Generation. NeurIPS 37, 2024.
  16. Hsieh et al. RULER: What's the Real Context Size of Your Long-Context Language Models? 2024. https://arxiv.org/abs/2404.06654
  17. Bai et al. LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding, 2023. https://arxiv.org/abs/2308.14508
  18. Ben Allal et al. SmolLM2: When Smol Goes Big, 2025.
  19. AbouElhamayed et al. SpArAMX: Accelerating Compressed LLMs Token Generation on AMX-Powered CPUs, 2025. https://arxiv.org/abs/2502.12444
  20. Intel. Accelerate AI Workloads with Intel Advanced Matrix Extensions, 2022.
  21. Ho et al. Constructing a Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps. COLING 2020.
  22. Tang et al. Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference, 2024. https://arxiv.org/abs/2406.10774
  23. Cai et al. PyramidKV: Dynamic KV Cache Compression Based on Pyramidal Information Funneling, 2024. https://arxiv.org/abs/2406.02069
  24. Tillet, Kung, Cox. Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations. MAPL 2019.