Abstract
Reasoning models have demonstrated impressive performance in self-reflection and chain-of-thought reasoning. However, they often produce excessively long outputs, leading to prohibitively large key-value (KV) caches during inference. While chain-of-thought inference significantly improves performance on complex reasoning tasks, it can also lead to reasoning failures when deployed with existing KV cache compression approaches. To address this, we propose Redundancy-aware KV Cache Compression for Reasoning models (R-KV), a novel method specifically targeting redundant tokens in reasoning models. Our method preserves nearly 100% of the full KV cache performance using only 10% of the KV cache, substantially outperforming existing KV cache baselines, which reach only 60% of the performance. Remarkably, R-KV even achieves 105% of full KV cache performance with 16% of the KV cache. This KV-cache reduction also leads to a 90% memory saving and a 6.6X throughput over standard chain-of-thought reasoning inference. Experimental results show that R-KV consistently outperforms existing KV cache compression baselines across two mathematical reasoning datasets.
Beyond the research prototype, R-KV has been deeply integrated and performance-optimized for production-grade LLM serving stacks. The open-source implementation includes optimized ports for vLLM and SGLang, together with an LMCache-compatible integration example. These implementations translate the algorithm into practical serving-system components spanning GPU kernels, KV-memory management, request scheduling, and distributed inference.
Open-source Impact
Optimized implementations for vLLM and SGLang, plus an LMCache-compatible example for integrating R-KV with external KV-cache management workflows.
True physical KV eviction, CUDA-graph decoding, fused and batched redundancy scoring, asynchronous scheduling, and compression-aware memory admission.
Tensor and data parallel execution validated on 8×H100 systems, including consistent cross-rank eviction decisions and near-linear DP scaling.
Public benchmark results and implementation notes cover accuracy, throughput, memory pressure, concurrency, and multi-GPU scaling under realistic serving workloads.