We are excited to share ExANS: Exponent Aligned Asymmetric Numeral Systems, a lossless GPU codec for structured BF16 tensors. ExANS exploits local duplication in the exponent field and preserves every input bit. It is built for fast decompression and high compressibility in constrained network environments.

OnGPU compression retrieves data at effective throughput beyond the physical wire rate. PCIe and network bandwidth are often the binding constraint in high performance systems: a codec that decodes faster than the link delivers moves that bound. For inference engines this means cached KV blocks arrive sooner: TTFT drops.

BF16 is notoriously hard to compress: duehigh entropy, no repeating bytes. Heavier codecs buy ratio at the cost of encode/decode latency. Within the KV cache, however, the 8 bit exponent repeats frequently. ExANS isolates and aligns this field, then applies a modified ANS (Asymmetric Numeral System) encoding: lossless compression at GPU speed.

Under the hood: KV BF16 exposes a useful field boundary

A BF16 value has 16 bits:

Sign        Exponent        Fraction
1 bit       8 bits          7 bits
s           EEEEEEEE        FFFFFFF

For normal values,

Value can be derived with = (−1)^sign × 2^(exponent − 127) × (1 + fraction / 128)

To exploit the low entropy of the exponent, we split each Bf16 world into two byte streams: the exponent and the concatenated sign + mantissa field. This is because the entropies differ significantly with the exponent being highly compressible.

Although BF16 retains the eight bit exponent range of FP32, KV cache is indexed by layer, key/value tensor, KV head. Exponents within the same KV block demonstrate a narrower range with high duplication based on our benchmarks.

Block Local Alignment

A physical KV block contains many BF16 values organized as 2L groups: one K group and one V group for each layer, where L is the layer. Every group contains d (hidden dimensions) spanning the block’s positions and KV channels.

Exponent distributions differ between these groups even inside the same physical block. Encoding them without alignment spreads probability mass across multiple exponent symbols. ExANS therefore processes every physical block independently and computes one exponent center c_g for each of its 2L groups.

The encoder builds a 256 entry exponent histogram over the d values in group g, selects its dominant exponent, and stores the resulting 2L centers in a 128 byte block local center table. Each exponent is translated using:

C = (E + 127 − c_g) mod 256

This maps the dominant exponent of every group to symbol 127. The transformation is exactly reversible and performs no rounding or quantization. Because the center table is stored inside the physical block’s record, the block can be decoded without calibration data or another KV block.

Encoding Pipeline

For each BF16 word (w), the GPU first separates the 8 bit exponent from the combined sign and fraction byte:

E  = (w >> 7) & 0xff
SF = ((w >> 8) & 0x80) | (w & 0x7f)

This produces two streams with different statistical properties:

• Stream 1: (E): the exponent stream, which has low entropy within a KV block.

• Stream 2: (SF): the sign and fraction stream, whose measured entropy is close to the full eight bits per value.

Table 1: Measured zero order entropy of the two BF16 byte streams.

For a byte-valued random variable X, Shannon entropy is:

H(X) = −Σₓ p(x) log₂ p(x)

A uniformly random byte has H(X) = 8 bits because all 256 symbols are equally likely. An intuitive interpretation is the effective alphabet size:

N_eff = 2^H(X)

The sign/mantissa field has H = 7.9718, corresponding to 251.05 effective symbols: almost indistinguishable from a uniformly random byte. Its eight most frequent symbols account for only 4.43% of observations.

The block-aligned exponent has H = 2.6500, corresponding to only 6.28 effective symbols. Although 42 exponent symbols occur at least once, 98.12% of all observations belong to the eight most frequent symbols. This concentration is what allows ANS to represent the exponent stream using approximately 2 to 3 bits per value instead of eight.

Entropy and effective alphabet size

The key distinction is:

• Eight-bit randomness: 256 approximately equally probable choices.

• Entropy of 2.65 bits: statistically equivalent to only 2^2.65 = 6.28 equally probable choices.

This does not mean each exponent is individually stored in 2.65 bits. ANS approaches that average over a sequence of symbols.

For example, within one group with center c_g = 130:

C = (E + 127 − c_g) mod 256

Suppose a group contains:

Table 2: Exponent alignment for one example group with center 130.

The significant exponent now maps to symbol 127. Applying the same transformation to other groups maps their respective exponents to that same symbol, leading to a more concentrated distribution for the entropy coder.

The new exponent stream is now compressed using a modified ANS, however sign and fraction stream (S2) is stored uncompressed because its entropy is already close to 8 bits per value and does not provide benefit.

We replace the ANS generic partitions with eight partitions aligned to the physical KV block. This reduces repeated state and descriptor overhead, lowering the compressed payload.

Finally we end up with:

ExANS physical KV block
├── Header
├── Block-local centers [2L]
├── Sparse rANS model
├── 8 independent exponent partitions
└── Raw sign + mantissa bytes

Decoding Pipeline

Decoding follows a reverse process. The block record carries the center table directly as metadata; the sparse probability table reconstructs the block’s ANS model, and 8 ANS partitions are decoded in parallel to recover the exponent symbols (C). The original exponent is recovered using:

E = (C + c_g − 127) mod 256

The 2 streams (S1 and S2) are then recovered via the following:

word = ((sign_fraction & 0x80) << 8)
     | (exponent << 7)
     | (sign_fraction & 0x7f)

ExANS then reconstructs 8 BF16 values per cuda thread directly on the destination GPU buffer. blocks are parallelized across cuda streams, with overlapping the 3 decode phases: data transfer, ANS decode, and reconstruction. This ensures the GPU is well saturated and keeps decompression throughput high.

ExANS builds on exponent based compression (DietGPU, ZipNN) and extends it to paged KV caches: per group exponent centering under a per block entropy model, with the added benefit of independently decodable blocks.

Performance

We evaluated ExANS using NVIDIA H100 via a real KV snapshot on real world datasets. This dataset was cached externally and retrieved with ExANS enabled and supplied as additional context to the inference engine. The context window was 256K.

Table 3: ExANS compression ratio and throughput on the H100 KV snapshot.

In both modes, decoding substantially exceeded the NIC bandwidth on Infiniband 400 Gb/s (~50 GB/s). ExANS reduced KV transfer volume at a decompression rate 9.6x faster than the NIC. For the TTFT this means the user sees a reduced delay when waiting for the first token.

Decode throughput

Table 4: Median decode throughput and observed range per codec.

SplitZip and Bitcomp attain higher decode throughput, but SplitZip reaches only 1.284886× compression and Bitcomp slightly expands this BF16 workload. ExANS targets a different operating point: 1.512208× lossless compression while retaining a median decode rate of 637.971 GB/s: 12.76 times the theoretical 50 GB/s payload rate of a 400 Gb/s network link.

Conclusion

ExANS turns the structure of the KV cache into compression: it aligns the low entropy exponent field block by block, entropy codes only where the redundancy lives, and leaves every other bit untouched. The result is 1.51x lossless compression with decode throughput above the NIC line rate, so retrieval stays network bound, not codec bound. Most importantly, inference output quality does not degrade: KV blocks are retrieved losslessly, bit for bit.

This work is available starting the OpenLake v0.8 release and can be enabled on vLLM with the following OpenLake connector config:

{
  "kv_connector": "OpenLakeConnector",
  "kv_connector_module_path": "openlake_client.openlake_connector",
  "kv_role": "kv_both",
  "kv_connector_extra_config": {
    "openlake_nodes": ["127.0.0.1:9400"],
    "openlake_device": "local",
    "openlake_compression": {
      "enabled": true,
      "codec": "expans",
      "mode": "lossless"
    }
  }
}

For any additional support, feel free to open an issue on the OpenLake github or send us a message on discord.

GitHub: github.com/openlake-project/openlake
Discord: Invite
Get Involved: See more
Contact Us: Drop a message