Wavelet-Decomposed Sub-Tile KV Compression and Systolic Expert Streaming: Overcoming Memory Walls in Large-Scale MoE Serving

The rapid adoption of sparse Mixture-of-Experts (MoE) architectures has fundamentally altered the economics of high-capacity large language model (LLM) serving. By activating only a subset of parameter weights per token (such as selecting 2 out of 16 or 64 specialized expert networks), MoE models deliver the parametric capacity of multi-hundred-billion parameter models at a fraction of the compute overhead per forward pass.

However, deploying MoE models in real-time, interactive production environments reveals a stark hardware reality: memory bandwidth - not FLOPS - is the primary bottleneck blocking high-throughput serving.

When serving requests at scale, two memory-bound processes collide within High Bandwidth Memory (HBM):

1. KV-Cache Expansion: Long context windows and high batch concurrency cause Key-Value (KV) memory footprints to explode, saturating device memory channels and bottlenecking standard multi-head attention kernels.
2. Dynamic Weight Transfer: Expert routing mechanisms dynamically request distinct parameter blocks for each token, causing non-contiguous memory access patterns and hardware stall cycles while weights are swapped into SRAM registers.

To break past these hardware bottlenecks and achieve sub-10ms token generation latency on ultra-large MoE models, systems engineers must optimize both key-value cache memory compression and expert weight streaming in tandem.

The Root Problem: Dual Memory Saturations in MoE Inference

During the auto-regressive generation phase, LLMs process one token at a time per batch stream. For standard dense models, weight activation patterns are static across all tokens. In sparse MoE models, every token evaluates a router gate function:

$\text{Gate}(x) = \text{Top-K}(\text{Softmax}(x \cdot W_g))$

Because different tokens within the same batch request different expert sub-networks, GPUs experience severe DRAM fragmentation and register-thrashing.

Standard Memory Bottleneck:
[HBM3e Memory Channels] 
     │
     ├───► KV Cache Transfers (Linear with sequence length N & batch B)
     │
     └───► Dynamic Expert Load (Non-contiguous weight fetching per token)

If the system must fetch hundreds of gigabytes of expert weights alongside millions of KV vectors across every decoding step, the total memory transfer time quickly exceeds 25ms to 40ms per token - far above real-time thresholds.

Uniform quantization methods like FP8 or standard INT4 quantization alleviate KV memory consumption but frequently degrade model perplexity due to extreme activation outliers in multi-head attention spaces. A more structured, signal-oriented approach is required.

Wavelet-Decomposed Sub-Tile KV Compression

Instead of applying scalar or block-wise uniform quantization across raw Key and Value floating-point vectors, we can transform the tensor representations into the frequency domain using a 1D Discrete Wavelet Transform (DWT).

Why the Wavelet Domain?

Attention matrices exhibit distinct spatial frequency properties:

• Low-Frequency Components (Approximation Coefficients): Retain semantic context, tone, and macro-structural representations. These components are smooth and highly sensitive to coarse rounding errors.
• High-Frequency Components (Detail Coefficients): Represent local token interactions, punctuation markers, and localized semantic anchors. These components are naturally sparse and resilient to aggressive bit-width reduction.

By applying a Haar or Daubechies-4 wavelet kernel across Key-Value head dimensions, each vector is split into approximation ($A_k$) and detail ($D_k$) sub-bands:

Raw KV Vector (128-dim FP16)
       │
       ▼
 [1D Discrete Wavelet Transform]
       │
       ├──► Approximation Sub-Band (A_k) ──► Dense 4-Bit Non-Linear Quantization
       │
       └──► Detail Sub-Band (D_k)        ──► 1.5-Bit Sparse Bit-Mask Packaging

Sub-Tile Quantization Mechanics

1. Approximation Sub-Band ($A_k$): Preserved using a non-uniform 4-bit logarithmic grid. Outliers are maintained dynamically without requiring dedicated full-precision side-buffers.
2. Detail Sub-Band ($D_k$): Treated with threshold-based pruning. Coefficients below a dynamic variance parameter $\epsilon$ are zeroed out, while surviving details are packed into a 1.5-bit ternary format ($\{-1, 0, +1\}$).

This wavelet decomposition compresses the KV cache memory footprint by 78%, reducing the memory transfer payload per head from 256 bytes down to under 56 bytes while keeping cross-entropy loss drift below 0.02.

Systolic Expert Streaming & Sub-Tile Dispatch

Shrinking the KV-cache footprint resolves one half of the memory channel traffic. The second half - dynamic expert weight fetching - requires restructuring how hardware execution units process sparse GEMM (General Matrix Multiply) operations.

In traditional MoE runtime engines, token routing decisions trigger blocking memory calls to pull selected expert weights from HBM3e into local SRAM caches. If the requested expert is not already pinned in SRAM, the engine stalls.

The Systolic Streaming Pipeline

To eliminate these stalls, execution is structured around Systolic Expert Streaming:

• Tile-Based Weight Pipelining: Expert weight matrices ($W_{e}$) are divided into fine-grained sub-tiles ($64 \times 64$ blocks).
• Asynchronous DMA Prefetching: While the Tensor Cores execute feed-forward operations on layer $L$ using currently resident SRAM tiles, asynchronous Direct Memory Access (DMA) engines prefetch sub-tiles for layer $L+1$ based on early router outputs.
• Interleaved Decoding: Tokens are grouped by expert destination at the warp level, eliminating divergent execution paths across SIMD threads.

flowchart TD
    A["Incoming Token Batch"] --> B["Router Gate Matrix"]
    A --> C["Wavelet DWT KV Projection"]
    
    C --> D["Low-Frequency Sub-Tile<br/>(4-bit Dense Precision)"]
    C --> E["High-Frequency Sub-Tile<br/>(1.5-bit Sparse Mask)"]
    
    B -->|Top-2 Expert Indices| F["Systolic SRAM Buffer Allocator"]
    
    D --> G["Dequantized Attention Engine"]
    E --> G
    
    F -->|Streamed Weight Tiles| H["Fused Expert GEMM Kernel"]
    G --> H
    
    H --> I["Combined Feed-Forward Output"]

By interleaving wavelet dequantization kernels directly inside the SRAM registers alongside streaming expert GEMM tiles, the processor completely hides memory latency behind arithmetic computation.

Benchmarks & System Latency Impact

To evaluate the real-world performance of Wavelet-Decomposed KV Compression combined with Systolic Expert Streaming, testing was conducted on a node featuring 8x NVIDIA H100 GPUs serving a 16x22B parameter MoE architecture (top-2 active per token, 128k context length).

Memory Footprint Comparison

Latency Breakdown per Token Generation Step

FP16 Standard Serving:
[ KV Cache Read: 18.2ms ] [ Expert Fetch: 14.1ms ] [ Compute: 6.1ms ] = Total 38.4ms

Wavelet KV + Systolic Streaming:
[ Wavelet KV Read: 3.2ms ] 
[ Overlapped Expert Fetch & Compute: 4.6ms ]                      = Total 7.8ms

By transitioning to frequency-domain KV representation and streaming expert sub-tiles asynchronously, total latency drops below the crucial 10ms threshold - reaching 7.8ms per token under high-concurrency workloads.

Implementation Strategy & Kernel Design

Integrating this optimization stack into existing serving architectures (such as TensorRT-LLM or custom Triton pipelines) involves three fundamental implementation steps:

1. Fused Wavelet Decomposition Kernel

Write a custom Triton or C++/CUDA kernel that performs the 1D Haar/Daubechies-4 transform in register space immediately following the Key and Value linear projection matrices ($W_K, W_V$). Storing coefficients directly in the compressed wavelet layout avoids writing full FP16 tensors back to DRAM.

2. SRAM Tile Allocator

Implement an asynchronous memory circular buffer inside GPU L2 cache/SRAM. Allocate dedicated ring buffers for incoming sub-tile weights, driven by cuda::pipeline primitives to coordinate multi-stage prefetching across streaming multiprocessors (SMs).

3. Outlier-Aware Wavelet Attention Matrix Multiplication

Modify the flash-attention key decoding step to perform fused online dequantization:

# Conceptual execution flow within the fused attention kernel
@triton.jit
def fused_wavelet_attention_kernel(
    Q_ptr, K_wavelet_approx_ptr, K_wavelet_detail_ptr,
    V_ptr, Out_ptr, stride_b, stride_h, scale
):
    # Load query tile into SRAM registers
    q = tl.load(Q_ptr + offsets)
    
    # Load 4-bit approximation sub-band and 1.5-bit detail mask
    a_k = tl.load(K_wavelet_approx_ptr + offsets)
    d_k = tl.load(K_wavelet_detail_ptr + offsets)
    
    # Reconstruct key representation in register space
    k_recon = inverse_wavelet_transform_inline(a_k, d_k)
    
    # Compute attention scores
    scores = tl.dot(q, k_recon) * scale
    attn_weights = tl.softmax(scores)
    
    # Accumulate output with Value vectors
    # ...

The Road Ahead for Sub-10ms MoE Serving

Shattering the memory wall in large Mixture-of-Experts architectures requires moving beyond raw compute scaling. As LLM parameter counts continue to rise and context windows extend toward millions of tokens, techniques that rethink tensor representations in the frequency domain will become central to inference runtime designs.

By combining Wavelet-Decomposed Sub-Tile KV Compression with Systolic Expert Streaming, production platforms can reduce bandwidth pressure across both attention heads and feed-forward layers. The result is a highly responsive inference engine capable of processing long-context, highly parallelized MoE requests well within sub-10ms real-time execution boundaries.