Engram
Conditional Memory via Scalable Lookup: a new sparsity axis for LLMs. Engram modernizes n-gram embeddings for O(1) retrieval, offloading static knowledge to memory while preserving compute for reasoning. A 27B model with Engram outperforms pure MoE baselines across all tasks.
Modern LLMs are remarkably good at reasoning but surprisingly inefficient at remembering. Consider asking “What is the capital of France?”—the model doesn't retrieve this fact from memory. Instead, it reconstructs it through billions of matrix multiplications. This is like solving a math problem every time you want to recall your phone number.
Current LLM Approach
- ✗All knowledge stored in neural weights
- ✗Every fact requires full forward pass to recall
- ✗Static patterns waste compute on reconstruction
- ✗No distinction between “lookup” and “compute”
Engram Approach
- ✓Static patterns stored in embedding tables
- ✓O(1) retrieval via direct table lookup
- ✓Free up compute for actual reasoning
- ✓Explicit separation of memory and compute
The Core Insight
MoE (Mixture of Experts) introduced conditional computation—only activate relevant experts. Engram introduces conditional memory—only retrieve relevant patterns. Together, they form two orthogonal axes of sparsity that complement each other.
What Gets Stored in Engram?
“capital of France” → Paris associations
Common phrases, collocations, idioms
“def __init__”, “import numpy”
Engram brings back a classic NLP technique: n-gram embeddings. Instead of storing individual token embeddings, we store embeddings for sequences of tokens. “the capital” gets its own embedding, distinct from “the” + “capital” individually.
Traditional Token Embeddings
Engram N-gram Embeddings
Tokenizer Compression
Before hashing, tokens undergo normalization to reduce vocabulary size:
“ß” → “ss”, “Ⅰ” → “I”
“The” ≡ “the” ≡ “THE”
128K vocab → ~99K canonical IDs
N-gram Orders
Engram uses bigrams (n=2) and trigrams (n=3):
“the capital”, “capital of”, “of France”
“the capital of”, “capital of France”
4-grams were tested but suboptimal under fixed parameter budget.
The key innovation is O(1) constant-time retrieval. Unlike attention which requires O(n) operations to scan the context, Engram uses direct hash-based table lookup. The same n-gram always maps to the same embedding—no learned routing, no computation.
Raw token sequence from model
["the", "capital"]NFKC normalization + lowercase
["the", "capital"]Get 2-grams and 3-grams from context
2-gram: "the capital"Hash each n-gram with K=8 different hash functions
h₁(the capital) = 42891, h₂(...) = 15723, ... h₈(...) = 98412Direct index into embedding tables (prime-sized)
E[42891], E[15723], ..., E[98412] → 8 × 160d vectorsMerge all retrieved embeddings
concat([e₁, e₂, ..., e₈]) → 1280d embeddingContext-aware gating controls injection strength
α = σ(q·k/√d) ≈ 0.73 → Y = α · V + conv(V)O(1) Lookup
Direct table indexing. No attention computation, no iteration over sequence.
Deterministic
Same n-gram always retrieves same embedding. Enables prefetching and caching.
Offloadable
Tables can live in host memory. Only 2.8% overhead for 100B params!
Multi-Head Hashing
For each n-gram, K=8 different hash functions map to K different embedding tables:
Different hash functions reduce collision risk.
Prime-Sized Tables
Each embedding table has a prime number of slots M_{n,k} for better hash distribution:
Primes minimize clustering from hash collisions.
Why O(1) Matters
Hash computed from tokens alone—start memory fetch early
Each n-gram lookup is independent—full batch parallelism
Deterministic access patterns enable host memory storage
Raw n-gram embeddings are static—they don't know if they're relevant in context.Context-aware gating solves this by dynamically modulating the injection strength based on the current hidden state.
Gating Mechanism
Non-linearity Enhancement
After gating, a depthwise causal convolution adds local context:
Kernel size 4, dilation = max n-gram order (3).
Multi-Branch Fusion
For multi-branch architectures (attention + MoE), Engram shares values but has branch-specific gates:
When Does Gating Activate?
Context matches n-gram pattern—inject memory
Context differs from pattern—suppress memory
A key contribution is the discovery of a U-shaped scaling law. Given a fixed parameter budget, how should you split between MoE experts (compute) and Engram tables (memory)? Pure MoE (ρ=100%) is not optimal—there's a sweet spot around ρ=75-80%.
ρ = 0%
Pure Engram. Too much static memory, not enough compute for reasoning.
ρ = 75-80%
Optimal allocation. Balanced compute and memory for best performance.
ρ = 100%
Pure MoE. Wastes compute reconstructing static patterns from weights.
The U-Shape Insight
Pure MoE (ρ=100%) is not optimal. At 6×10²&sup0; FLOPs, allocating 25% of inactive parameters to Engram memory improves validation loss by 0.028—equivalent to significant additional pretraining. The U-shape persists across compute budgets, suggesting a fundamental property of the compute-memory tradeoff.
Too Much Compute (ρ=100%)
Pure MoE wastes compute reconstructing static patterns. The network “memorizes” facts in weights instead of looking them up. Less capacity for actual reasoning.
Too Much Memory (ρ=0%)
Pure Engram can only retrieve patterns, not reason about them. Complex tasks requiring multi-step inference fail. Memory without compute is just a database.
Optimal Allocation: ρ ≈ 75-80%
At 6×10²&sup0; FLOPs, ρ=75% reduces validation loss by 0.028 compared to pure MoE. This U-shape persists across compute budgets, suggesting a fundamental property of the compute-memory tradeoff in language modeling.
Infinite Memory Regime
When scaling Engram slots from 2.58×10&sup5; to 1.0×10&sup7;:
This means more memory always helps—no diminishing returns observed yet.
Engram-27B is the flagship model demonstrating the approach. It reallocates 5.7B parameters from MoE experts to Engram memory, keeping total parameters at 26.7B (iso-parameter with baseline).
Layer 2: Engram-Enhanced
Engram retrieves static n-gram patterns via O(1) lookup and adds them to the hidden state via residual connection: Hℓ ← Hℓ + Y
MoE Experts
Reduced from 72 to reallocate 5.7B params to Engram
Engram Memory
Parameters in n-gram embedding tables
Total Params
Iso-parameter with MoE-27B baseline
Model Configuration
Engram modules are trained end-to-end with the backbone using different optimization strategies. The embedding tables require special handling due to their sparse access patterns.
Training Configuration
Distributed Training
Embedding tables are sharded across GPUs:
- •Forward: All-to-All to gather active rows
- •Backward: All-to-All to dispatch gradients
- •Sparse updates only for accessed rows
Why Different Optimizers?
Muon uses Newton-Schulz orthogonalization which assumes dense gradients. Engram tables have extremely sparse gradients (only accessed n-grams update). Adam handles sparse gradients naturally through per-parameter momentum.
Identity Initialization
Convolution parameters are initialized to zero, making the conv layer an identity at the start. This ensures Engram doesn't disrupt the pretrained backbone early in training.
Engram-27B was evaluated against a strictly iso-parameter, iso-FLOPs MoE-27B baseline. The improvements are consistent across all domains—knowledge, reasoning, code, and math.
Knowledge & Reasoning
Code & Math
Long Context (RULER)
Under iso-loss comparison, Engram dramatically improves long-context retrieval:
A practical benefit: Engram tables can be offloaded to host memory. Unlike neural network weights that need GPU compute, embedding tables just need random access. This enables massive memory capacity without consuming precious HBM.
Why Offloading Works
- ✓Deterministic access: Hash computed from tokens alone
- ✓Prefetchable: Start memory load during prior layer
- ✓No GPU compute: Just memory read + simple addition
Caching Strategy
N-gram access follows Zipfian distribution—few patterns are very common:
- •L1: GPU registers for active batch
- •L2: GPU memory for hot n-grams
- •L3: Host DRAM for full table
Offloading Performance
Implications for Scaling
This changes how we think about model capacity. With offloading, a server with 8×H100 (640GB HBM) can effectively have terabytes of model capacity by using host RAM for Engram tables. Memory becomes cheap again.
Why does Engram help reasoning tasks even though it's designed for memory? Mechanistic analysis reveals that Engram effectively deepens the network by offloading pattern matching from early layers.
LogitLens Analysis
Tracking when hidden states converge to final predictions:
Earlier convergence = more layers free for complex reasoning.
CKA Similarity Analysis
Comparing representation similarity across layers:
Engram Layer 5 ≈ MoE Layer 12 representations
Engram compresses 7 layers of pattern matching into memory lookup.
The Depth Hypothesis
Transformers use early layers for “shallow” tasks like pattern matching and later layers for “deep” tasks like reasoning. By handling patterns via lookup, Engram frees early layers to start reasoning sooner. The network becomes effectively deeper for complex tasks without adding parameters.
Attention Capacity
By delegating local dependencies (n-grams) to memory lookup, attention heads can focus on global context:
Attention wastes heads on “the capital of” patterns
All heads available for long-range dependencies
Extensive ablations reveal which components matter most and how to optimally configure Engram.
Component Importance
Validation loss regression from reference (1.768):
Layer Placement
Early layers benefit most; splitting memory helps.
Sensitivity Analysis
Performance retention without Engram:
Confirms Engram stores facts, not reasoning.
Why Not 4-grams?
Under fixed parameter budget, 4-grams don't help. They capture rarer patterns but require more table slots, reducing capacity for common patterns. The 2+3 gram combination is the sweet spot for efficiency.
Engram opens up a new research direction: explicit memory in transformers. Several exciting extensions are possible.
Near-Term Extensions
- →Larger tables: 100B+ parameter memory
- →Dynamic insertion: Learn when to use Engram
- →Cross-lingual: Shared memory across languages
- →Domain adaptation: Swap memory for different tasks
Research Questions
- ?Can Engram store semantic patterns, not just n-grams?
- ?How does the U-shape change at trillion-parameter scale?
- ?Can we update memory at inference time (RAG-style)?
- ?Does Engram help with multimodal models?
The Bigger Picture
Engram challenges the assumption that “all knowledge must be in neural weights.” By separating static patterns (memory) from dynamic computation (reasoning), we may be able to build models that are both more capable and more efficient. The brain doesn't reconstruct every memory from scratch—neither should LLMs.
Industry Implications
Less GPU compute for factual recall = cheaper inference
Memory bandwidth matters more; HBM less critical
Swap memory tables for domain-specific models
Engram: The Complete Picture
Engram introduces conditional memory as a new sparsity axis for LLMs. By modernizing n-gram embeddings for O(1) lookup, it separates static pattern storage from dynamic computation. The U-shaped scaling law guides optimal allocation, and memory offloading enables massive capacity at minimal inference cost.