DeepSeek-OCR 2
Visual Causal Flow: Teaching AI to read images the way humans actually do—not as a rigid grid of pixels scanned like a typewriter, but as a meaningful flow of visual attention following the natural structure of what you're looking at.
This paper introduces a paradigm shift in how vision-language models process images. Instead of forcing visual information into a fixed raster-scan order, DeepSeek-OCR 2 learns to dynamically reorder visual tokens based on semantic importance—achieving state-of-the-art performance with dramatically fewer tokens.
Your Deep Dive Into This Paper
Imagine you're reading a newspaper. Do you start at the top-left pixel and scan right, then drop down, then scan right again, like a 1950s typewriter? Of course not.
Your eyes jump to the headline. Then to the photo. Then to the caption. Then maybe to a pull quote. You follow meaning, not geometry. This is called semantic reading order—and it's fundamentally different from how computers have been processing images.
How Computers Read (Before)
- • Start at pixel (0,0)
- • Move right pixel by pixel
- • At edge, drop down one row
- • Repeat until bottom-right
- • No understanding of content
How Humans Read
- • Identify salient regions first
- • Jump to most important element
- • Follow logical reading order
- • Group related information
- • Meaning drives attention
The Fundamental Mismatch
Every Vision-Language Model before DeepSeek-OCR 2 processed images in raster scan order—top to bottom, left to right. This creates a catastrophic mismatch:
When the headline is at token position 15 but references information at token position 847, the language model must bridge an enormous gap. The attention mechanism can technically do this, but it's inefficient and error-prone.
Raster scan processes patches left-to-right, top-to-bottom. Notice how it processes the sidebar's top row before the main content's second row—ignoring semantic relationships.
The Cost of Misalignment
This spatial-to-causal mismatch has real consequences:
- • More tokens needed to capture relationships
- • Longer attention spans increase computational cost quadratically
- • Harder to learn because related info is scattered
- • More prone to hallucination when relationships span many tokens
To understand why Visual Causal Flow is revolutionary, we need to understand what came before and why those approaches hit fundamental limitations.
Timeline of Vision-Language Model Evolution
Contrastive learning between images and text. Global image embeddings—no spatial structure. Great for retrieval, poor for fine-grained understanding.
Interleaved image-text with perceiver resampler. Fixed number of visual tokens per image. Showed few-shot learning but couldn't handle dense OCR tasks.
Direct projection of ViT features to LLM. Raster-scan ordering. 256-576 tokens typical. Better at reasoning but still fundamentally spatial.
Higher resolution, more tokens (2000+). Dynamic resolution. Performance improved but token count exploded.
Visual Causal Flow—dynamic semantic reordering. 91% accuracy with only 1,120 tokens. The paradigm shift.
The Token Scaling Problem
Before Visual Causal Flow, the only way to improve document understanding was to add more tokens:
| Model | Tokens | Score |
|---|---|---|
| LLaVA | 576 | 72% |
| Qwen-VL | 2,048 | 83% |
| InternVL | 2,304 | 86% |
| MinerU 2.0 | 6,790 | 87% |
Attention cost scales O(n²)—doubling tokens = 4× compute.
The DeepSeek-OCR 2 Solution
Instead of more tokens, use smarter ordering:
| Metric | Before | After |
|---|---|---|
| Tokens | 2,880 | 1,120 |
| Score | 86% | 91% |
| Compute | 1.0× | 0.15× |
Better performance with 85% less attention compute.
Why Nobody Did This Before
The idea of reordering tokens seems obvious in hindsight. So why didn't earlier researchers try it?
Position encoding lock-in: ViT and CLIP encode absolute positions. Reordering would break the pre-trained position representations.
No obvious learning signal: What's the "right" order? There's no ground truth for semantic reading order.
Discrete reordering is non-differentiable: You can't backprop through a sorting operation directly.
DeepSeek's insight: Use learnable queries with causal attention to softly reorder tokens. The order emerges from end-to-end training.
Here's the breakthrough idea: instead of fixing the order of visual tokens, let the model learn the best order for each image. This is Visual Causal Flow.
The Three-Step Process
Image passes through vision encoder, producing N visual tokens in spatial order. This is the same as traditional VLMs.
K learnable "query" tokens attend to all visual tokens with causal self-attention. Query 1 picks the most important region. Query 2 sees Query 1's choice and picks the next. This cascade creates a learned semantic order.
The output of the K queries becomes the input to the language decoder. Now tokens are in semantic order, matching how the autoregressive LLM naturally processes sequences.
Old Way: Raster Scan
Related content scattered across hundreds of tokens.
New Way: Visual Causal Flow
Related content adjacent. Semantic structure preserved.
① Extract Features
Standard ViT: image → 16 patches → 16 tokens in spatial order (0,1,2,3...). Token 4 (headline) comes after token 3 (search bar) just because of position. Content doesn't matter.
② Causal Queries
K learnable queries attend to all tokens. The key: Q2 sees what Q1 picked before choosing (causal mask). Q1 grabs the headline. Q2 knows this, grabs the lede. Sequential decisions → semantic order.
③ Semantic Sequence
Query outputs become LLM input: headline → lede → body → quote. The autoregressive decoder now sees important context first, matching how it naturally processes text.
🔑 Why the Causal Mask is Essential
Without it, all queries would independently pick the most important region—you'd get 6 copies of the headline! The causal constraint forces diversity through sequential decision-making. Q2 must pick something different because it knows Q1 already claimed the headline. This is how the model learns document structure: not from labels, but from the sequential selection game.
Why "Causal"?
The word "causal" here refers to the attention pattern, not cause-and-effect reasoning:
By using causal attention for the queries, each query's selection is conditioned on all previous queries. This creates an ordered dependency chain—exactly what the language decoder expects.
The Elegant Insight
Traditional VLMs treat visual tokens and text tokens differently:
- • Visual: bidirectional (everything sees everything)
- • Text: causal (left-to-right generation)
Visual Causal Flow bridges this gap by making visual tokens also causal. When token 5 is generated, it knows tokens 1-4 came first. This matches the LLM's inductive bias and enables dramatically better information flow.
DeepSeek doesn't just add a reordering step—they redesign the entire visual encoder. DeepEncoder V2 is built from the ground up to support Visual Causal Flow.
Raw image at variable resolution (e.g., 1024×1024)
Bidirectional Stage
Like standard ViT: every token sees every other token. This builds rich representations where each token knows about the entire image context. Output: N spatially-ordered tokens.
Causal Flow Stage
The key innovation: K learnable queries with causal self-attention. Q2 sees Q1's selection before choosing. This sequential game produces K tokens in semantic order.
Architecture Overview
Image → Patches → Linear projection → Position embeddings.
Output: N visual tokens (e.g., 256 for 1024×1024 with 64×64 patches)
Visual tokens attend to each other bidirectionally.
Each token aggregates global context about the entire image.
Output: N context-enriched visual tokens
K learnable queries (K=64 by default) attend to visual tokens.
Queries use causal self-attention: Query i sees Queries 1...i-1.
Output: K reordered visual tokens in semantic order
K visual tokens + text prompt → autoregressive generation.
Visual tokens now match the causal structure the LLM expects!
Dynamic Resolution Support
DeepEncoder V2 supports variable resolution through a clever tiling strategy:
The model automatically selects tile count based on image aspect ratio and content density.
Why Not Just Use CLIP?
CLIP is the standard visual encoder for VLMs. Why build DeepEncoder V2 from scratch?
- • Fixed position embeddings (no reordering)
- • Optimized for global similarity, not local detail
- • No causal attention variant
- • Pre-trained on natural images, not documents
- • Learnable queries enable reordering
- • Hybrid attention (bidirectional + causal)
- • Document-first training objective
- • End-to-end optimization with LLM
The magic of Visual Causal Flow happens in the attention layers. Let's understand exactly how queries learn to reorder tokens.
The Standard Attention Formula
In Visual Causal Flow
The attention has two stages with different participants:
K = visual_tokens (N×d) # N visual features
V = visual_tokens (N×d) # Same as keys
Each query attends to all visual tokens, selecting which regions to aggregate.
K = queries_1_to_i-1 # Previous queries only!
V = queries_1_to_i-1 # Causal mask applied
Query i can see what queries 1...i-1 selected, creating ordered dependency.
Why Causal Masks Create Order
In the causal mask, Query 3 can see what Queries 1 and 2 selected, but not what Queries 4, 5, 6 will select. This forces each query to make a sequential decision—creating an ordered chain of selections that becomes the semantic reading order.
The Causal Mask Visualized
The causal mask is a triangular matrix that prevents queries from "seeing the future":
✓ = Can attend, ✗ = Masked out. Query 3 can see Q1 and Q2's outputs, but not Q4 or Q5.
Why This Creates Semantic Ordering
The training objective (next token prediction) teaches queries to select tokens that make the decoder's job easier:
- • Query 1 learns to select the most globally salient region (often the title)
- • Query 2 sees Q1 selected the title, learns to select logically-next content
- • Query K has full context, handles remaining details
No explicit supervision for reading order—it emerges from end-to-end training!
One of the paper's deepest insights: 2D images are hard for 1D language models. But you can understand 2D structure through two passes of 1D reasoning.
The 2D Understanding Problem
Consider a table. To understand cell (row 3, col 2), you need to know:
Raster scan only gives you 1D context—you see (3,1) but (1,2) was 100 tokens ago!
The Two-Pass Solution
DeepSeek-OCR 2 applies 1D causal attention twice, along different axes:
Query tokens left-to-right
Each position sees all left positions
Builds horizontal context
After this pass: each cell knows its row neighbors.
Query tokens top-to-bottom
Each position sees all above positions
Builds vertical context
After this pass: each cell knows its column neighbors.
Combined, cell (3,2) now has context from its entire row AND column—full 2D understanding using only 1D attention operations!
Mathematical Intuition
Think of it like matrix factorization. To capture a 2D relationship matrix R:
Instead of modeling all N² pairwise relationships directly (expensive!), we model N row relationships + N column relationships. This is O(N) instead of O(N²).
Why Two Passes Are Enough
A natural question: why not 3 passes? Or N passes?
- Information theory perspective: Most document structures are "low-rank" in the 2D sense. Tables have row and column headers. Documents have sections and paragraphs. Two passes capture most of this structure.
- Diminishing returns: Experiments show pass 3+ provides<1% improvement at 50% more compute. The ROI drops dramatically.
- Residual connections: The architecture uses residual connections, so information from pass 1 persists into pass 2's output.
The training of DeepSeek-OCR 2 involves careful data curation, multi-stage training, and specific loss functions designed for OCR tasks.
Training Data Composition
- • PDF documents (academic, legal, financial)
- • Scanned documents with various quality levels
- • Multi-column layouts, tables, figures
- • ~100 languages with focus on Chinese/English
- • Natural images with captions
- • Scene text (signs, labels, packaging)
- • Diagram and chart understanding
- • Preserves LLM language capabilities
- • Prevents catastrophic forgetting
- • 8,192 token sequences
Two-Stage Training
Loss Function
The training uses standard cross-entropy loss for next-token prediction:
The key insight: by making visual tokens causally ordered, the model learns to produce them in an order that minimizes this loss—which naturally becomes semantic order.
Training Efficiency
Production capacity: After training, DeepSeek-OCR 2 can process 200k+ pages per day on a single A100-40G. With the full 20-node cluster, this scales to 33 million pages per day for training data generation.
Use these demos to build deep intuition about Visual Causal Flow. Each explores a different aspect of the paper.
How Queries Learn
The model learns importance scores through end-to-end training. Regions that help predict the next token get higher importance. The top-K regions by importance become the query targets.
Causal Dependency
Query 2 sees Query 1's selection before choosing. This cascade means later queries adapt to earlier choices, creating a coherent semantic sequence.
Token Count vs. Performance
Why Fewer Tokens = Better?
It seems counterintuitive: more information should mean better understanding, right? But not all visual information is equal. A sky region above a document contributes little but uses tokens. DeepSeek's causal flow queries compress uninformative regions and expand important ones—like dynamically adjusting resolution based on semantic importance.
Token Allocation Comparison
Experiment: Document Types
Switch between document types in the demos above. Notice how:
- • Tables: queries follow row-then-column order
- • Articles: queries follow headline → body flow
- • Mixed layouts: queries adapt to content structure
Experiment: Edge Cases
Try these scenarios to understand limitations:
- • Empty regions: queries skip them efficiently
- • Very small text: may need higher resolution
- • Non-standard layouts: may require more tokens
The Key "Aha" Moment
Traditional vision models treat all tokens equally—the sky above a document gets as many tokens as the text. Visual Causal Flow compresses uninformative regions and expands important ones.
It's like dynamic resolution based on meaning, not spatial position. A 10-word title might get 5 query tokens, while a 1000-pixel sky gets 1.
What happens when we remove or modify key components? Ablations reveal which design choices actually matter.
Component Ablation Results
| Configuration | Score | Δ |
|---|---|---|
| Full Model (Visual Causal Flow) | 91.09% | — |
| Without causal attention (bidirectional) | 87.2% | -3.9% |
| Without two-pass (single pass only) | 88.5% | -2.6% |
| Fixed raster order (no reordering) | 85.3% | -5.8% |
| 32 queries instead of 64 | 89.1% | -2.0% |
| 128 queries instead of 64 | 91.3% | +0.2% |
Critical Component: Causal Attention
Removing causal attention (making it bidirectional) drops performance by 3.9%. This confirms that the ordering emerges from causality, not just from having learnable queries.
Diminishing Returns: Query Count
Doubling queries from 64 to 128 gives only 0.2% improvement. But halving to 32 loses 2.0%. 64 is the sweet spotfor efficiency vs. quality.
The Raster Order Baseline
The most important ablation: what if we keep everything else but use fixed raster order?
The 5.8% gap proves that ordering matters enormously—it's not just about the architecture or the attention mechanism, but specifically about learning semantic order.
Resolution Sensitivity
Higher resolution helps, but with diminishing returns:
- • 512×512 (64 tokens): 85% — good for simple documents
- • 768×768 (144 tokens): 89% — good for most documents
- • 1024×1024 (256 tokens): 91% — optimal for complex layouts
- • 1280×1280 (400 tokens): 91.3% — minimal gain, 56% more tokens
Let's analyze the benchmark results in detail to understand where Visual Causal Flow excels and where it still has room to improve.
OmniDocBench Results (Edit Distance - Lower is Better)
| Model | Tokens | Overall | English | Chinese |
|---|---|---|---|---|
| GOT-OCR 2.0 | 256 | 0.287 | 0.287 | 0.411 |
| MinerU 2.0 | 6,790 | 0.133 | 0.133 | 0.238 |
| InternVL | 2,304 | 0.139 | 0.139 | 0.210 |
| DeepSeek-OCR 2 (Small) | 100 | 0.221 | 0.221 | 0.284 |
| DeepSeek-OCR 2 (Base) | 256 | 0.137 | 0.137 | 0.240 |
| DeepSeek-OCR 2 (Gundam) | 795 | 0.127 | 0.127 | 0.181 |
Document-Type Breakdown
Best Performance Categories:
- • Financial reports: 0.022 edit distance
- • Books: 0.037 edit distance
- • Slides: 0.085 edit distance
More Challenging Categories:
- • Newspapers: 0.122 (complex layouts)
- • Dense tables: 0.098 (many cells)
- • Handwritten: 0.185 (not the focus)
Compression vs. Accuracy Tradeoff
One of the paper's key findings: there's a predictable relationship between compression ratio and accuracy.
This gives practitioners a clear guide: use Small mode (100 tokens) for simple documents, Gundam mode (795 tokens) for complex layouts.
How do you actually use DeepSeek-OCR 2 in your projects? Here's a practical guide to deployment and integration.
Quick Start with Transformers
Available Prompt Modes
Convert the document to markdown.
Full document OCR with structureFree OCR.
Raw text extractionParse the figure.
Chart/diagram extractionDescribe this image in detail.
General VQA
Resolution Recommendations
- Simple docs: base=512, image=512
~64 tokens, fastest - Standard docs: base=768, image=768
~144 tokens, balanced - Complex layouts: base=1024, image=768
~400 tokens, best quality - Dense text: base=1280, image=1024
~795 tokens, maximum detail
Production Deployment with vLLM
For high-throughput production use, vLLM provides significantly better performance:
vLLM enables batched inference, continuous batching, and PagedAttention for optimal GPU utilization. Expect 3-5× throughput improvement over native Transformers.
Common Pitfalls
- • Memory: BF16 model needs ~6GB VRAM. FP32 needs ~12GB.
- • Flash Attention: Required for efficient inference. Install with
pip install flash-attn - • Image preprocessing: Model expects RGB images. Convert RGBA before inference.
- • Prompt format: The
<image>and<|grounding|>tags are required.
No paper is complete without discussing limitations. Understanding where Visual Causal Flow struggles points toward future research directions.
Current Limitations
Performance on handwritten documents is noticeably weaker (0.185 vs 0.037 on printed). The model was trained primarily on printed/digital documents.
Beyond 20× compression, accuracy drops to ~60%. For very long documents with dense text, you may need to process pages individually.
Very unusual layouts (artistic documents, infographics with no clear reading order) can confuse the causal flow mechanism.
Training focused on Chinese and English. Performance on other languages (especially RTL scripts like Arabic) may be degraded.
Future Research Directions
The same idea could apply to video understanding: reorder frames by semantic importance rather than temporal order. A 10-minute video might be compressed to key moments.
For 3D understanding (point clouds, medical imaging), extend to three cascaded 1D passes: x-axis, y-axis, z-axis.
Currently 64 queries is fixed. A learned mechanism could dynamically allocate more queries to complex images, fewer to simple ones.
Extend to jointly reorder visual and audio tokens for video with speech. Learn unified semantic order across modalities.
The Broader Impact
Visual Causal Flow represents a paradigm shift in how we think about vision-language integration:
- Before: Visual information → fixed transform → language model. The visual encoder's output format was fixed.
- After: Visual information → learned reordering → language model. The transform adapts to match the language model's inductive bias.
This suggests a broader principle: when combining two modalities, the interface between them should be learnable, not fixed. The optimal representation depends on what the downstream model expects.
Key Takeaways
The core insight: How you order information matters as much as what information you have. By matching the visual representation to the language model's expectations, we unlock dramatic efficiency gains.