DeepSeek-OCR
Contexts Optical Compression: How to squeeze 6,000+ text tokens into just 100-800 visual tokens while maintaining 97% accuracy. The key to practical, production-scale document AI.
This paper presents DeepEncoder—a vision encoder that combines SAM's local perception with CLIP's global understanding, connected by a 16× convolutional compressor. The result: OCR at 200,000+ pages per day on a single A100 GPU.
Your Deep Dive
Imagine you have a 50-page PDF document. You want an LLM to answer questions about it. What are your options?
Option 1: Raw Text Extraction
- • OCR each page → text
- • Concatenate all text
- • Feed to LLM
Exceeds most LLM context windows. Even if it fits, costs ~$6 per query with GPT-4.
Option 2: Visual Compression
- • Encode each page as image
- • Compress to visual tokens
- • LLM decodes visual tokens
20× compression. Fits in context. Costs ~$0.30 per query. And it works!
The Key Insight
Text has incredible redundancy when represented as characters/tokens. The word "information" is 11 characters, but visually it's a single recognizable unit. A well-trained visual encoder can capture this semantic density directly from pixels.
Of course, compression has limits. If you compress too aggressively, information is lost. The question is: what's the optimal compression ratio? This paper provides the first systematic study of this tradeoff.
Before diving into architecture, let's understand the real-world impact. Document AI is a massive market with specific operational requirements.
Production Requirements
Enterprises process millions of documents daily. A legal firm might ingest 100,000 contracts per month.
At $0.03/1K tokens, processing 100K pages with raw text costs ~$180K/month. With compression, it's ~$9K.
Users expect real-time responses. 300K token context = 30+ second inference. 15K tokens = 3 seconds.
Financial documents require 99%+ accuracy. Medical records even higher. Compression must preserve critical information.
DeepSeek-OCR Production Numbers
The Business Case
At 33M pages/day with 20 nodes, the cost per page is approximately $0.002 in compute. Traditional OCR pipelines cost 10-100× more when you factor in the LLM inference costs for the extracted text. This makes previously impossible use cases economically viable.
DeepEncoder is the heart of the system—a 380M parameter vision encoder that achieves extreme compression while preserving semantic content.
Architecture Overview
Window attention for local perception. Processes high-resolution patches without quadratic memory cost.
Two conv layers with stride 2 reduce spatial dimensions by 4× in each dimension, achieving 16× total token reduction.
Global attention for semantic knowledge. Can now afford dense attention because tokens are compressed.
Why Two Encoders?
SAM excels at low-level visual features (edges, shapes, spatial structure) while CLIP excels at semantic understanding (what objects mean, text-image alignment). The compressor sits between them, learning which SAM features matter for CLIP's semantic task. This two-stage design captures both local detail and global meaning.
Why This Design?
The key insight is that local and global processing have different computational needs:
- Local (SAM): Processing 4,096 tokens with window attention is O(4096 × 49) = O(200K) operations per layer.
- Global (CLIP): Processing 256 tokens with full attention is O(256²) = O(65K) operations per layer.
- Combined: This hybrid is 10× more efficient than full attention on 4,096 tokens, which would be O(16M) operations.
The choice of SAM and CLIP isn't arbitrary—each brings specific strengths that complement the other perfectly.
SAM (Segment Anything Model)
Facebook's SAM was trained to segment any object in any image. This gives it:
- Fine-grained perception: Can distinguish individual letters, table cells, diagram elements
- Edge detection: Knows where text ends and background begins
- Window attention: Efficient at high resolution without memory explosion
CLIP (Contrastive Language-Image)
OpenAI's CLIP was trained on 400M image-text pairs. This gives it:
- Semantic understanding: Knows what text means, not just what it looks like
- Layout comprehension: Understands document structure (headers, body, etc.)
- Global context: Can relate elements across the entire image
The Synergy
Neither model alone would work well for OCR:
Knows shape of letters but not meaning. Would output "Q" as similar to "O" because shapes are similar.
Good at overall understanding but lacks precision. Would recognize "invoice" but might miss specific numbers.
SAM extracts precise local features, compressor reduces dimensionality, CLIP adds semantic context. The decoder gets both precision and understanding.
Transfer Learning Advantage
Both SAM and CLIP come with excellent pre-trained weights. DeepEncoder fine-tunes them for OCR rather than training from scratch. This gives:
- • Faster training (2 epochs vs. typical 10+)
- • Better generalization to unseen document types
- • Lower risk of overfitting to training data
The compressor is the bridge between SAM's fine-grained output and CLIP's semantic processing. It's deceptively simple but critically important.
Compressor Architecture
Why Convolutions, Not Attention?
The compressor could have used attention-based pooling (like Perceiver), but convolutions have specific advantages here:
Each output token has a fixed 4×4 receptive field. Spatial relationships are preserved deterministically.
O(N) vs O(N²) for attention. At 4,096 tokens, this is 16× faster.
The 3×3 kernel learns what local patterns to combine. Different filters for text vs. images vs. tables.
Same text processed the same way regardless of position. Important for document generalization.
The 16× Number
Why 16× and not 8× or 32×? The paper experimented with different ratios:
- • 4×: 1,024 tokens—CLIP attention too expensive, marginal accuracy gain
- • 16×: 256 tokens—sweet spot for accuracy vs. compute
- • 64×: 64 tokens—too aggressive, loses fine details
The decoder reconstructs text from compressed visual tokens. DeepSeek uses their 3B MoE (Mixture of Experts) architecture—a clever choice for OCR.
DeepSeek-3B-MoE Architecture
Full model size including all experts
Per token during inference
(3B knowledge, 570M compute)
Why MoE for OCR?
OCR involves diverse "skills": recognizing Latin letters, Chinese characters, numbers, mathematical symbols, tables, etc. MoE naturally specializes:
The router learns to activate different expert combinations for different document types. A Chinese financial report routes differently than an English research paper.
The Reconstruction Objective
The decoder is trained with standard language modeling:
Given compressed visual tokens, predict the original document text autoregressively. The compression ratio (visual tokens / text tokens) determines difficulty.
DeepSeek-OCR supports multiple resolution modes, letting you trade accuracy for speed based on document complexity.
Resolution Modes
| Mode | Resolution | Tokens | Best For |
|---|---|---|---|
| Tiny | 512×512 | 64 | Simple forms, receipts |
| Small | 640×640 | 100 | Standard documents |
| Base | 1024×1024 | 256 | Detailed documents |
| Large | 1280×1280 | 400 | Small text, tables |
| Gundam | Dynamic | ~795 | Complex multi-column |
| Gundam-M | Dynamic+ | ~1,120 | Dense newspapers |
Click to select primary • Right-click to compare
Token Count Comparison
Accuracy Trade-off
Amber line: 97% production threshold
Small Mode
100 tokensThe sweet spot for most applications. Excellent accuracy-efficiency tradeoff for invoices, contracts, and general documents.
Choosing the Right Mode
The 97% accuracy sweet spot. Only scale up if you see errors in your specific document types.
High-volume batch jobs where occasional errors are acceptable. Receipts, simple forms.
Legal, medical, financial documents where errors have high cost. Worth the extra tokens.
Use document classifiers to route different types to appropriate resolutions automatically.
Dynamic Resolution (Gundam Mode)
For complex documents, Gundam mode dynamically tiles the image:
The model automatically determines n (2-9) based on document aspect ratio and text density.
When to Use Which Mode
- • Tiny/Small: High-volume batch processing
- • Base: General-purpose default
- • Large: Financial statements, contracts
- • Gundam: Newspapers, dense layouts
Automatic Mode Selection
In production, you can let the model auto-select based on a quick analysis of text density. This typically adds <10ms latency but optimizes token usage.
Training DeepSeek-OCR required massive data curation and careful multi-stage optimization.
Training Data Composition
Two-Stage Training
Pipeline Parallelism Layout
PP Stage 1: CLIP-large
PP Stage 2: Decoder layers 1-6
PP Stage 3: Decoder layers 7-12
This layout balances compute across stages. Encoder stages are memory-bound, decoder stages are compute-bound.
The central question: how much can you compress before losing critical information? This paper provides the first systematic study.
Compression Ratio vs. Accuracy
The 10× Threshold
Notice the steep drop after 10× compression. Below 10×, accuracy stays above 90%. Beyond 10×, each additional 2× compression costs roughly 10% accuracy. This is the fundamental information-theoretic limit of visual encoding.
Compression Ratio Results
| Compression | Vision Tokens | Text Tokens | Accuracy |
|---|---|---|---|
| ~7× | 100 | 600-700 | 98.5% |
| ~10× | 100 | 900-1000 | 96.8% |
| ~13× | 64 | 800-900 | 83.8% |
| ~17× | 64 | 1000-1100 | 79.3% |
| ~20× | 64 | 1200-1300 | 59.1% |
The 10× Sweet Spot
The data reveals a clear threshold: below 10× compression, accuracy stays above 95%. Beyond 10×, it degrades rapidly.
For most documents with <1000 words per page, use Small mode (100 tokens). You'll get 97%+ accuracy.
For 2000+ word pages (newspapers, legal), use Gundam mode (795 tokens) to stay under 10× compression.
Why 20× Fails
At 20× compression, each visual token must encode ~20 text tokens worth of information. This exceeds the information capacity of the encoder. You're literally asking 64 numbers to represent 1,300 characters—the math doesn't work.
Let's analyze performance on OmniDocBench, the most comprehensive document understanding benchmark.
OmniDocBench Results (Edit Distance - Lower is Better)
| Model | Tokens/Page | Overall | English | Chinese |
|---|---|---|---|---|
| GOT-OCR 2.0 | 256 | 0.287 | 0.287 | 0.411 |
| MinerU 2.0 | 6,790 | 0.133 | 0.133 | 0.238 |
| DeepSeek-OCR Small | 100 | 0.221 | 0.221 | 0.284 |
| DeepSeek-OCR Base | 256 | 0.137 | 0.137 | 0.240 |
| DeepSeek-OCR Gundam | 795 | 0.127 | 0.127 | 0.181 |
Document Category Analysis
Excels At:
- • Financial reports: 0.022 (near perfect)
- • Books: 0.037 (excellent)
- • Slides: 0.085 (clean layouts)
- • Academic papers: 0.091
Challenges:
- • Newspapers: 0.122 (needs Gundam mode)
- • Dense tables: 0.098 (many small cells)
- • Handwritten: 0.185 (not the focus)
DeepSeek-OCR is designed for production from day one. Here's how to deploy it.
Quick Start
Hardware Requirements
- • Minimum: 16GB VRAM (RTX 4080)
- • Recommended: 40GB VRAM (A100)
- • BF16: ~6GB for inference
- • FP32: ~12GB for inference
Throughput by Hardware
- • RTX 4090: ~50K pages/day
- • A100-40G: ~200K pages/day
- • 8× A100: ~1.6M pages/day
- • 160× A100: ~33M pages/day
vLLM Deployment (Recommended)
vLLM provides 3-5× throughput improvement through continuous batching and PagedAttention.
Common Issues
- • OOM errors: Reduce batch size or use BF16
- • Slow inference: Enable Flash Attention 2
- • Poor accuracy: Check resolution mode for document type
- • CUDA errors: Requires CUDA 11.8+ and PyTorch 2.0+
Every system has limitations. Understanding them helps you use the tool effectively.
Current Limitations
Performance on handwritten documents is weaker. The model was trained primarily on printed/digital text.
Beyond 20× compression, accuracy drops sharply. Very dense documents may need to be split into sections.
Strong on Chinese and English, but performance on Arabic, Thai, and other scripts may be degraded.
Highly artistic layouts, overlapping text, and extreme rotations can confuse the model.
Future Directions
Can we push to 50× with new architectures? Techniques like learned tokenization and hierarchical compression show promise.
Currently processes pages independently. Future work could maintain context across pages for better document understanding.
Extend to video documents (presentations, lectures) where temporal context provides additional signal.
Key Takeaways
The breakthrough: Visual information is fundamentally more compressible than text tokens. By encoding documents as images first, we can achieve massive efficiency gains that make production-scale document AI economically viable.