DeepSeek-R1
The first open-source model to match OpenAI o1 on reasoning tasks. Built through pure reinforcement learning, R1 develops emergent reasoning behaviors—self-reflection, chain-of-thought, and even “aha moments”—without any supervised fine-tuning on reasoning data.
In September 2024, OpenAI released o1—a model that could “think” before answering, solving problems that stumped GPT-4. The secret? Test-time compute scaling. Instead of just generating tokens, o1 reasons through problems step by step, sometimes for thousands of tokens before producing an answer. DeepSeek-R1 is the first open model to match this capability.
Traditional LLMs
- ✗Answer immediately without deliberation
- ✗Fixed compute per problem regardless of difficulty
- ✗Struggle on multi-step reasoning
- ✗Can't recover from reasoning errors
- ✗AIME 2024: ~15% (GPT-4)
Reasoning Models (o1, R1)
- ✓Think before answering in <think> blocks
- ✓Spend more compute on harder problems
- ✓Chain-of-thought extends dynamically
- ✓Self-reflect and correct mistakes
- ✓AIME 2024: ~80% (R1, o1)
The Breakthrough: Open-Source Reasoning
DeepSeek-R1 is fully open—model weights, training details, and even the training code. This democratizes access to reasoning capabilities that were previously locked behind API calls. More importantly, the paper reveals how to build reasoning models: pure reinforcement learning, no supervised reasoning data required.
Why This Matters
Competition-level problems that require multi-step proofs
Complex algorithms that need careful planning and debugging
Hypothesis generation and systematic analysis
The traditional approach to teaching LLMs to reason is supervised fine-tuning (SFT) on chain-of-thought examples. This works, but it has a fundamental limit: the model can only learn reasoning patterns present in the training data. Reinforcement learning offers something better.
SFT Limitations
- •Imitation learning: Can only copy patterns from examples
- •Data bottleneck: High-quality CoT data is scarce and expensive
- •Format lock-in: Model learns specific reasoning format, not reasoning itself
- •No exploration: Can't discover novel reasoning strategies
RL Advantages
- •Self-improvement: Model discovers better strategies through trial and error
- •Outcome-based: Only care if answer is correct, not how you got there
- •Emergent behaviors: Surprising capabilities appear from simple rewards
- •No human ceiling: Can surpass human reasoning templates
The Key Insight
For math and coding, we have verifiable rewards. A math answer is either correct or wrong. Code either passes tests or doesn't. This means we can train with RL without needing expensive human feedback—just check the answer! DeepSeek-R1 uses this to train on hundreds of thousands of problems with automatic reward signals.
Reward Structure
+1 if final answer is correct, 0 otherwise
Small bonus for using <think>...</think> format
Penalty for mixing languages (solved in later stages)
Before building the full R1 pipeline, DeepSeek ran an experiment: what happens if you do pure RL on a base model with zero supervised fine-tuning? The result was R1-Zero—a model that spontaneously develops reasoning capabilities just from getting rewards for correct answers.
R1-Zero Training Setup
DeepSeek-V3-Base: A 671B parameter base model pre-trained on text, with no instruction-following or reasoning training.
Only accuracy rewards on math/coding problems. No human demonstrations, no chain-of-thought examples, no format requirements.
Emergent CoT
Without being told to “think step by step,” R1-Zero spontaneously develops long reasoning chains. It discovers that breaking problems into steps leads to more rewards.
Self-Reflection
The model learns to check its own work. Phrases like “Wait, let me verify...” and “That doesn't seem right...” emerge naturally from RL.
“Aha Moments”
R1-Zero exhibits sudden insights during reasoning, re-evaluating approaches when stuck. The paper calls these “aha moments”—a hallmark of human problem-solving.
R1-Zero Performance
These scores surpass many models trained with supervised reasoning data.
The Readability Problem
R1-Zero has a major drawback: its outputs are messy. It mixes languages (Chinese and English randomly), uses inconsistent formatting, and produces reasoning that's hard for humans to follow. This is why the full R1 pipeline adds cold-start SFT and rejection sampling.
Standard RL for LLMs uses PPO (Proximal Policy Optimization), which requires training a separate “critic” model to estimate value functions. This doubles training compute. DeepSeek developed GRPO (Group Relative Policy Optimization)—a simpler algorithm that uses the group average as baseline.
GRPO: Group Relative Policy Optimization
Step 1: For each question, sample 8 different outputs from the model. Each output is a potential reasoning chain and final answer.
Why GRPO Instead of PPO?
PPO requires a critic model (same size as policy) to estimate value functions. GRPO skips this by using the group average as baseline. This halves training compute while achieving similar or better results.
PPO (Traditional)
- •Critic Model: Same size as policy (671B params)
- •Value Estimation: Critic predicts expected future reward
- •Advantage: A = r - V(s) (reward minus value estimate)
- •Memory: 2× model parameters during training
GRPO (DeepSeek)
- •No Critic: Just the policy model
- •Group Baseline: Average reward of sampled outputs
- •Advantage: A_i = r_i - mean(r_group)
- •Memory: 1× model parameters (half the cost!)
The GRPO Formula
Training Hyperparameters
Pure RL (R1-Zero) works but produces messy outputs. The full R1 pipeline combines RL with targeted SFT to get the best of both worlds: emergent reasoning from RL and readable outputs from SFT.
Multi-Stage Training Pipeline
Base Model
DeepSeek-V3-Base: Pre-trained on massive text data, no reasoning training yet
Why Multi-Stage?
Pure RL (R1-Zero) works but produces poor readability and language mixing. The multi-stage approach uses cold start to establish format, then RL for reasoning, then rejection sampling for quality data, then more RL for refinement.
Stage 0: Base Model
Start with DeepSeek-V3-Base (671B params). This is a pre-trained model with no instruction-following or reasoning capabilities. AIME 2024: 15.6%.
Stage 1: Cold Start SFT
Fine-tune on ~1,000 high-quality chain-of-thought examples. This establishes the <think>...</think> format and readable reasoning style. The examples are carefully curated for quality, not quantity.
Stage 2: RL Round 1 (Reasoning-Only)
Large-scale GRPO training on math and coding problems. The model develops reasoning capabilities while maintaining the format from cold start. AIME jumps from ~20% to 71%.
Stage 3: Rejection Sampling
Generate 64 solutions per problem. Keep only correct ones with high quality scores. This creates ~800K high-quality SFT examples from the RL-trained model's own outputs.
Stage 4: RL Round 2 (All Tasks)
Second round of GRPO, now including writing, general QA, and safety tasks alongside reasoning. This refines quality and ensures the model works well on diverse tasks.
Why This Specific Order?
Cold start → RL → Rejection → RL is carefully designed. Cold start prevents RL from developing bad habits. The first RL round focuses only on reasoning to avoid distraction. Rejection sampling creates clean data from the RL model's strengths. The second RL round then generalizes to all tasks.
The most fascinating aspect of R1-Zero and R1 is the emergent behaviors that arise from pure RL training. These capabilities weren't programmed or demonstrated—they evolved because they help get correct answers.
Extended Chain-of-Thought
R1 produces much longer reasoning traces than traditional CoT models. It's not unusual to see 10,000+ tokens of reasoning for a single problem.
Let me break this down...
First, I'll consider...
[...thousands of tokens...]
</think>
Self-Reflection
The model spontaneously checks its work and catches errors. This emerged purely from RL—catching errors leads to more correct final answers.
"Let me verify this step..."
"I made an error above..."
“Aha Moments”
Sometimes R1 gets stuck, tries different approaches, and then has a breakthrough realization. These moments are strikingly human-like.
"What if I try it from a different angle..."
"Oh! I see it now! The key insight is..."
Dynamic Compute Allocation
R1 automatically spends more tokens on harder problems. Easy problems get short reasoning; hard problems get extensive exploration.
Why Do These Behaviors Emerge?
The RL reward is simple: correct answer = reward. But the model discovers that certain behaviors lead to more correct answers. Self-reflection catches errors. Aha moments help unstick. Longer reasoning allows more complex problem decomposition. These aren't programmed—they're discovered by the model as winning strategies in the RL game.
Unwanted Emergent Behaviors (R1-Zero)
Pure RL also produces undesirable behaviors: language mixing (Chinese/English randomly), poor formatting, repetitive loops, and overly verbose reasoning. The multi-stage pipeline with cold start SFT addresses these issues while preserving the good emergent behaviors.
The cold start phase is surprisingly simple yet critical. It's just ~1,000 examples of good reasoning, but it sets the format and style that RL then builds upon. Without cold start, RL produces capable but unreadable outputs.
Cold Start Data
What Cold Start Establishes
- ✓<think> format: Reasoning in a distinct block before answer
- ✓Consistent language: English (or Chinese) throughout, not mixed
- ✓Readable structure: Clear step labeling, organized flow
- ✓Good formatting: Markdown, equations, proper spacing
What Cold Start Does NOT Teach
- ✗Deep reasoning: That comes from RL
- ✗Self-reflection: Emerges from RL rewards
- ✗Problem-solving strategies: Discovered through RL
- ✗Extended thinking: RL learns to think longer when needed
Example Cold Start Data
Looking for two numbers that multiply to 6 and add to 5.
Those numbers are 2 and 3.
So x² + 5x + 6 = (x + 2)(x + 3) = 0
Therefore x = -2 or x = -3
Simple, clean, well-formatted. This sets the template that RL then expands upon.
After RL Stage 1, the model reasons well but the training data is sparse. Rejection sampling generates abundant high-quality data from the model itself: sample many solutions, keep only the best ones.
Rejection Sampling Process
For each problem, generate 64 different solutions using temperature sampling.
For math: verify final answer. For code: run test cases. Only correct solutions proceed.
Among correct solutions, score for readability, conciseness, and reasoning clarity.
Select top-scoring solutions for the SFT dataset. Creates ~800K high-quality examples.
Why It Works
- •Self-distillation: Model learns from its own best outputs
- •Quality filtering: Only correct, high-quality solutions kept
- •Diverse reasoning: Different solutions for same problem
- •Scale: Can generate millions of examples cheaply
Statistics
Beyond Reasoning Tasks
Rejection sampling also creates data for non-reasoning tasks (writing, summarization, QA). For these, quality scoring is done by a separate reward model or LLM judge. This makes the final R1 a strong general-purpose model, not just a reasoning specialist.
R1's 671B parameters are too large for most users. DeepSeek's solution: distillation. Train smaller models (1.5B to 70B) on R1's outputs. The student models learn R1's reasoning patterns at a fraction of the size.
Distilled Model Family
Distillation Process
- 1.Start with a pre-trained base model (Qwen or Llama)
- 2.Generate R1 outputs for 800K+ prompts
- 3.Fine-tune student on (prompt, R1-output) pairs
- 4.No RL needed—pure supervised learning
Distillation Results (AIME 2024)
The Surprising Finding
The 14B distilled model achieves 69.7% on AIME—nearly matching the full 671B R1 (79.8%) at 50× smaller size. This suggests that the reasoning patterns can be efficiently compressed. The bottleneck isn't raw capability but learning the right thinking process.
Open Source Impact
All distilled models are open-weight under the MIT license. A 7B model that achieves 55% on AIME can run on consumer hardware. This puts reasoning capabilities in reach of researchers, students, and hobbyists who can't afford API costs.
DeepSeek-R1 was evaluated on a comprehensive suite of benchmarks covering mathematics, coding, scientific reasoning, and general knowledge. The results show state-of-the-art performance on reasoning tasks while maintaining strong general capabilities.
Mathematics & Logic
Coding
General Knowledge
DeepSeek-R1 is the first open model to match OpenAI o1 on reasoning benchmarks. The comparison reveals similar capabilities achieved through different approaches.
Head-to-Head Comparison
| Benchmark | DeepSeek-R1 | OpenAI o1 | GPT-4o |
|---|---|---|---|
| AIME 2024 | 79.8% | 79.2% | 13.4% |
| MATH-500 | 97.3% | 96.4% | 76.4% |
| Codeforces Elo | 2029 | 1891 | 807 |
| GPQA Diamond | 71.5% | 77.3% | 56.1% |
| MMLU | 90.8% | 91.8% | 87.2% |
DeepSeek-R1 Advantages
- ✓Open weights: Full model publicly available
- ✓Open training: Paper reveals full methodology
- ✓Distilled models: Small versions available
- ✓Self-hostable: Run on your own infrastructure
- ✓Lower cost: API significantly cheaper
OpenAI o1 Advantages
- ✓First mover: Released 4 months earlier
- ✓GPQA leader: Better on PhD-level science
- ✓Multimodal: o1 supports images
- ✓API stability: Well-established infrastructure
- ✓Safety investment: Extensive safety testing
The Bigger Picture
R1's release proves that reasoning capabilities can be achieved openly. The gap between closed and open models—once thought to be fundamental—has narrowed dramatically. This democratizes access to advanced AI reasoning and accelerates research by making the techniques inspectable and reproducible.
Despite its achievements, R1 has notable limitations. Understanding these helps set appropriate expectations and identifies areas for future research.
Current Limitations
- ✗Text-only: No image or audio understanding
- ✗Language mixing: Sometimes switches languages in output
- ✗Slow inference: Long reasoning = high latency
- ✗Overthinking: Sometimes reasons excessively for simple tasks
- ✗Factual errors: Can hallucinate during long reasoning
- ✗Safety concerns: Extended reasoning could enable jailbreaks
Future Directions
- →Multimodal R1: Reasoning over images and video
- →Faster inference: Speculative decoding, distillation
- →Compute-optimal: Learn when to think long vs short
- →Better distillation: Smaller models that match R1
- →Agentic applications: Reasoning for tool use, planning
- →Safety research: Understanding risks of reasoning models
The Overthinking Problem
R1 sometimes produces thousands of tokens for simple questions like “What is 2+2?” The model hasn't learned when to think briefly. This wastes compute and increases latency.
“What is the capital of France?” → 500 tokens of reasoning
Simple questions → brief answers; complex questions → long reasoning
Open Questions for the Field
- ?What is the fundamental limit of reasoning capability in LLMs?
- ?Can we achieve reasoning without the massive compute of test-time scaling?
- ?How do we ensure safety when models can reason through restrictions?
- ?What other capabilities might emerge from pure RL training?
DeepSeek-R1: The Complete Picture
R1 demonstrates that reasoning capabilities emerge naturally from reinforcement learning with simple rewards. No supervised reasoning data required—just reward correct answers and let the model figure out how to think. The result matches closed models while being fully open.