DeepSeek R1 Paper Expands to 86 Pages, Details Reinforcement Learning for AI Reasoning

DeepSeek has released an expanded 86-page research paper for its R1 model, detailing how artificial intelligence reasoning capabilities can be enhanced through reinforcement learning alone. The updated paper, which grew from an initial 22 pages, provides a fully reproducible technical report for the open-source community.

The new content in the DeepSeek-R1 paper includes a precise data recipe, specifying 26,000 math problems and 17,000 code snippets used in training. It also describes the infrastructure, including schematic diagrams for vLLM/DualPipe settings, and a breakdown of training costs, which totaled approximately $294,000. The paper further reviews "failed attempts," explaining why the Process Reward Model (PRM) was not successful, and includes a 10-page safety report detailing evaluation and risk analysis.

Schematic diagram of vLLM/DualPipe architecture on a transparent screen in a modern server room.

DeepSeek R1 Performance and Comparison

DeepSeek R1's capabilities in various aspects are presented as comparable to OpenAI o1, and in some areas, it surpasses o1-mini, GPT-4o, and Claude 3.5. The paper includes a systematic comparison with these models and DS-V3.

Evaluation results cover mathematical reasoning, coding, general knowledge, and instruction following. DeepSeek-R1 generally outperforms DS-V3 on educational knowledge benchmarks such as MMLU, MMLU-Pro, and GPQA Diamond, with significant improvements in STEM-related problems attributed to reinforcement learning. The model also demonstrated strong performance in long-context question-answering tasks (FRAMES).

In math and code tasks, DeepSeek-R1 is largely on par with OpenAI-o1-1217. While OpenAI-o1-1217 performed better on Aider for practical programming tasks, DeepSeek-R1 showed similar performance on SWE Verified. DeepSeek suggests that a lack of engineering-related reinforcement learning training data may be limiting R1's current capabilities in this area.

DeepSeek-R1's scores in the AIME Math Competition have surpassed the average human level, and in the Codeforces Programming Competition, it outperformed 93.6% of participants. However, human experts still generally outperform DeepSeek-R1 in GPQA Scientific Q&A. DeepSeek posits that internet access could enable R1 to match or exceed human levels in this domain.

Digital trophy icon glowing over abstract math equations and code, symbolizing AI competition success.

Human evaluations using ChatbotArena reflected impressive results for R1, particularly in "style control," where it tied for first place with OpenAI-o1 and Gemini-Exp-1206. DeepSeek highlights this as a significant milestone for an MIT-licensed open-source model achieving performance comparable to multiple closed-source AI systems, especially given its lower usage cost.

The paper details the scale of reinforcement learning data, including 26,000 math, 17,000 code, 22,000 STEM, 15,000 logic, and 66,000 general samples. The fine-tuning phase involved approximately 800,000 samples covering reasoning, general instruction tasks, and format/language consistency.

Distillation and Training Costs

DeepSeek explored whether R1's learned reasoning ability could be transferred to smaller models. By acting as a "teacher" model to generate high-quality reasoning trajectory data, DeepSeek used supervised fine-tuning (SFT) to distill this ability to smaller "student" models ranging from 1.5B to 70B parameters. This process allowed smaller models to learn effective reasoning patterns without re-exploring the reward space, leading to improved performance across the board for distilled models.

The training of DeepSeek-R1-Zero utilized 64x8 H800 GPUs for approximately 198 hours. The DeepSeek-R1 training phase, using the same GPU configuration, was completed in about 80 hours. Building the supervised fine-tuning dataset consumed approximately 5,000 GPU hours, contributing to the total cost of $294,000.

Abstract visual of a large glowing sphere transferring data streams to a smaller glowing sphere, symbolizing model distillation.

Emergent Intelligence in DeepSeek-R1-Zero

On the MATH dataset, DeepSeek-R1-Zero demonstrated human-like performance. The model quickly mastered simpler reasoning tasks early in training, while its ability to solve complex problems (difficulty levels 3–5) significantly improved with continued training. For instance, accuracy for level 4 problems increased from approximately 0.78 to 0.95, and for level 5 problems, it rose from about 0.55 to 0.90. DeepSeek observed that reinforcement learning plays a crucial role in generating long chains of intermediate tokens and that AI models learn different forms of reflection at specific training stages. The frequency of reflective vocabulary, such as "wait," "mistake," and "retry," increased by 5 to 7 times compared to the beginning of training.

Graph showing increasing accuracy of AI in complex math problems over training time, observed by a researcher.

Safety Considerations

DeepSeek-R1's safety risk assessment covers five aspects: the risk control system for its official service, evaluation against six public safety benchmarks, classification research based on internal safety test sets, multilingual safety evaluation, and robustness against jailbreak attacks.

The risk control system involves filtering potential risky conversations using predefined keyword lists and a model-based review by DeepSeek-V3 for flagged conversations. Experimental results indicate that DeepSeek-R1's overall safety performance is comparable to other advanced models, though it performed worse on "intellectual property" issues in the HarmBench test. In other safety categories, such as discrimination, bias, violence, extremism, and privacy infringement, R1 performed stably.

DeepSeek constructed an internal safety evaluation dataset with 1,120 test questions across 4 primary categories and 28 subcategories of content safety challenges. With risk control enabled, the unsafe rate significantly decreased, although the refusal rate increased to approximately 25%. R1 performed well in handling illegal and criminal issues and ethical matters but was average in addressing discrimination, bias, and harmful behaviors.

A multilingual safety test set of 9,330 questions in 50 languages showed that DeepSeek-V3 (86.5%) and DeepSeek-R1 (85.9%) achieved overall safety scores close to Claude-3.7-Sonnet (88.3%) with risk control.

Jailbreak attack tests concluded that such attacks pose a significant threat to all models, reasoning models rely more on risk control systems, and open-source models face higher jailbreak risks.

Digital shield deflecting jagged red lines, symbolizing AI safety measures against jailbreak attacks.

Key Takeaways

The paper emphasizes the importance of base models and verifiers. Initial attempts with smaller models for reinforcement learning (RL) training did not yield substantial performance improvements, leading to the use of larger, more capable models where significant gains from pure RL training were observed. This indicates that the effectiveness of reinforcement learning depends on the base model's capacity.

Verifiers are crucial for the training effectiveness of DeepSeek-R1-Zero, as they ensure the reliability and accuracy of the reward signal. DeepSeek found that rule-based Reward Models (RMs) and using large language models to check consistency with standard answers effectively mitigate reward cheating.

The multi-stage training pipeline, incorporating supervised fine-tuning (SFT) and reinforcement learning (RL), is deemed indispensable. Relying solely on RL can lead to reward cheating and suboptimal behavior, while relying only on SFT may limit the model's ability to improve reasoning through exploration. DeepSeek also noted failures with methods like Process Reward Model (PRM) and Monte Carlo Tree Search (MCTS, though these methods are not inherently ineffective for building reasoning models.