P-GenRM Introduces Personalized Generative Reward Model for Open-Domain AI

The Tongyi Lab Natural Language Intelligence Team has developed P-GenRM, a personalized generative reward model designed for open-domain scenarios. This model addresses the challenge of traditional reward systems in accommodating diverse individual preferences, which often vary significantly for the same question across different users and situations. P-GenRM has been accepted as an Oral presentation at ICLR 2026.

Previous personalization methods typically embed user preferences, such as demographic attributes, user features, historical conversations, and preference tags, directly into models. However, these approaches often face two primary limitations: static preference modeling and poor generalization for new users.

Addressing Static Preference Modeling and Cold Start

Many existing methods simplify user preferences into a few fixed dimensions, such as "more concise" or "more detailed," which struggles to capture the nuanced and dynamic nature of real-world preferences. For example, a user might require a structured response in a professional context but a relaxed tone in a casual chat. Similarly, expectations for "travel advice" differ significantly from those for "life advice."

The second limitation, poor generalization for new users, commonly known as the "cold start" problem, arises when models lack sufficient historical data to adapt to the preferences of new users effectively.

Hand holding a tablet displaying P-GenRM's structured scoring chain with persona, rubric, and itemized scores.

P-GenRM's innovation lies in generating a structured "scoring chain" rather than a black-box score. This process is analogous to an examiner creating personalized grading rubrics. The model first analyzes the user's persona and preferences for a given scenario, then defines a grading rubric with weighted criteria, and finally scores each candidate answer item by item to produce a total score.

Conceptually, P-GenRM's output comprises three elements: a Persona/Preference Profile identifying the user's characteristics and priorities in the current scenario; a Rubric/Scoring Criteria, such as "Accuracy 0.35, Structure 0.25, Tone Friendliness 0.20, Creativity 0.20"; and an Itemized Scoring + Total Score, which provides interpretable reasons and scores for each criterion. This approach transforms preferences from an opaque vector within the model into a readable, adjustable, and auditable scoring process.

Three-Stage Training Framework

To enable the model to learn this "set rules first, then score" capability, P-GenRM utilizes a three-stage training framework.

Initially, supervised fine-tuning (SFT) trains the model to convert explicit user preferences (e.g., "I prefer concise answers") and implicit preferences (e.g., stylistic tendencies from conversation history) into a comprehensive scoring chain. This stage teaches the model to summarize user preferences in specific contexts and derive corresponding scoring criteria from various preference signals.

To address scenarios where user preferences are not explicitly stated, reinforcement learning (RL) is employed to enhance the model's ability to generate scoring chains. Through RL training, P-GenRM learns to infer robust evaluation criteria even with incomplete information, ensuring the continuity and rationality of the scoring process.

Finally, an "easy-to-hard" curriculum learning strategy exposes the model to a higher proportion of challenging samples, gradually improving its accuracy in judging difficult negative samples.

Three interconnected glowing spheres representing the three-stage training framework of P-GenRM.

Test-Time User-Based Scaling for Enhanced Stability

Even with advanced training, personalized inference can be affected by noise from historical conversations, insufficient current scenario signals, or limited new user data. P-GenRM addresses this through a Test-time User-based Scaling mechanism during inference, employing "think multiple times" and "refer to similar people" strategies.

For a single user, the model samples the scoring chain multiple times and aggregates the results, generating different scoring rubrics to make a comprehensive decision. This method effectively reduces deviations caused by single-generation errors.

Additionally, P-GenRM clusters user preferences offline into several User Prototypes. During inference, the system identifies the prototype most similar to the current user, selects preference signals from similar users, and generates a set of "reference scores" that are merged with individual-level results. This dual approach offers two key benefits: noise reduction, as group-level "preference priors" are less susceptible to single historical interactions, and enhanced cold-start performance for new users through "similar users."

Abstract visualization of a central user surrounded by multiple thought bubbles and connected to 'similar users,' representing test-time scaling.

Performance and Downstream Applications

Systematic evaluation on the PersonalRewardBench benchmark, which includes Chatbot Arena-personalized and PRISM-personalized datasets, demonstrated P-GenRM's effectiveness. The 8B version of P-GenRM outperformed existing methods, showing an average improvement of approximately 2.3%. Notably, the 8B model surpassed the previously strongest 70B baseline model, suggesting that architectural design can enable smaller models to achieve significant advancements in personalization tasks.

Introducing a dual-layer extension of individual and prototype preferences further improved accuracy by about 3%. This indicates that "referring to similar users" provides effective incremental information beyond simply scaling computational power.

P-GenRM's practical value was also verified in downstream tasks by training Policy models. An 8B policy combined with P-GenRM achieved statistically significant personalized alignment benefits under DPO/GRPO settings, in some cases exceeding the performance of 70B models. This suggests P-GenRM acts as both a "judge" and a "coach" for AI systems.

Additional evidence highlighted P-GenRM's robustness: it maintains stable performance with as few as a few historical interactions, performs well for small groups when evaluated with metrics like macro-accuracy, and maintains lower latency at higher accuracy with controllable additional time costs for extensions.

P-GenRM aims to provide answers that are "more suitable for this person, in this scenario" in open-ended questions, moving beyond superficial customization. Its core value lies in transforming personalized alignment into an interpretable and generalizable process. This involves inferring personas from preference signals, generating weighted scoring rubrics, evaluating responses item by item, and performing multiple inferences for uncertainty reduction. For new users, it leverages similar user experiences at the prototype level to enhance cold-start generalization.

This mechanism enhances the applicability of personalized technology in real-world scenarios such as customer service, education, and content creation. The system decomposes, absorbs, and reduces uncertainty through a structured evaluation chain. P-GenRM demonstrates the potential of generative reward models in personalized alignment by converting diverse preference signals into structured, interpretable scoring chains and improving stability and generalization through test-time user extensions. This development contributes to more accurate personalized reward signals and advances AI systems that can "understand you and explain why they do what they do."