Better Context, Better Verifiers.
Prompt-Aware Rubrics (PAR) is a verifier (reward) framework for video generation, where success is often difficult to verify with fixed rules. It draws inspiration from the workflow of human evaluation experts: define the criteria, translate them into task-specific checks, and train evaluators to apply those checks consistently.
PAR Components. PAR implements this workflow as a Rubric Handbook, a Rubric Generator, and a trained Rubric Verifier. The handbook and generator construct better context for the verifier, turning open-ended evaluation into explicit, task-grounded questions. This gives PAR a path to scale with the answering model: as the model’s underlying capabilities improve, targeted rubrics help elicit those capabilities.
PAR Evolution. PAR provides three versions: Single PAR → Agentic PAR V1 → Agentic PAR V4. All three share the same Rubric Handbook and Rubric Generator. The progression expands the Rubric Verifier from a single answering model into a reward system that combines multiple specialist tools and a workflow harness.
Single PAR uses one trained answering model. Agentic PAR V1 supplements that model with specialist tools through fixed routing. Agentic PAR V4 binds individual rubrics to multiple tools and organizes their evidence through a workflow harness for rubric-level judgment.
Across the three versions, the task-specific checks provide a shared interface, while the answering process gains additional capabilities and more structured evidence handling. Single PAR and V1 were deployed; V4 is exploratory.
Future Direction. The longer-term direction is a hybrid verifier that combines System 1 for fast, direct decisions with System 2 for more elaborate chain-of-thought reasoning and tool-assisted inspection on selected rubric categories.
For checks already within the base Rubric Verifier’s capabilities, the Rubric Handbook and Rubric Generator can elicit those capabilities through better context, while training aligns its decisions with evaluation experts. These checks can use System 1 for fast verification. Harder tasks that remain beyond reliable direct judgment—such as recognizing a tiny malformed region—need a richer answering process. Here, System 2 can combine CoT reasoning with crop and zoom tools, and inspect neighboring frames when needed, before producing a verdict.
Motivation: building better context from human evaluation
Same model, different context
Consider an instruction with several editing requirements:
Change every red delivery cart in the courtyard to blue, add two trees near the gate, and preserve the camera motion, pacing, and duration.
A fixed, generic evaluation prompt might ask, “Does the edited video satisfy the user’s instruction?” The model must parse every requirement, decide where to look, compare the relevant regions, and compress its findings into one judgment. It can overlook the red carts while attending to the overall scene.
Now ask, “Have all red delivery carts in the courtyard been changed to blue?” The video is unchanged. The verifier is unchanged. But the context explicitly identifies an object, an attribute, and a target change. A failure under the general question does not necessarily establish an inability to recognize color; the relevant capability may not have been effectively elicited.
This suggested a practical research question: how can we systematically construct context that helps a verifier use the capabilities it already has?
Borrowing the workflow of human evaluation experts
Human evaluation begins before anyone assigns a score. Experts first decide what matters, distinguish different failure modes, and document the criteria. Evaluators are then trained to apply those criteria consistently. Asking a model to “judge the video” skips much of that work.
PAR has three components. The Rubric Handbook stores expert knowledge as the generator’s system prompt. The Rubric Generator converts a user instruction into concrete checks. The Rubric Verifier answers those checks using the video evidence and an expert-aligned decision rule.
What a prompt-aware rubric looks like
The rubric pool has two levels: category and subcategory. A generated question grounds a selected criterion in the actual task. The following fictional examples demonstrate that structure without reproducing an internal rubric inventory.
Change every red delivery cart in the courtyard to blue, add two trees near the gate, and preserve the camera motion, pacing, and duration.
| Category | Subcategory | Task-specific rubric |
|---|---|---|
| Visual Editing | Attribute Changes | Have all red delivery carts in the courtyard been changed to blue? |
| Visual Editing | Element Addition | Have two trees been added near the gate? |
| Content Preservation | Background and Props | Are the original gate and other unedited courtyard elements preserved? |
| Content Preservation | Camera and Motion Continuity | Is the original camera motion preserved? |
| Editing Quality | Edit Integration | Is the cart recoloring temporally stable, without residual red areas or deformation? |
| Editing Quality | Edit Integration | Do the added trees blend into the courtyard without visible seams? |
The five top-level categories—Visual Editing, Audio Editing, Cross-Modal Editing, Content Preservation, and Editing Quality—were distilled from practical experience with video-editing tasks and expert evaluation in the production setting. They capture recurring editing requirements and failure modes. A visual-only task selects relevant checks; it does not need questions from every category.
At inference time, the workflow is handbook + instruction → rubrics → video verification → rubric-level verdicts and an aggregate score. The per-check probability of “1” can supply a soft score for aggregation. The same representation supports interpretable evaluation, data filtering, and reward construction for post-training.
PAR: a single verifier
The handbook is the generator’s system prompt
The handbook is implemented as text in the rubric generator’s system prompt. It has two parts: fixed generation requirements and a structured rubric pool.
The fixed requirements favor atomic checks, prioritize important requirements within a question budget, enforce consistent polarity, and discourage contradictions and duplicates. A positive answer should always mean that the requirement is satisfied; a negative answer means it is unmet, incomplete, or unsupported by sufficient evidence.
The rubric pool supplies the two-level categories and criteria from which the generator selects. It spans broad categories and finer criteria, with coverage expanded through iteration. Given a user prompt or editing instruction, the generator selects relevant entries and grounds them in specific objects, attributes, actions, and preservation constraints.
We built it in three stages. First, expert experience established the initial rules and pool. Second, inference failures and RL feedback exposed task requirements and failure modes that were not yet covered. The purpose of that feedback stage was to expand the pool’s coverage of real data and training scenarios. Third, an automated search proposed additions, removals, splits, and merges.
The search loop evaluated the initial handbook, selected a current or historical anchor, and assigned an edit budget. It inspected incorrect or tied cases, proposed changes to pool entries, checked that the patch obeyed the allowed scope and format, regenerated rubrics, and evaluated the candidate using the verifier. Only an improvement passing the generation-success gate replaced the current best.
Handbook refinement changed both what the generator could ask and how precisely it could ask it. Compared with the earlier version, three changes were especially concrete:
Three changes to the Rubric Handbook
- A larger question budget: 12 → 14 rubrics per case. The higher maximum gave the generator more room to cover important requirements with separate, atomic checks.
- Finer criteria for attributes, actions, text, and audio. Broad requirements were broken into more specific checks—for example, facial features, clothing, props, action order and pacing, text layout, and individual audio requirements—so distinct errors were less likely to be bundled into one question.
- A new rubric: [Editing Quality][Overall Visual Quality]. This added an explicit check on the overall visual appearance of the edited result, alongside checks on the requested edits and local defects.
Bench1 results from the handbook iteration study were:
A strong handbook reduces dependence on the generator
Before training a better answering model, we checked whether the generated questions were useful. Human review of 1,000 samples and 9,728 rubrics found approximately 94% of generated rubrics fully relevant to the task. Relevance varied across categories, with audio editing remaining a weaker area.
We then compared a strong external generator with a lightweight internal generator using the same detailed handbook. In that experiment, both yielded the same downstream Bench1 accuracy.
| Rubric generator | Bench1 accuracy |
|---|---|
| Strong external model | 77.1% |
| Lightweight internal model | 77.1% |
Approximately 80% of samples had more than 60% rubric overlap; approximately 85% of same-category questions were semantically consistent. The 94% relevance result comes from the reviewed rubric set, rather than a separate measurement for each generator.
| Wording A | Wording B |
|---|---|
| Have all red delivery carts in the courtyard been changed to blue? | Are all delivery carts that were red in the original courtyard now blue? |
An illustrative semantic pair, not verbatim outputs from the generator-comparison experiment.
A well-specified handbook can make PAR less sensitive to the strength of the rubric generator. In this comparison, the strong external and lightweight internal generators produced semantically similar checks and the same downstream accuracy. Refining the handbook is therefore a way to improve the generator’s output: encode task knowledge and coverage in its instructions, then let the model instantiate that guidance for each editing task.
Direct GRPO, expert labels, and shortcut learning
For a video pair with N rubrics, the verifier outputs an N-position binary string. Each position answers one check. We skipped an SFT stage because the training data was limited and the base model already knew the required format: given N checks, emit N binary answers. Direct GRPO could therefore focus on judgment quality rather than relearning the output schema.
The reward checks length, valid binary output, and per-rubric correctness. A wrong output length invalidates the sequence; illegal symbols are penalized. The accuracy-related term for rollout i and rubric j is:
Here, h is positive for a correct verdict and negative otherwise; p is the probability of the emitted token. The format-to-accuracy emphasis was 1:9, and rubrics were equally weighted in this verifier setup.
Data design mattered as much as the loss. The original training set contained approximately 11k samples and was dominated by mixed pass/fail sequences. Only about 5% were all-pass and 0.4% were all-fail. The verifier could learn a shortcut: a long list of rubrics probably contains a failure.
The first augmentation removed failed checks from selected samples, retained a subset of passing checks, and shuffled their order. The next version further balanced all-pass, mixed, and all-fail examples, while reducing correlations between rubric count and labels.
| Training variant | Approx. samples | Data change | Bench1 accuracy |
|---|---|---|---|
| Training baseline | 11k | Original distribution | 77.1% |
| Augmentation V1 | 15k | Increase all-pass examples | 80.7% |
| Augmentation V2 | 22k | All-pass / mixed / all-fail ≈ 30 / 50 / 20 | 81.4% |
| V2 + Gold-Rescue | No new source examples | Target all-incorrect rubric groups | 82.7% |
These are experiment variants rather than an isolated data-only ablation; V1 also disabled thinking. The 77.1% training baseline is the no-augmentation experiment, rather than an untrained model.
Gold-Rescue: a learning signal for all-incorrect groups
For each prompt, ordinary GRPO samples eight rollouts (K = 8) and computes group-relative advantages per rubric. A difficult rubric may be answered incorrectly in all K. Our Gold-Rescue trigger operates on that rubric: when every sampled answer is wrong, inject one gold answer into the group and use it to provide a positive target.
For a binary correctness illustration, add one gold answer to K incorrect answers. With population-standard-deviation normalization, the incorrect answers receive advantage \(-1/\sqrt{K}\), and the gold answer receives \(\sqrt{K}\). This explains how rescue restores a learning signal to an otherwise uniform group.
Ordinary rollout tokens retain the PPO clipped loss. Writing the probability ratio as \(\rho_t=\pi_\theta(y_t\mid x,y_{<t})/\pi_{\mathrm{old}}(y_t\mid x,y_{<t})\), the per-token loss is:
The gold token is evaluated under a gold prefix. Following the policy-shaping function in LUFFY: Learning to Reason under Off-Policy Guidance, we use \(f(p)=p/(p+\gamma)\). With the gold teacher probability treated as 1, the corresponding loss term is:
For a positive gold advantage, minimizing this loss increases the probability of the correct token. The saturating transformation reshapes the off-policy update; it is not the usual negative-log-likelihood objective. Gold-Rescue adapts this ingredient to a targeted rubric-level trigger, rather than reproducing the complete LUFFY training procedure.
On Bench1, Gold-Rescue improved the V2 verifier from 81.4% to 82.7% (+1.3 percentage points), without collecting additional source training examples.
Generalization, diagnosis, and PAR for RL
On Bench3, a held-out editing evaluation, the largest improvements appeared in instruction following. Quality changed much less, a clue for the later failure analysis.
| Method | Overall | Instruction following | Consistency | Quality |
|---|---|---|---|---|
| Fixed-prompt baseline | 0.474 | 0.630 | 0.747 | 0.727 |
| Single PAR | 0.747 | 0.857 | 0.805 | 0.734 |
| External SOTA model | 0.630 | 0.662 | 0.792 | 0.792 |
Bench3 reports the original evaluation scores, on a 0–1 scale; these are separate from Bench1 accuracy. On the audio-focused Bench4, accuracy improves from 67.2% for the fixed-prompt baseline to 73.2% for trained PAR.
PAR for RL. PAR can also provide rewards for video-generation post-training. In policy-gradient-style methods where the update is weighted by a reward or advantage, more accurate verification can provide a more reliable learning signal for editing instructions. Better verifier accuracy can therefore translate into better downstream editing responsiveness, although the effect depends on the optimizer and training setup. Category-level rubric weights offer another control: increasing the weight on instruction following, preservation, or quality changes the emphasis of the reward and steers what the generator is optimized to improve.
The fictional dialogue example below illustrates how atomic checks make an error interpretable. It is newly constructed for this article and does not reproduce a project sample or a measured model response.
Change the first speaker’s line from “Meet me at noon” to “Meet me at dusk,” and the second speaker’s reply from “I’ll bring lunch” to “I’ll bring a lantern.” Keep the lip movements synchronized and preserve the unedited dialogue.
The new words are correct, but both speakers’ lip movements remain out of sync.
11001Example verdicts, not measured outputs| Category | Subcategory | Example rubric | Illustrative answer |
|---|---|---|---|
| Audio Editing | Dialogue Content | Does the first speaker now say “Meet me at dusk”? | 1 · Pass |
| Audio Editing | Dialogue Content | Does the reply now say “I’ll bring a lantern”? | 1 · Pass |
| Cross-Modal Editing | Lip Synchronization | Are the first speaker’s lip movements synchronized with the new speech? | 0 · Fail |
| Cross-Modal Editing | Lip Synchronization | Are the second speaker’s lip movements synchronized with the new reply? | 0 · Fail |
| Content Preservation | Audio Preservation | Is dialogue outside the edited exchange preserved? | 1 · Pass |
Agentic PAR: multi-tool integration and harness design
Deployment exposes a different kind of failure
A more informative question can elicit an existing capability. It cannot guarantee that the capability is strong enough. After deployment, new data exposed weaknesses in audio preservation and editing quality.
A representative failure pattern is a visual-only object replacement that also changes the soundtrack. A verifier focused on the visible instruction may accept the edit while overlooking the audio-preservation failure. This suggested a targeted intervention: add an audio-consistency specialist for tasks whose sound should remain unchanged.
From business failures to targeted specialist tools
In a real production setting, expert evaluation explanations tell us why an edit failed, while its task-specific rubrics tell us which requirement the answering model missed. Together, they make it possible to locate a concrete capability gap—for example, failing to detect changed background audio when the task only requested a visual edit.
A practical path to rapid deployment is to supplement that weak capability with an expert operator. The iteration loop runs the current verifier, analyzes business explanations and error cases, locates the affected rubrics, then adds a specialist and adjusts routing before evaluating the updated system. This expands a single answering model into a model supported by a pool of specialist tools.
Expanding Step 3: operator discovery and validation. We searched for suitable operators through local sandbox experiments. A focused validation set contained 50 abnormal and 50 normal cases. A strong model proposed and executed candidate operators in the sandbox; candidates were assessed on both failure-detection rate and false-rejection rate. Only operators that detected the targeted failures while preserving valid samples were selected for integration into the fixed routing workflow.
For audio consistency on visual_only tasks, the sandbox investigation identified complementary signal-level checks. The configuration used the following thresholds:
| Signal-level check | Threshold |
|---|---|
| Waveform correlation | ≥ 0.15 |
| Spectral cosine similarity | ≥ 0.65 |
| Mean coherence | ≥ 0.15 |
| Residual signal-to-noise ratio (SNR) | ≥ 0 dB |
These are configuration values for this audio-preservation setting. The check applies when the editing instruction requires the original audio to remain unchanged. A task that explicitly requests new dialogue or music needs checks matched to that request. The same discovery process can add a Lip-Sync Verifier for synchronization failures and a Visual Quality Specialist for visual defects.
Agentic PAR V1: specialists in a fixed workflow
Agentic PAR V1 put these capabilities into a fixed workflow: use applicable preservation prechecks, run PAR verification, and route relevant rubric types to specialists. The task matters: an audio-preservation check should not reject a sample merely because the user explicitly requested an audio change.
Agentic PAR V4: an evidence-based harness
V4 explored how to use a larger tool pool more effectively. Its harness has three stages. Rubric gap review checks for omissions and adds task-relevant synthetic rubrics. Tool orchestration binds checks to tools and collects evidence bundles. Evidence-based judgment makes a decision for each rubric, with an independent adjudicator when evidence conflicts.
Two advantages of the PAR architecture
1. Locate and supplement specific weaknesses. Business evaluation explanations, paired with task-specific rubrics, identify the requirements that the current answering model handles poorly. Specialists can then target those gaps, making the system easier to extend as new failure modes appear.
2. Build orchestration from explicit rubric-to-tool bindings. Each rubric can be bound in advance to multiple relevant tools. Those bindings and their associated rules define how evidence is collected and handled by the workflow harness. We did not first collect an expensive tool-calling training dataset or train a general autonomous tool-use policy. The system’s flexibility comes from extensible tools, task-specific checks, and evidence handling; it can be extended without requiring that tool-calling training stage.
Comparison with an external SOTA ensemble
Bench2 evaluates filtering on a real-distribution editing set. Here, positive means usable: precision measures the usable fraction among accepted samples, while recall measures how many usable samples are retained. This definition differs from the artifact-positive diagnostic experiment in Section 5.
| Method | Precision | Recall | Accuracy | F1 |
|---|---|---|---|---|
| Single PAR | 65.61% | 79.17% | 76.96% | 71.76% |
| Agentic PAR V1 | 69.85% | 74.40% | 78.20% | 72.06% |
| Agentic PAR V4 | 72.94% | 74.56% | 79.60% | 73.74% |
| External SOTA ensemble Gemini-3-Pro-Preview + GPT-5.6-Sol | 67.70% | 80.30% | — | — |
Agentic PAR V4 exceeds the Gemini-3-Pro-Preview + GPT-5.6-Sol external SOTA ensemble on precision: 72.94% versus 67.70%, a gain of 5.24 percentage points. Accuracy increases from 76.96% for Single PAR to 79.60% for V4 (+2.64 percentage points). The external ensemble retains higher recall, so the precision result reflects a filtering trade-off.
The error review makes the progression concrete. From Single PAR to V1, audio-consistency operators identified 35 inconsistent-audio cases, and quality specialists identified 24 low-quality cases. From V1 to V4, both the workflow harness and the expanded tool pool contributed: per-rubric judgment, rubric-gap review, conflict adjudication, local operators, and external VLM expertise each corrected decisions, while also introducing some errors.
Production impact: 2.20× usable + tolerable data
In a real data synthesis pipeline, deployed Agentic PAR V1 increased the share of usable + tolerable data from 31.0% to 68.06%. The final share comprises 60.92% strictly usable and 7.14% tolerable data, with the latter stacked above the strictly usable portion in the chart.
The combined outcome includes tolerable data. This production result alone does not quantify end-to-end cost savings.
Why PAR is a better choice
Better context activates atomic capabilities
The first source of improvement is explicit task context. A rubric identifies what to inspect: a target object, a color change, a preserved background, or a synchronization requirement. Instead of asking the verifier to discover every relevant test on its own, the generator supplies expert-informed hints.
The handbook study—63.6% → 71.2% → 73.2%—is consistent with this mechanism. It shows useful headroom in how the evaluation problem is specified. It does not imply that every remaining failure can be fixed by prompting.
Training aligns the decision rule with experts
The second source is alignment with the evaluation standard. Even after the right question is asked, models and experts may disagree about whether an edit is complete enough or whether a defect should cause rejection. Expert labels provide the decision boundary; balanced data discourages shortcuts, and rescue supplies supervision on difficult checks.
The training comparison reflects the full expert-aligned setup, including data design and optimization. Gold-Rescue further improves the V2 verifier from 81.4% to 82.7%. Together, the context and training results explain why PAR improves both the questions and the answers.
They also explain its ceiling. A well-posed question cannot make an invisible detail visible, and an expert-aligned decision rule cannot reliably use evidence the model fails to perceive. That is where the answering process itself needs to become more capable.
The future of PAR: a hybrid System 1 and System 2 verifier
The rubric is present, but the answer is wrong
Among 186 remaining failures analyzed, 15 involved missing rubrics, 162 had a relevant rubric but an incorrect tool/verifier judgment, and 9 involved evidence aggregation. In this subset, improving the question inventory alone would leave the main bottleneck untouched.
Quality breakdowns: local details and transient defects
Quality breakdowns illustrate a capability that direct binary verification can struggle to cover. Two patterns explain why. Spatially local defects can hide inside an otherwise acceptable frame: the overall scene looks plausible, while a small face, hand, prop, or object part is malformed. Transient defects can appear in one frame and disappear immediately afterward: recognizing the problem requires comparing how a region changes over time.
Both patterns call for a more deliberate answering process: locate the suspicious region, crop or zoom into it, inspect neighboring frames when necessary, and then decide whether the artifact is real. The following fictional examples separate these two patterns.
A diagnostic experiment with tool-assisted reasoning
We then compared artifact detection on 63 confirmed editing-artifact cases from Bench5. Direct binary verification detected 14.3%, the quality specialist 42.9%, and the basic artifact detector 41.3%. These baselines motivated two questions: is the model failing to recognize a small defect in the available evidence, or failing to find the frames that contain it?
At the same 5 FPS sampling rate, CoT with crop-based inspection helps the model examine the local evidence and recognize more defects. This supports strengthening the answering process after the rubric has already specified what to inspect.
Manually selected frames with CoT performed 3.2 percentage points below uniform 5 FPS. In this diagnostic setup, targeted frame selection added no benefit, while CoT with crop-based inspection improved detection. The results point to recognizing detailed local evidence as the main bottleneck, rather than finding the key frames: uniform 5 FPS was sufficient for this comparison. The useful next capability is closer inspection of what is already visible.
Let PAR define the checks; let the answering process adapt
We propose a hybrid answering system organized around what the base answering model can already do. For rubric categories already covered by its capabilities, expert-aligned training can teach a reliable decision rule; these checks can use System 1 for fast, direct binary verification.
For categories that remain difficult for the base model—such as recognizing a defect in a small local region—PAR can instead invoke System 2: more elaborate chain-of-thought reasoning combined with crop, zoom, or neighboring-frame inspection. The rubric specifies the task requirement; the answering path supplies the process needed to verify it. Temporal inconsistencies or conflicting evidence can likewise call for this more deliberate process.
Both paths can end in the same binary verdict. The distinction is the process used to obtain the evidence, not whether the final answer is 0 or 1. PAR supplies a stable interface for that process: a task-grounded question, the evidence needed to answer it, and a result that can feed evaluation, filtering, or reward construction.
Jev-style RLCD: making rubric scores less overconfident
From a binary verdict to a useful reward probability
Like a Jev-style verifier, PAR’s answering model can provide both a binary verdict and a probability that a rubric is satisfied. That probability matters when rubric scores are aggregated for filtering or used as rewards for RL. A model that assigns nearly 1.0 to uncertain or incorrect judgments can produce an inflated reward even when its output format and decision rule are correct.
We explored an RLCD-inspired calibration variant for answering-model training, using a Brier-based probability correction alongside the GRPO setup. The experiment used a Brier-inspired implementation rather than an official reproduction of Jev’s RLCD training recipe, which was not publicly available in the experiment record.
Brier-based correction reduces pressure toward extreme confidence
For a binary rubric label \(y\in\{0,1\}\) and a predicted satisfaction probability \(p\), the Brier loss is:
The experiment summarized its probability-correction signal as \(y-p\). This is the correction direction associated with the Brier loss. When a positive rubric is already assigned high probability, the remaining push upward becomes small. When a negative rubric is assigned high probability, the correction is strongly negative. The intent is to reduce overconfidence, particularly confident false positives.
The population interpretation is simple. If \(q=P(Y=1\mid x)\), then \(\mathbb E[y-p\mid x]=q-p\), with a stationary point at \(p=q\). This explains the calibration objective; it does not by itself guarantee convergence of the full neural-network training procedure.
Answering-model results and confidence shifts
In a separate calibration experiment on Bench2, the GRPO-RLCD variant improved precision and accuracy while keeping recall close to the standard-GRPO baseline:
| Metric | Standard GRPO | GRPO-RLCD | Change |
|---|---|---|---|
| Precision | 68.91% | 71.56% | +2.65 pp |
| Recall | 75.04% | 74.40% | −0.64 pp |
| F1 | 71.84% | 72.95% | +1.11 pp |
| Accuracy | 75.33% | 76.87% | +1.54 pp |
Rubric probabilities also became less extreme. The mean score decreased from 0.803 to 0.775, and the share of scores in [0.95, 1.00] decreased from 75% to 67%, with fewer inflated predictions near 0.999.
The practical motivation is to make rubric probabilities more useful as reward signals. This training direction complements the main architecture: expert alignment improves the decision rule, System 2 improves difficult inspection, and probability correction targets overconfident scores.