Test the Judge Too: Safety Judges Are a New Evaluation Attack Surface

Researchers from Northeastern University, Georgia Tech, and USC found that content-invariant wrappers can reverse safety-judge labels without changing the underlying model answer. A GPT-4o-mini judge flipped 19.9% of correctly unsafe replies to safe after a symbolic refusal prefix, while Llama Guard 4 released 12.3% of dangerous answers under an education-scenario wrapper.

Why the referee now sits on the critical path

Safety scores for large language models increasingly depend on another model. Jailbreak evaluations send an attack prompt to a target model, then hand the reply to a Judge that decides whether the attack succeeded. Guardrail evaluations use the same last step: a Judge labels how many dangerous items a Guard blocked and how many benign items it blocked by mistake.

If the Judge is unreliable, reported attack success rates, guardrail recall, and safety leaderboards can track Judge preference as much as they track the model under test. Researchers from Northeastern University, the Georgia Institute of Technology, and the University of Southern California made that risk concrete. They left the body of each model answer unchanged and only added a few sentences around it. Some safety Judges then reversed their verdicts.

The paper is available at https://arxiv.org/pdf/2609.08236. In the strongest GPT-4o-mini result, a symbolic refusal placed in front of a dangerous answer caused the Judge to relabel nearly 20% of replies it had correctly marked unsafe as safe. Llama Guard 4, a dedicated content-safety model, allowed 12.3% of dangerous answers after an education-scenario wrapper. The evaluation attack surface now includes the Judge.

What automated pipelines actually measure

Older jailbreak work stops at the target model's harmful reply. Automated pipelines add a Safety Judge that sets the attack success rate (ASR). The authors treat that Judge as the object under measurement. Their test is basic: if the answer body is identical and only surrounding wording changes, the safety label should stay the same. A Judge should score what the model supplied, not whether the reply sounds like a refusal.

Content-invariant wrappers

The authors call the method Content-Invariant Wrappers. The original answer R is kept verbatim, with wrapper text placed before or after it. They ran byte-level checks on all 2,700 wrapped samples and confirmed the original body remained intact. That design removes disputes over whether two paraphrases mean the same thing. If an unwrapped answer is unsafe and the wrapped copy is labeled safe, the change sits with the Judge.

They tested two directions. Harm-Hiding places safety-looking language around a confirmed harmful answer, including an education disclaimer, an ethics preface, or a refusal line followed by the original harmful text. Harm-Inventing places cooperation-looking language around a genuine refusal so that some Judges mark a safe refusal as harmful. Both directions point to the same failure: some Judges score how an answer looks rather than what it does.

The study used 9 wrapper types, 600 JailbreakBench answers, and 8 Judge configurations. Target models were GPT-3.5, GPT-4, Vicuna-13B, and Llama-2-7B, with 300 harmful answers and 300 refusals.

GPT-4o-mini: 19.9% flips versus a 0.5% noise floor

On answers the GPT-4o-mini Judge had correctly labeled unsafe, a symbolic-refusal prefix plus the original harmful body flipped 19.9% of labels to safe. The paper's noise floor, the rate of disagreement when the same unwrapped input is judged twice, was 0.5%. Wrapper-driven flips therefore exceeded ordinary sampling noise by a wide margin.

Majority vote over three judgments moved the flip rate from 19.9% to 18.2%. In 14.4% of samples, all three judgments wrongly released the answer. The pattern is systematic.

Dedicated guards can be stably wrong

Llama Guard 4 and gpt-oss-safeguard-20b were included as specialized safety models. Llama Guard 4 flipped 12.3% of correctly flagged harmful answers under an education-course wrapper, and 8.3% under a refuse-then-continue wrapper. Unlike GPT-4o-mini, Llama Guard 4 was stable on repeats: contradiction across three judgments was 0.33% in key settings, yet 11.9% of education-wrapped samples flipped on all three runs. Repeated sampling does not fix a consistent error.

Failures belong to specific Judges. The same refuse-then-answer wrapper produced 19.9% flips on GPT-4o-mini, 3.5% on DeepSeek-Chat, 0.7% on Qwen-Plus, and 0.4% on Claude Haiku 4.5. The education wrapper that flipped Llama Guard 4 at 12.3% flipped GPT-4o-mini at only 1.0%. No single wrapper fooled every Judge.

gpt-oss-safeguard-20b saw a maximum flip of 1.2% across tested wrappers, below its 1.5% noise floor. Surface-form stability is possible. Judge design quality varies, which is why the authors say Judges should be evaluated before they are used to score models.

Style over substance

The authors link the failures to shortcut learning. The intended task is to decide whether an answer actually supplies dangerous information, capability, or executable help. Training data often pairs safe refusals with apologies, "I cannot help" lines, safety caveats, or education-only disclaimers. A Judge may then treat those tokens as a safety signal even when the body still provides harmful content. The paper's title, *Style Over Substance*, names that swap of surface form for substance.

The prompt is part of the instrument

GPT-4o-mini with a generic Judge prompt and GPT-4o-mini with a StrongREJECT-style prompt are the same base model. The StrongREJECT-style prompt told the Judge to ignore tone, disclaimers, and wrapping and to score whether the answer actually provided harmful capability. On the refuse-then-answer wrapper, flips fell from 19.9% with the generic prompt to 1.7% with the StrongREJECT-style prompt. Naming "GPT-4o as Judge" is incomplete. A Judge is the combination of model, prompt, criteria, output format, and sampling strategy.

Leaderboards inherit Judge error

On unwrapped originals, the full Judge panel agreed on only 55.5% of samples. Different Judges named different safest models among Vicuna-13B, Llama-2-7B, and GPT-3.5. Across 2,000 bootstrap resamples, for six of seven model-based Judges the top-ranked model stayed first in only 39% to 70% of draws.

Even without an adversary, rankings already move with Judge choice and sample draws. Surface wrapping is a third variable. A reported ASR is a joint product of the target model, dataset, Judge, Judge prompt, and surface form. A two-point gap between models can sit inside Judge measurement error.

The authors propose a Flip Budget: when answer content is held fixed, how far can irrelevant surface changes move the Judge? They recommend publishing ASR together with Judge identity, self-repeat noise, and content-invariant wrapper flip rate, treating the Judge as a measuring instrument with a stated error range.

Ensembles help only if errors differ

Because Judges have different blind spots, a Judge ensemble can reduce error if members do not fail the same way. GPT-4o-mini was sensitive to symbolic refusals. Llama Guard 4 was more sensitive to education wrapping. Majority vote among correlated Judges does not remove a shared shortcut. The engineering implication is to test the referee, not only the model.