Shanghai AI Lab Moves Agent Safety from Prompt Rules to Harness-Policy Evolution
Shanghai Artificial Intelligence Laboratory, with Fudan University, Shanghai Jiao Tong University, the Hong Kong University of Science and Technology, and Zhejiang University, proposed SHE and SafeEvolve to evolve agent safety from full execution traces. SHE reduced average attack success on Agent-SafetyBench from 17.1% to 5.5%, and SafeEvolve cut AgentDojo attack success on Qwen3.5-4B from 2.37%
Why final-reply checks fall short
Once a large model agent enters browsers, email, databases, and business systems, a check that only looks at whether the final reply contains harmful text leaves gaps. Agents plan steps, read external information, and call tools. Risk can appear at any point in execution.
Hidden instructions in a page can change the task goal. Email or file content can be treated as a user command. Over-broad tool permissions can produce unauthorized calls. Faulty memory and stale plans can pull multi-turn runs away from the original authorization. These failures are usually not the product of a single model output. They stack across the policy model, harness, tools, memory, and environment state.
Fixes are also hard to isolate. Rewriting the whole prompt makes it unclear which change worked. Adding more refusal rules can cut legitimate task completion. Updating model weights alone does not put new risks into runtime control. Turning problems seen during execution into reusable, checkable safety experience is a basic agent-safety question.
Shanghai Artificial Intelligence Laboratory, with Fudan University, Shanghai Jiao Tong University, the Hong Kong University of Science and Technology, and Zhejiang University, proposed SHE and SafeEvolve around that question. Both start from full execution traces. SHE studies how a safety harness can evolve from failures. SafeEvolve studies how verified experience can be absorbed by the policy model.
SHE: updating the safety harness from traces
The harness organizes context, manages memory, schedules tools, and constrains permissions. Existing safety controls are often written into the system prompt, a rule store, or tool policy before deploy, then left fixed. When task flow, the external environment, or attack methods change, those bounds can fail.
SHE splits the harness into four safety components rather than one block of config. The system prompt holds overall principles and task bounds. The rule bank stores structured, reusable rules. Safety memory keeps recurring failures that are not yet stable enough to become rules. Tool policy sets conditions and scope for tool calls. The split allows local edits, checks, and rollbacks.
Each evolution round starts from a full trace: user request, intermediate context, model responses, tool calls, environment feedback, and task result. The system locates the step where risk appeared, why current constraints failed, and which component should take the fix. If the agent treats page or email content as a user command, updates go to instruction priority and untrusted-content rules. If it calls a sensitive tool unrelated to the task, tool policy is the main edit. Cases that are not yet stable rules can go into safety memory first.
Each candidate version also goes through an effectiveness check and a safety-utility check, to block apparent gains that come only from wider refusals or weaker tools. One paper example: the user asked only for an introduction to an app, but the agent tried a device-level install. SHE did not ban app-related tasks. It kept introduce, compare, and official-download behavior, required explicit authorization before install, and barred describing unfinished operations as complete.
Experiments evolved the harness on 15 Agent-SafetyBench tasks and evaluated on the remaining 185, covering context poisoning, indirect injection, tool tampering, memory injection, and combined attacks. Against a static SafeHarness, SHE cut average attack success from 17.1% to 5.5% and raised task availability under attack from 31.6% to 47.6%. On AgentHarm, which was not used in evolution, the harm score fell from 19.8% to 9.8%. Evolved bounds also transferred to other agent models without repeating the full process per model.
SHE builds a path from trace diagnosis to harness updates. External constraints can limit behavior without the policy itself holding the same bounds.
SafeEvolve: writing verified experience into the policy
If safety knowledge lives only in the harness, the model can still lean on outer interception. On risk variants outside the rules, or on longer tool-call chains, early constraints can fade. SafeEvolve puts policy optimization into the evolution loop so verified harness experience becomes more stable model decisions.
On the harness side, SafeEvolve extracts a safety prompt and a layered SkillBank from traces. SkillBank stores general safety principles and task- or tool-specific operating experience, such as spotting untrusted instructions on a page, checking attachment permissions before sending mail, and asking the user when authorization is unclear. At inference time, the system retrieves skills for the current task instead of writing every rule into context at once.
Policy training has two stages. Harness-use SFT collects traces under the evolved harness. Only samples that pass both safety and task-utility checks enter training, so the model learns when to call a safety skill, how to separate user goals from environment prompts, and how to continue a valid task after blocking a dangerous operation.
Harness-augmented RL then runs multi-step exploration with the safety prompt and dynamically retrieved skills. A verifier scores task completion, dangerous behavior, and tool-call validity. Compared with SFT, RL looks at the whole trace and further constrains runs that follow rules early and drift later. Normal tasks should continue. Clear harmful or unauthorized requests should be refused. Malicious instructions from the environment should be ignored while the original user task continues. When authorization is unclear, confirm first, then act.
On Qwen3.5-4B, SafeEvolve cut AgentDojo attack success from 2.37% to 0.79% and raised clean-task utility from 59.79% to 61.86%. On AgentHarm, the harm score fell from 56.45 to 12.27 and the refusal rate rose from 28.98% to 83.83%. Qwen3-4B showed the same direction: AgentDojo attack success from 13.38% to 2.42%, clean-task utility from 44.33% to 60.82%, and under-attack task utility from 35.91% to 52.05%.
Three shifts in the two papers
First, from static safety config to trace-driven updates. Safety experience does not stay in after-the-fact logs. After diagnosis and checks, it enters the next harness round.
Second, from one-block rules to component-level responsibility. System prompt, rule bank, safety memory, and tool policy take different jobs, so a problem can be located and an edit can be checked and rolled back on its own.
Third, from harness-only updates to harness and policy co-evolution. The harness can record new risks quickly. The policy absorbs that experience through SFT and RL. The updated policy then produces new traces that expose gaps still missing from the harness.
The path does not mean agent safety is solved automatically. Trace diagnosis and verifier scores can still err. Gains on benchmarks do not mean production runs cannot be bypassed. The two papers move safety from a one-time pre-deploy config toward a system capability that can be revised from execution experience.
Papers and authors
SHE: Trajectory-driven Safety Harness Evolution for LLM Agents. Paper: https://arxiv.org/abs/2608.09885
SafeEvolve: Harness-Policy Co-Evolution from Agent Experience for Safety Alignment. Paper: https://arxiv.org/abs/2609.02786
Qu Wanying, Fudan University doctoral student, agent safety alignment and self-evolution.
Mao Qinghua, Shanghai Jiao Tong University doctoral student, trustworthy graph neural networks, large model safety alignment, and agent safety.
Liu Dongrui, head of the agent attack-defense systems center at Shanghai AI Lab, recipient of the WAIC Yunfan Award and Shanghai high-level talent status. The center led development of the agent guard model AgentDoG and formed the DoGNAVY team. Credits include a CVPR 2024 best paper candidate award and an ACL 2025 outstanding paper award.