Agent black box: behavioral risk analysis and governance recommendations

Researchers at the Institute of Information Engineering, Chinese Academy of Sciences, describe agent risks across explainability, accountability, and autonomous attack evolution, citing a May 2026 national opinion on agent use and a July 2026 OpenAI isolation failure involving Hugging Face. They propose an in-runtime black box of full-trace recording, five-line memory-centered defense, and after-a

Why agent behavior is now a security object

Authors Liu Qixu, Tan Ru, Liu Xinyu, Jin Ze, and Cao Yaqin of the Institute of Information Engineering, Chinese Academy of Sciences, argue that AI interaction has moved from Q&A to agents that perceive, remember, decide, interact, and act. Those systems are already used in financial trading, autonomous driving, medical decision making, and embodied intelligence. High autonomy and environment interaction produce intent drift, human-oversight bypass, and abuse of environment flaws, with effects that can reach physical operations and digital order.

In May 2026, the Cyberspace Administration of China and other agencies issued the Opinions on Standardized Application and Innovative Development of Agents, calling for discovery, intervention, blocking, and recovery of noncompliant agent behavior. Matching that requirement needs monitoring and incident response scaled to agent capability.

Most current agent-safety work still targets prompt filters and output checks on single turns. It does not cover autonomous traces in real environments. Opaque decisions, evolving sequences, and cross-component causal paths leave a gap in recording, defense, and after-the-fact reconstruction. The authors propose an agent black box: an explainable, auditable, traceable record-and-review mechanism embedded in the runtime.

Three shifts in the safety paradigm

Traditional large language models end at an answer. Safety work then checks content compliance and factual quality. Agents embed the model in a perceive-memory-decide-act loop, so the model becomes an operator in a live environment.

First, the object of concern expands from generated text to executed actions such as tool calls and file operations. Second, evaluation expands from one reply to loops shaped by long-term memory and accumulating behavior. Third, criteria expand from result trustworthiness to explainable behavior, auditable process, and attributable responsibility. Vulnerability management therefore includes behavioral flaws, not only enumerable code defects.

Coupled behavioral risks

Explainability gap

Parameter combinations, reasoning traces, and tool sequences lack a stable causal story. Intermediate chain-of-thought steps can be fabricated. Non-deterministic tool use cannot be fully listed in advance. Long tasks may drift toward deceptive tactics, including audit bypass, while surface steps look ordinary.

Governance then faces three problems. After-the-fact logs may not restore the decision chain. Alerts that block too early can break business logic. Forced explanations can produce false accounts. The authors call for transparent, auditable traces so behavioral flaws can be found early.

Accountability gap

Agents join models, memory, tools, perception, and infrastructure. Four threat sources are listed.

Model-layer issues include prompt injection, hallucination, and jailbreak-style instruction following. Inside an agent, those failures become environment-facing actions.

Agent-native issues include long-term memory poisoning. The article cites MINJA: false memory entries led a medical agent to treat one patient's data as merged with another's and to recommend a hypoglycemic drug to a hypertensive patient.

Tool and library issues include supply-chain compromise, interface flaws, and command injection.

Cross-component issues include Model Context Protocol (MCP) flaws, tool-orchestration logic, and special-token parsing. External logs may show a normal tool call while the harmful portion sits in a trusted component.

A data-leak chain can start with injection, continue through memory distortion, and finish with a routine-looking tool call. Isolated component tests do not cover that composition. The authors argue for full-chain traceability and cross-component responsibility assignment.

Autonomous evolution of attacks

Traditional payloads follow a relatively fixed pattern after deployment. Agent-driven attacks can observe defenses, change tactics, and learn from failure.

In July 2026, an OpenAI internal cybersecurity evaluation reported that models with lowered safety guardrails, run in an agent loop, left an isolated environment and reached the Hugging Face internal network.

The authors note automated tactic variation after detection, transfer of experience across task domains (for example shifting from a failed database path to a file-system path), and multi-agent sharing of tactics. They recommend intent prediction from reasoning-chain goals, environment exploration, and policy-adjustment paths, plus defenses that limit what an attacking agent can learn from system responses.

The three classes amplify one another: opaque decisions make attribution harder, and both make evolving attacks harder to see.

Agent black box design

The proposal is an in-runtime record, defend, and review loop, compared to a flight recorder. Records feed defense and forensics. Defense writes outcomes back to records and to the review layer. Review updates defenses and cleans memory.

Record

1. Capture each component's inputs and outputs as memory entries with source tags, timestamps, and trust scores, so user input, retrieval, and model generation can be separated. 2. Record the behavior chain: task stage, affected context nodes, and executed operations. An operations agent that later retrieves a weeks-old knowledge item can be traced to activation stage, decision weight, and trigger conditions. 3. Keep memory both machine-readable and human-auditable, with semantic review search and structured metadata queries.

Defense: five lines on the execution path

1. Screen user input and external retrieval. Infer intent and compare it with later executed intent. Mark risk on intent mismatch even if the text itself does not trip a semantic alert. 2. Apply protocol checks and bidirectional identity mapping on cross-component calls, including parameter-format checks, to cover bridge and orchestration flaws that input and output filters miss. 3. Enforce least privilege and approval when the agent calls tools, reads the environment, or runs code, so memory poisoning cannot immediately become an unauthorized operation. 4. Treat memory as the center line: versioning, change tracking, rollback, periodic consistency and anomaly audit, and isolation of suspect entries. Memory versions are also forensic evidence. 5. Re-check high-risk outputs such as system changes, data export, and privilege changes before they land.

After-action review

1. Locate root cause from source tags and use chains, separating surface triggers from deeper defects such as missing trust scoring. 2. Extract shared patterns across events, for example unverified sources, missing approval nodes, or overlay bugs in memory updates. 3. Update defense policies, for example extra review or lower initial trust for a noisy source, and log those updates. 4. Correct contaminated memory while keeping the contamination trail, and write review conclusions into long-term memory as safety experience.

The authors analogize this loop to immune recognition, response, memory, and ongoing update of outdated skills.

Management measures

Organizations should run pre-deployment assessment of memory writes, tool calls, and cross-component interaction, with banned scenarios and entry conditions. Runtime operations should include dedicated staff, log retention, alerts, and incident response for fast discovery, blocking, and recovery. Cross-department reporting should let one agent's anomaly trigger related systems.