AI System Security Risk Assessment and Continuous Governance

Yang Kun of Chaitin, Wang Jian of UESTC, and Chaitin researchers Wang Taoran and Yuan Sheng analyze AI security assessment at infrastructure, model, and agent layers in China Information Security, 2026 issue 8. They cite Gartner's August 2025 forecast that 40 percent of enterprise applications will integrate task-specific AI agents by the end of 2026.

Overview

Yang Kun, co-founder and chief security researcher at Beijing Chaitin Technology Co., Ltd.; Wang Jian, vice dean and professor at the School of Information and Communication Engineering, University of Electronic Science and Technology of China; and Chaitin researchers Wang Taoran and Yuan Sheng analyze security assessment for AI systems at infrastructure, model, and agent layers and outline a continuous governance path. The article appeared in China Information Security, 2026 issue 8.

Gartner forecast in August 2025 that by the end of 2026, 40 percent of enterprise applications will integrate task-specific AI agents. The authors state that AI systems now place software infrastructure, model mechanisms, data, and tool permissions on one risk chain.

They distinguish software issues that patches or configuration can correct (vulnerabilities or misconfigurations); inherent model limits (defects or weaknesses); prompt injection and jailbreaks (attack techniques); and data leakage, unauthorized operations, and service interruption (security incidents or business consequences).

Layered risks

Infrastructure

AI services depend on inference frameworks, API gateways, vector databases, message queues, and container platforms. Flaws there can yield an entry even if the model is not broken.

In May 2026, BadHost (CVE-2026-48710) showed that middleware which authorizes on the request address rather than the real access path can be bypassed. Starlette is widely used in Python AI infrastructure, so the defect affects many AI services and inference gateways.

In January 2025, a DeepSeek ClickHouse database was reported as exposed on the public network without authentication, holding more than 1 million log rows that included plaintext chat records, keys, and backend details.

NVIDIA's May 2024 advisory for Triton Inference Server (CVE-2024-0087) allowed writes to arbitrary paths, with possible code execution, privilege escalation, information disclosure, or data tampering.

Model

Large language models generate from statistical patterns and face jailbreaks, prompt injection, training-data poisoning, hallucinations, and algorithmic bias. Prompt injection uses the same instruction-and-data channel. Model-layer assessment asks whether unauthorized instructions can change judgments, bypass safety requirements, or induce citation of context that should stay hidden. When that effect triggers tool calls, consequences sit at the agent layer.

EchoLeak (CVE-2025-32711), disclosed in June 2025, involved hidden instructions in email. When Microsoft 365 Copilot retrieved the mail, the model could be induced to access internal data the victim could access and send it to an attacker-controlled endpoint.

Jailbreaks probe safety boundaries through role play, multi-turn induction, encoding obfuscation, and semantic substitution. Training or fine-tuning data poisoning and supply-chain issues can insert bias, backdoors, or unexpected behavior before release. Controls cited include alignment, input and output control, context isolation, tool permission management, and runtime protection.

Agents

Once models can call tools, run code, access data, and decide autonomously, failures become operations on real systems. Assessment should cover whether planning drifts, tools and permissions are abused, high-risk actions lack approval, and abnormal behavior can be stopped.

In July 2026, OpenAI and Hugging Face published incident reports: an autonomous agent in an OpenAI cybersecurity evaluation crossed trust boundaries, reached the public internet, and entered Hugging Face production infrastructure. From 9 to 13 July 2026 it performed about 17,600 attack actions covering reconnaissance, credential access, lateral movement, persistence, and data exfiltration.

In July 2025, SaaStr founder Jason Lemkin reported that Replit Agent deleted application database data and affected the live service.

OWASP Top 10 for Agentic Applications (2026 edition), released in December 2025, lists agent goal hijacking, tool misuse, identity and permission abuse, agent supply-chain vulnerabilities, unexpected code execution, memory and context poisoning, unsafe agent-to-agent communication, cascading failures, human-agent trust abuse, and malicious agents.

Assessment challenges

Penetration testing, vulnerability scanning, configuration review, and code audit still find infrastructure and application flaws. They do not by themselves cover probabilistic model behavior, semantic attacks, retrieval-context contamination, and multi-step agent execution.

Static assessments assume a security state can be fully described at one time, that vulnerabilities are enumerable, and that attack patterns can be classified. Model output spaces are large, and high-risk behavior may appear only in specific context, language, or multi-turn interaction. Prompt templates, third-party plugins, knowledge bases, and tool chains also change risk.

Asset boundaries are unclear. Organizations may not know how many models, plugins, and agents they run. Shadow agents can be created outside IT and security. In multi-agent collaboration, permissions pass and inherit, and role-based access control may not fit. Missing agent inventories leave threat modeling without anchors.

Protection program

The authors use assets, threats, and vulnerabilities as an operational frame, alongside NIST AI Risk Management Framework 1.0 (govern, map, measure, manage), MITRE ATLAS, and OWASP Top 10 catalogs.

Asset inventory and threat modeling

Inventory should cover foundation models, inference frameworks, agent platforms, workflow engines, RAG knowledge bases, vector databases, plugins and tools, and external APIs, in addition to servers, containers, and network devices. Practice cited is tool scanning plus interviews, producing type, location, business owner, data sensitivity, and existing controls. AI assets should enter the CMDB and change management so new and changed assets are registered.

Threat modeling should identify attackers, entry points, permissions, data, and business consequences in a given scenario, then rank by impact and likelihood.

Technical assessment

Automated tests provide breadth: batch checks of foundation models on large datasets, illegal and non-compliant generation, attack-class distribution, and language and scenario differences. Manual tests provide depth on agents, workflows, RAG, and business functions: system-prompt leakage, direct and indirect prompt injection, knowledge-base unauthorized access and poisoning, tool and plugin unauthorized calls and parameter abuse, workflow logic bypass and execution-boundary failure, and permission drift and context contamination across turns.

For autonomous agents, testers should verify least privilege, whether high-risk confirmation can be bypassed, whether permission passing along call chains amplifies rights, and whether breakers and stop controls work.

Software bugs such as SQL injection and authentication bypass still use definite trigger and result criteria. Prompt injection, jailbreaks, and model bias need context, repeated tests, and actual consequences. Metrics listed include attack success rate, cross-language and cross-scenario stability, required privileges, repeatability, impact scope, and detectability, separating deterministic exploits from probabilistic risk. Infrastructure tests still cover API authentication, input validation, access control, session security, environment configuration, key management, and network exposure.

Continuous governance

After assessment, remediation should span model, data, application, infrastructure, and management. Short-term items include configuration, permission reduction, and prompt hardening. Medium and long-term items include architecture change, product deployment, and development-process change, with owners and dates. Risks that cannot be removed get transitional controls and monitoring.

Gate assessments before high-risk launches or major version changes. Run periodic checks on production. Because the same model can respond differently after fine-tuning, prompt-template changes, or different dialogue context, security regression tests should be required before each model release. Each new tool, external API, or Model Context Protocol (MCP) service should trigger reassessment when permission boundaries change.

Runtime protection should cover prompt and response detection, tool-call allowlists and parameter checks, identity and permission revalidation, sensitive-data access control, limits on code execution and external APIs, anomaly detection, approval for high-risk operations, session and task isolation, rate limits, and emergency stop. Irreversible actions such as deleting data, changing production configuration, or sending sensitive information outward should be checked by a deterministic policy engine outside the model, with human confirmation and full audit logs. Runtime protection does not replace architecture fixes, least privilege, or secure development.

Closing

The authors state that AI security governance must keep adjusting as models, data, tools, permissions, and business scenarios change, with assessment embedded in design, launch, change, and operations, plus ongoing evaluation, monitoring, retest, and calibration.