AgentPort-Bench Finds Attack Type Drives Agent Security More Than Model or Framework

A controlled study of 9,360 runs across seven agent frameworks found attack type explained about 28 percent of security-outcome variance. Model explained about 4.14 percent and the agent framework about 0.05 percent.

AgentPort-Bench tests whether swapping an agent framework changes security outcomes when the model and the attack stay fixed.

Production agents usually sit on a framework rather than a raw model call. Common stacks include LangChain, CrewAI, AutoGen, LlamaIndex, the OpenAI Agents SDK, Google ADK, and Semantic Kernel. Those layers can add system text, change message roles, rewrite tool descriptions, and reorder context. Two agents can therefore send different inputs to the same model even when a tester starts from one user string.

Paper: https://www.researchgate.net/profile/Waqar-Javed-3/publication/414611060_AgentPort-Bench_A-Controlled-Seven-Framework-Evaluation-of-Agentic-AI-Security-Portability/links/6ab34c874b53e25e421f581e/AgentPort-Bench-A-Controlled-Seven-Framework-Evaluation-of-Agentic-AI-Security-Portability.pdf

Three factors, 9,360 runs

The study crossed attack type, model, and agent framework. It covered the seven frameworks above plus a direct HTTP API baseline. Models came from Anthropic, OpenAI, and Google, six capability tiers in total. The completed set was 9,360 controlled runs.

The authors checked whether frameworks sent equivalent payloads to the model. The comparison covered system prompt, user message, message role, tool description, instruction, and serialization. Extra text added by a framework was treated as an adapter difference, not as a framework security effect.

CrewAI adapter gap

CrewAI agents are commonly defined with Role, Goal, and Backstory. An early adapter wrapped the original system prompt into a longer persona and added safety-related wording, including the word safely.

Under that adapter, CrewAI's PASS rate was 66.4 percent. After the adapter was corrected to keep the system prompt aligned, the authors reran 1,170 related experiments. The PASS rate fell to 61.3 percent, a drop of more than five percentage points. Without the adapter audit, that gap could have been read as a CrewAI security advantage.

Missing attack surface

In some configurations, the HTTP baseline and LangChain did not pass a system prompt. One test category was system-prompt extraction. Attack success was near zero because no system prompt was present. The authors treat matching attack surface as a prerequisite for comparing outcomes.

Variance split

After input-equivalence checks, analysis of variance attributed outcome differences as follows:

• Attack type: about 28.00%
• Model: about 4.14%
• Agent framework: about 0.05%

Attack type accounted for roughly 500 times the framework share. The model share was about 80 times the framework share. Under the controlled conditions, the order was attack type, then model, then framework.

Equivalence tests

A non-significant p-value does not prove two frameworks match. The authors ran equivalence tests against a pre-set small-effect bound. Eight execution modes yield C(8,2) = 28 pairwise comparisons. All 28 fell inside the practical-equivalence interval for the security questions in the design.

CrewAI still showed a small residual effect in some statistical models after the adapter fix. The difference was statistically detectable and stayed inside the practical-significance bound. The authors note that CrewAI's Role, Goal, and Backstory message structure may change how a model reads context even when wording is aligned.

What same prompt covers

Equal prompt text does not guarantee equal model input. Role labels, message order, tool schema, function-calling format, templates, context bounds, provider serialization, and hidden framework instructions can still differ. The paper flags full model-input representation as a tighter check for later benchmarks.

Empty output and reasoning budget

One model, on some test inputs, spent its internal reasoning-token budget and returned an empty visible answer. The authors describe the pattern as reasoning-resource exhaustion: long internal reasoning hits the token cap before a final answer is produced. The observation is separate from the main portability result.

Scope

The measured path is attack prompt to framework to model to text output to a safety check. Production agents also plan tasks, select tools, apply permissions, call tools, update state and memory, and loop. The study did not cover tool permissions, MCP, credentials, sandboxes, memory, state, hooks, callbacks, multi-agent delegation, human approval, retries, or external actions.

The supported claim is that, when attack payload and model input are held as consistent as possible and the score is mainly the text response, orchestration-framework differences are small. Runtime equivalence for those untested components was outside the measured design.