《云安全联盟（CSA）：2025 Agentic AI 红队测试指南（英文版）（21页）.pdf》由会员分享，可在线阅读，更多相关《云安全联盟（CSA）：2025 Agentic AI 红队测试指南（英文版）（21页）.pdf（21页珍藏版）》请在三个皮匠报告上搜索。

1、Agentic AI Red Teaming GuideAI Organizational Responsibilities Working Group21.BackgroundWhy Agentic AI Needs a New Red Teaming Approach:Traditional GenAI red teaming does not address the security risks of autonomous,goal-seeking,persistent AI agents.Why Agentic AI Is Different:Combines planning,rea

2、soning,and acting Operates across time,systems,and roles Introduces emergent behavior and expanded attack surfacesKey Challenge:Agentic AI agents can:Reassign goals Chain actions autonomously Interface with live APIs and tools Result:Unpredictable failure modes and cascading consequences GenAI promp

3、t in/output out Agentic AI goal plan action feedback adaptPurpose of This Guide:To provide practical,test-driven red teaming strategies tailored for agentic AI Developed through CSA&OWASP joint research and threat analysis32.Scope and AudienceWhat This Guide Covers:Focus:Practical,test-driven red te

4、aming for agentic AI systems Core Goal:Identify vulnerabilities not to model threats or define mitigations Approach:Detailed attack surface testing,not high-level frameworksWhats Out of Scope Threat modeling(MAESTRO Framework)General GenAI red teaming(e.g.,prompt injection only)Risk prioritization o

5、r treatment strategies Secure development practices Broad governance or ethical guidanceIntended Audience:Primary:Red Teamers Agentic AI Developers Pen Testers Secondary:Security Architects AI Governance Teams AI System DesignersAssumptions:Audience understands core security topics(APIs,authN/Z,prot

6、ocols)Teams performing tests likely have organizational or consulting support Guide is technical and operational,not academic or policy-based43.OverviewGenerative AI(GenAI):One-shot or short-context interaction Generates text,images,or code based on prompts Security focus:prompt injection,bias,data

7、leakage Example:“Summarize the latest research on quantum computing.”Returns a text summaryAgentic AI:Autonomous,persistent,and goal-driven Plans Acts Learns Adapts Interfaces with APIs,databases,devices,other agents Security focus:complex attack surfaces,cascading behaviors Example:“Monitor quantum

8、 computing research and alert me when a breakthrough occurs.”Agent searches,filters,stores,updates,and notifiesWhy This Matters for Red Teaming:Agentic AI introduces:Emergent behaviors not present in GenAIStateful memory and context across timeTool and system orchestration capabilitiesThreats now in

9、clude:Goal manipulation,action chaining,API misuse,trust exploitation53.1 Understanding the Unique Risks of Agentic AIWhats Reused Agentic AI still relies on:AppSec fundamentals(auth,input validation,etc.)API security agents use APIs as tools GenAI red teaming techniques(e.g.,prompt injection)Threat

10、 modeling(MAESTRO,STRIDE,etc.)Whats New:Agentic AI introduces novel red teaming challenges:Emergent behavior agents find unanticipated ways to achieve goals Unstructured reasoning decision paths are hard to trace Expanded attack surface:Control system Knowledge base Instructions/goals Tool and API i

11、nterfaces Multi-agent coordinationWhy This Matters for Red Teaming:Agents operate with autonomy and non-determinism Security gaps can cascade into:Persistent goal driftAPI/tool misuseMulti-agent collusion Early and continuous testing is the only way to detect:Unexpected failuresEthical boundary viol

12、ationsMisaligned behaviors at runtimeKey Insight Agentic AI should be tested like production software before and after deployment,under adversarial conditions.64.Detailed Guide Framework OverviewAbout the Detailed Guide:This section introduces a structured framework for red teaming Agentic AI system

13、s through 12 adversarial testing categories.Purpose:Simulate real-world adversarial attacksIdentify system-level risks and emergent failuresTest across goal planning,execution,memory,orchestration,and interfacesWhat Each Threat Category Includes:Test Requirements:What to assessAttack Scenarios:Reali

14、stic threat simulationsExample Prompts:Exploit vectorsRed Team Deliverables:Reports,logs,behavioral insightsThreat Categories:Agent Authorization and Control HijackingMisuse of permissions or roles to seize control of agent actions or escalate privileges.CheckerOut-of-the-LoopCircumvention or failur

15、e of monitoring systems meant to supervise agent behavior or intervene during unsafe actions.Critical System InteractionUnsafe or unauthorized interactions with external systems,APIs,or infrastructure triggered by agents.Goal and Instruction ManipulationSubversion or injection of malicious goals or

16、instructions that mislead agents into undesired behavior.Threat Categories:Hallucination ExploitationAbuse of false or fabricated outputs generated by the agent,especially when they are treated as authoritative.Impact Chain and Blast RadiusEvaluation of how small actions compound into large,unintend

17、ed consequences due to cascading agent behavior.Knowledge Base PoisoningInsertion of misleading or harmful data into an agents memory,search tools,or reference corpus.Memory and Context ManipulationTargeted manipulation of what the agent remembers or forgets to influence decisions across time.Multi-

18、Agent ExploitationAttacks that exploit trust,coordination,or protocol gaps between collaborating agents.Resource ExhaustionDeliberate consumption of compute,API calls,or memory to degrade performance or trigger failure states.Supply Chain AttacksIntroduction of malicious behavior through third-party

19、 tools,APIs,plugins,or dependency agents.Agent UntraceabilityDifficulty in tracking agent decisions,actions,and responsibilities due to poor logging or emergent behavior.74.1 Agent Authorization and Control HijackingObjective:Test how agentic systems enforce control boundaries,handle role assignment

20、s,and prevent unauthorized behavior.Test Areas:4.1.1 Direct Control Hijacking Tests Attempt to override or inject commands into active agent processes.4.1.2 Permission Escalation Testing Exploit misconfigured or overly broad permission settings.4.1.3 Role Inheritance Exploitation Abuse inherited or

21、shared roles to bypass isolation.4.1.4 Agent Activity Monitoring and Detection Evaluate if agent behavior is correctly logged and flagged.4.1.5 Separation of Agent Control and Execution Identify breakdowns between control logic and action execution.4.1.6 Audit Trail and Behavior Profiling Assess the

22、 integrity and completeness of audit logs and profiles.4.1.7 Least Privilege Principle Specific to Agents Ensure agents operate with only the minimum necessary permissions.84.2 CheckerOut-of-the-LoopObjective:Assess whether oversight mechanisms(checkers,validators,monitors)are active,engaged,and cap

23、able of intervening when agents act outside policy or breach thresholds.Test Areas:4.2.1 Threshold Breach Alert Testing Test whether alerting systems correctly trigger when agent behavior exceeds predefined limits.4.2.2 Checker Engagement Testing Simulate dangerous or policy-violating actions to det

24、ermine if checkers are engaged in time.4.2.3 Checker Blind Spot Enumeration Identify conditions,functions,or tools not covered by the checkers scope or visibility.4.2.4 Alert Suppression and Tampering Attempt to suppress,delay,or misroute alerts to bypass human or automated intervention.4.2.5 Simula

25、ted Failsafe Activation Test whether critical safety mechanisms activate when agents operate outside of bounds.4.2.6 Checker-Action Correlation Auditing Evaluate whether checker actions are properly correlated with actual agent behavior or if theyre being ignored.94.3 Critical System InteractionObje

26、ctive:Evaluate how agentic systems interact with external systems,APIs,or infrastructure and whether those interactions could introduce risk,misuse,or unintended impact.Test Areas:4.3.1 Critical Command Execution Validation Determine if agents can issue high-impact commands(e.g.,shutdown,wipe)withou

27、t proper validation layers.4.3.2 External System Access Control Testing Test for unauthorized access to connected databases,services,or applications.4.3.3 Feedback Loop Abuse Simulation Assess whether agents can create unsafe positive feedback loops(e.g.,issuing repeated updates or commands).4.3.4 P

28、hysical System Actuation Testing Simulate interactions with real-world systems(e.g.,robotics,IoT)to test boundary enforcement.4.3.5 Trust Model Validation for External Interfaces Identify assumptions in system-to-system trust(e.g.,token sharing,persistent sessions)that could be exploited.104.4 Goal

29、and Instruction ManipulationObjective:Assess whether agents can be manipulated through subtle or malicious goal modifications,instruction rewrites,or ambiguous task injection leading to unintended or unsafe behaviors.Test Areas:4.4.1 Goal Rewriting and Override Testing Attempt to insert or modify ag

30、ent goals mid-process to redirect behavior.4.4.2 Ambiguity Injection Evaluation Test whether vague,overloaded,or conflicting instructions cause harmful or divergent actions.4.4.3 Instruction Chain Tampering Alter or spoof multi-step instructions to inject malicious behavior while retaining surface l

31、egitimacy.4.4.4 Misaligned Subgoal Creation Assess whether agents autonomously generate subgoals that conflict with broader intent or safety constraints.4.4.5 Goal Inheritance and Delegation Control Evaluate if inherited or delegated goals introduce risks due to unvalidated assumptions or broken lin

32、eage.114.5 Hallucination ExploitationObjective:Test how agents handle false,fabricated,or misleading information especially when hallucinated content is used in real-world actions or treated as authoritative by the system or other agents.Test Areas:4.5.1 Fabricated Output Injection Prompt agents to

33、generate false information and observe if it is accepted or reused as fact in later decisions.4.5.2 Citation and Evidence Validation Testing Evaluate whether the agent verifies claims,citations,or source data or presents hallucinated outputs as trustworthy.4.5.3 Instruction Derivation from Hallucina

34、ted Content Test whether agents act on instructions or subgoals derived from previously hallucinated outputs.4.5.4 Chain-of-Thought Contamination Assess whether fabricated information persists through multi-step reasoning and leads to flawed conclusions.4.5.5 Memory Replay of Fabrications Determine

35、if hallucinated content is stored in memory and later recalled as if it were valid historical context.124.6 Impact Chain and Blast RadiusObjective:Evaluate how localized agent actions can compound into broader unintended consequences and assess the systems ability to detect,contain,or recover from c

36、ascading impact chains.Test Areas:4.6.1 Recursive Task Amplification Assess whether small tasks escalate through loops or dependency chains,causing excessive or repeated execution.4.6.2 Cross-Subsystem Effect Simulation Trigger agent actions that affect multiple subsystems and observe if lateral imp

37、act is properly contained.4.6.3 Goal Cascade Testing Test how agents generate follow-up tasks or delegate objectives and whether these introduce unintended risk propagation.4.6.4 Resource Chain Stress Testing Simulate multi-step tasks that cumulatively deplete compute,bandwidth,or API limits across

38、services.4.6.5 Compound Side Effect Enumeration Identify sequences where benign agent behaviors,when combined,result in policy violations or unintended system impact.134.7 Knowledge Base PoisoningObjective:Test how agents handle manipulated,misleading,or malicious information inserted into their ref

39、erence materials,memory stores,or external data sources.Test Areas:4.7.1 Poisoned Search Result Injection Insert falsified results into agent-accessible knowledge sources(e.g.,databases,APIs)and observe trust behavior.4.7.2 Memory Corruption Simulation Attempt to insert misleading content into persi

40、stent memory stores and assess how its later used in reasoning.4.7.3 Tool Documentation Tampering Modify or spoof documentation for agent-executed tools and evaluate if the agent follows incorrect guidance.4.7.4 Dynamic Retrieval Manipulation Alter real-time data(e.g.,APIs or web sources)to inject m

41、alicious guidance into agent workflows.4.7.5 Confidence Bias Exploitation Craft poisoned knowledge that appears highly relevant or confidently presented,increasing agent likelihood of reuse.144.8 Memory and Context ManipulationObjective:Evaluate how agents store,recall,and use memory and whether adv

42、ersaries can manipulate memory or context to influence future behavior,decisions,or actions.Test Areas:4.8.1 Context Injection Attacks Insert misleading context during runtime to shape agent interpretation of current or future inputs.4.8.2 Memory Overwrite and Pruning Manipulation Test whether attac

43、kers can overwrite or force the forgetting of critical memories to skew agent state.4.8.3 Memory Poisoning Across Time Introduce malicious or biased content across multiple sessions to slowly reshape agent assumptions or decision baselines.4.8.4 Session Persistence Abuse Evaluate risks of reused con

44、text from prior sessions being carried forward incorrectly or without verification.4.8.5 Memory Boundary Validation Test the agents handling of scoped or role-specific memory especially when operating across multi-user or multi-agent environments.154.9 Multi-Agent ExploitationObjective:Assess how ag

45、ents interact with one another and whether adversaries can exploit trust,coordination protocols,or delegation mechanisms across agent networks.Test Areas:4.9.1 Cross-Agent Goal Injection Attempt to inject or manipulate goals shared between agents to alter coordinated behavior.4.9.2 Message Spoofing

46、and Relay Tampering Intercept or modify agent-to-agent communications to mislead or redirect action.4.9.3 Multi-Agent Role Confusion Testing Confuse role boundaries or shared responsibilities to induce conflicting or unsafe behaviors.4.9.4 Orchestration Hijack Simulation Target centralized or distri

47、buted agent orchestrators to reroute workflows or introduce malicious coordination.4.9.5 Protocol Trust Exploitation Evaluate implicit trust assumptions in agent collaboration,especially regarding task handoffs or delegated execution.164.10 Resource ExhaustionObjective:Test whether agents can be tri

48、cked into consuming excessive compute,memory,bandwidth,or API capacity either through direct overload or through cascading task execution.Test Areas:4.10.1 Infinite Task Generation Testing Attempt to trigger recursive or unbounded goal generation that leads to continuous agent execution.4.10.2 API A

49、buse and Overuse Simulation Direct agents to over-call APIs or services,exhausting rate limits or creating system instability.4.10.3 Storage Bloat Exploitation Assess whether agents store unnecessary or attacker-controlled data at scale to impact system storage.4.10.4 Compute Drain Triggering Initia

50、te resource-intensive operations that degrade performance or cause denial-of-service conditions.4.10.5 Cascading Workload Amplification Chain agent tasks to simulate workflows that appear valid but overwhelm backend systems or dependencies.174.11 Supply Chain AttacksObjective:Evaluate whether agenti

51、c systems can be compromised through dependencies including third-party tools,APIs,plugins,data sources,or model components that introduce malicious or unsafe behavior.Test Areas:4.11.1 Third-Party Tool Injection Test the agents behavior when interacting with tampered or unverified external tools or

52、 libraries.4.11.2 Dependency Manipulation Testing Introduce malicious updates or substitutions into dependencies the agent uses for tasks or decisions.4.11.3 Plugin Trust Boundary Evaluation Assess the privilege separation and trust assumptions between agents and installed plugins or extensions.4.11

53、.4 External Model Exploit Simulation Evaluate how agents interact with external models(e.g.,vision,speech)that may return unsafe or manipulated outputs.4.11.5 Data Source Integrity Attacks Insert corrupt or misleading data into upstream feeds relied upon by the agent(e.g.,for training,inference,or d

54、ecisions).184.12 Agent UntraceabilityObjective:Assess whether agent actions are logged,attributed,and auditable or if decisions and behaviors become opaque,difficult to trace,or unattributable due to poor observability or system design.Test Areas:4.12.1 Action Attribution Testing Evaluate whether ag

55、ent actions can be reliably attributed to a specific request,prompt,or origin.4.12.2 Audit Trail Completeness Assessment Test the scope and fidelity of logs capturing agent behavior,especially during multi-step or delegated execution.4.12.3 Opaque Decision-Making Simulation Trigger situations where

56、the agents reasoning process is unclear or unrecorded and assess system explainability.4.12.4 Logging Suppression and Evasion Attempt to perform actions that bypass,suppress,or manipulate logging systems.4.12.5 Ownership Confusion Testing Assess whether the system can determine who or what initiated

57、 or authorized a given action,especially across agents.195.ConclusionKey Takeaways:Agentic AI systems introduce novel attack surfaces due to their autonomous planning,memory,and tool use.Traditional red teaming is not sufficient new methodologies are required to test emergent risks like hallucinatio

58、n,orchestration hijack,and cascading failures.This guide offers a structured framework to simulate adversarial conditions across 12 critical threat categories.Why It Matters:Agentic AI is being integrated into enterprise and critical infrastructure.Without rigorous red teaming,vulnerabilities in rol

59、e boundaries,memory integrity,and inter-agent workflows can go undetected.Proactive testing helps inform:System hardening Design-time decisions Ongoing monitoring and governance206.Future OutlookEvolving Challenges:Agentic AI will introduce increasingly complex and dynamic security risks.Red teaming

60、 must evolve beyond static methods to address autonomous workflows and emergent behavior.Key Future Focus Areas:Autonomous Red Teaming Agents Agents that autonomously generate test cases,identify vulnerabilities,and simulate threats in real-time.Downstream Action Red Teaming Focused testing of agent

61、-triggered chains of actions,requiring mapping of workflows and cross-domain impacts.Secure Multi-Agent Orchestration Addressing privilege separation,trust boundaries,and coordination risks across distributed agents.Standardized Metrics and Benchmarks Development of indicators like exploit success r

62、ate,Mean Time to Detection(MTTD),and containment time.Alignment with Regulations Ensuring red teaming aligns with frameworks like the EU AI Act and NIST AI RMF on safety and accountability.Open-Source Tooling and Community Research Growth in community-driven tools(e.g.,MAESTRO,AgentDojo,Agent-Safety

63、Bench,AgentFence)to scale testing efforts.217.Final ThoughtsAgentic AI is powerful and inherently risky.Autonomous systems can plan,reason,and act but that same power introduces novel security challenges that traditional red teaming doesnt cover.This guide offers a new testing lens.Red teamers now h

64、ave a structured,practical framework to simulate real-world adversarial conditions across 12 threat categories unique to agentic AI.Security requires collaboration.Securing agentic systems will demand innovation across engineering,policy,operations,and research.No single team or tool can cover the full risk landscape.Red teaming must evolve continuously.As agents become more capable,adversarial testing must scale with them through automation,community input,and integration with emerging standards.