Prompt injection is the #1 risk for LLM applications. It occurs when an attacker crafts inputs that cause the model to deviate from its intended behavior—overriding system instructions, exfiltrating data, or triggering unauthorized actions.

Types

• Direct injection: User input directly manipulates the model's behavior (e.g., "Ignore previous instructions and...")
• Indirect injection: Malicious content embedded in external data sources (web pages, documents, emails) that the model processes
• Jailbreaking: A subset of prompt injection aimed at bypassing safety guardrails

Why it's hard to fix

Unlike SQL injection, there's no reliable separation between instructions and data in natural language. Models process both through the same mechanism, making perfect input sanitization theoretically impossible with current architectures.

Mitigations

• Input validation and character filtering
• Instruction hierarchy and privilege separation
• Output monitoring and anomaly detection
• Sandboxing tool access with least-privilege
• Continuous red-teaming and adversarial testing

Training data attacks compromise the model before it ever reaches production. An adversary who controls even a small fraction of training data can embed backdoors, bias outputs, or create hidden vulnerabilities.

Attack vectors

• Data poisoning: Inserting malicious samples into training datasets
• Backdoor injection: Creating trigger patterns that activate specific model behaviors
• Model supply chain compromise: Pre-trained models from public repositories with embedded vulnerabilities

Mitigations

• Data provenance tracking and validation
• Statistical anomaly detection on training data
• Model behavior testing against known trigger patterns
• Vetting third-party models and datasets before use

LLMs can inadvertently leak PII, proprietary data, API keys, or internal system information through their outputs. This has surged to the 2nd most critical vulnerability in the 2025 OWASP list.

How it happens

• Training data memorization — the model "remembers" and regurgitates sensitive data
• System prompt leakage — attackers extract the system instructions
• Context window exposure — sensitive data in conversation context gets revealed
• Model inversion attacks — reconstructing training data from model outputs

Mitigations

• PII detection and redaction in both inputs and outputs
• Data minimization in training and context
• Output filtering and classification
• Differential privacy in training
• Regular audits for data leakage

Model weights, architectures, and fine-tuning data represent significant intellectual property. Adversaries use API access to extract or replicate proprietary models.

Techniques

• Model extraction: Systematic querying to build a functionally equivalent copy
• Weight theft: Direct access to model files through infrastructure compromise
• Side-channel attacks: Inferring model architecture from response timing or patterns

Mitigations

• Rate limiting and query monitoring
• Response watermarking
• Encryption of model assets at rest and in transit
• Access control and audit logging

Employees using unsanctioned AI tools create security blind spots. Sensitive data gets uploaded to unknown third-party services, bypassing governance and creating compliance risks.

Mitigations

• Deploy CASB tools to detect unsanctioned AI usage
• Provide secure, company-approved AI alternatives
• Create clear acceptable-use policies for AI tools
• Monitor network traffic for AI service endpoints

Manipulating LLM behavior through crafted inputs. Includes both direct manipulation and indirect injection via external data. Jailbreaking is a specific form where attackers bypass safety protocols.

Key defenses

• Input sanitization and validation layers
• Instruction hierarchy enforcement
• Output anomaly detection
• Human-in-the-loop for sensitive operations

LLMs can expose PII, proprietary algorithms, or confidential business data through model outputs. This vulnerability has surged in importance as organizations feed increasingly sensitive data into AI systems.

Key defenses

• Data Loss Prevention (DLP) for AI outputs
• PII detection and auto-redaction
• Data minimization principles in training and prompts
• Output classification and filtering

Risks from third-party pre-trained models, datasets, and deployment platforms. Unlike traditional software supply chains, LLM supply chains include model weights and training data which are harder to audit.

Key defenses

• Model provenance verification
• Dataset auditing and documentation
• Model Bill of Materials (MBOM)
• Sandbox testing before production deployment

Attackers manipulate training data or fine-tuning processes to embed vulnerabilities, biases, or backdoors in models.

Key defenses

• Robust data pipeline validation
• Anomaly detection on training distributions
• Regular model behavior audits
• Secure fine-tuning environments

Failing to validate, sanitize, or appropriately handle LLM outputs before passing them to downstream components. Can lead to XSS, SSRF, code execution, or privilege escalation in connected systems.

Key defenses

• Treat all LLM outputs as untrusted input
• Apply context-appropriate encoding and sanitization
• Limit LLM output use in privileged operations
• Output schema validation

Granting LLMs too much autonomy, permissions, or access to tools and systems. When an LLM can take real-world actions (send emails, execute code, access databases), the blast radius of any vulnerability multiplies.

Key defenses

• Principle of least privilege for all tool access
• Human-in-the-loop for consequential actions
• Action allow-listing instead of block-listing
• Rate limiting on automated actions

Attackers extract the system prompt, revealing internal instructions, safety constraints, tool schemas, and business logic. This information can then be used to craft more effective attacks.

Key defenses

• Don't rely on system prompt secrecy for security
• Minimize sensitive info in system prompts
• Output monitoring for prompt content leakage
• Defense-in-depth beyond the prompt layer

RAG (Retrieval-Augmented Generation) systems introduce attack vectors through their embedding stores. Adversaries can poison knowledge bases, manipulate retrieval results, or exploit access control gaps in vector databases.

Key defenses

• Access control on vector store contents
• Input validation for documents entering RAG pipelines
• Source attribution and provenance tracking
• Regular audits of embedded content

LLMs can generate plausible but inaccurate content (hallucinations), which can erode trust, cause reputational harm, or lead to incorrect decisions when used in high-stakes contexts.

Key defenses

• Grounding outputs in verified data sources
• Confidence scoring and uncertainty signaling
• Human review for high-stakes outputs
• Retrieval-augmented generation with cited sources

Previously focused on Denial of Service, this category now encompasses broader resource management risks. Large-scale LLM deployments face challenges managing compute costs, API abuse, and resource exhaustion.

Key defenses

• Per-user and per-request rate limiting
• Token budget enforcement
• Cost monitoring and alerting
• Input length and complexity limits

The NIST AI RMF provides a flexible, voluntary framework for managing AI risks across the entire development lifecycle. It's organized around four core functions:

Core Functions

• GOVERN: Establish and manage AI risk management policies, roles, and processes across the organization
• MAP: Identify and document AI system context, capabilities, intended uses, and potential impacts
• MEASURE: Analyze, assess, and track AI risks using quantitative and qualitative methods
• MANAGE: Prioritize and act on identified risks with appropriate mitigation strategies

Key strengths

Flexible and scalable for any organization size. Addresses technical, social, and ethical dimensions. Expected to receive v1.1 updates with expanded evaluation methodologies through 2026.

The world's first comprehensive AI regulation with binding legal force. It uses a risk-based classification system and applies to anyone deploying AI systems within or affecting the EU market.

Risk tiers

• Unacceptable risk: Banned (social scoring, real-time biometric surveillance, etc.)
• High risk: Strict requirements — documented risk management, data governance, human oversight, conformity assessments
• Limited risk: Transparency obligations (users must know they're interacting with AI)
• Minimal risk: No specific requirements

Key obligations for high-risk AI

• Full data lineage tracking
• Human-in-the-loop for workflows impacting safety, rights, or financial outcomes
• Risk classification tagging for each model
• Conformity assessments before market deployment

Enforcement timeline

General-purpose AI obligations are in effect. High-risk system rules see phased enforcement through August 2026. Maximum penalties reach €35 million or 7% of global turnover.

The first international certifiable standard for AI management systems. It provides a systematic approach to responsible AI development and deployment, similar to how ISO 27001 works for information security.

Value proposition

• Provides externally verifiable assurance of AI governance
• Builds stakeholder trust through independent certification
• Helps demonstrate compliance with regulatory requirements (like the EU AI Act)
• Aligns with existing management system standards (ISO 27001, etc.)

Anthropic achieved ISO 42001 certification in January 2025, one of the first major AI companies to do so.

The Adversarial Threat Landscape for AI Systems (ATLAS) extends MITRE's ATT&CK framework to AI. It provides a knowledge base of adversary tactics and techniques targeting machine learning systems.

Why it matters

• Maps AI-specific attacks to a structured taxonomy
• Enables threat modeling specific to ML/AI systems
• Provides a shared language for AI security teams
• Covers scenarios from model poisoning to service denial

Catalog every AI system in your organization. For each, determine its risk classification under the EU AI Act and document its intended purpose, data sources, and stakeholders.

• Create an AI system registry with ownership and lifecycle stage
• Classify each system's risk level (minimal, limited, high, unacceptable)
• Document data lineage and model provenance
• Identify shadow AI usage across the organization

Assign clear ownership of AI risk management. Typical owners include GC, CISO, or Chief Risk Officer. Build cross-functional teams spanning legal, engineering, and product.

• Define AI-specific roles and responsibilities
• Create AI acceptable-use policies
• Establish incident response procedures for AI failures
• Integrate AI governance into existing risk management processes

Deploy the technical safeguards required by your framework of choice. Map controls to NIST AI RMF functions and EU AI Act requirements simultaneously.

• Input validation and output filtering
• Access controls and authentication for AI services
• Monitoring, logging, and audit trails
• Data protection and privacy controls
• Model testing and red-teaming pipelines

AI security isn't a one-time effort. Models drift, threats evolve, and regulations update. Build continuous monitoring into your operations.

• Monitor model behavior for drift and anomalies
• Regular adversarial testing and red-teaming exercises
• Track regulatory updates and adjust controls accordingly
• Report and iterate on AI incidents and near-misses

Character & length limits

Enforce maximum input length and strip or reject suspicious character patterns. This is your first line of defense against prompt injection.

Keyword & pattern filters

Block known injection patterns while being careful not to create bypasses through overly specific rules. Combine with semantic analysis for better coverage.

Input classification

Use a lightweight classifier to flag inputs that look like injection attempts before they reach the main model. This adds latency but catches a significant fraction of attacks.

System prompt hierarchy

Structure prompts with clear privilege levels. System instructions should be hardened against override attempts. Never rely on prompt secrecy alone—assume attackers can read your system prompt.

Context separation

Separate user data from instructions using clear delimiters and encoding. While not foolproof, this raises the bar for injection attacks.

RAG security

When using retrieval-augmented generation, treat all retrieved content as untrusted input. Apply access controls to vector stores and validate retrieved content before injection into prompts.

PII detection

Scan all model outputs for personally identifiable information before returning to users. Use both pattern matching (regex for SSNs, emails, etc.) and ML-based entity recognition.

Content classification

Classify outputs for harmful, off-topic, or policy-violating content. This catches both adversarial manipulation and benign model failures.

Schema validation

When LLM outputs feed into downstream systems (APIs, databases, code execution), enforce strict schema validation. Treat LLM output as untrusted input to every downstream component.

Authentication & authorization

Enforce strong API authentication (OAuth 2.0, API keys with rotation). Apply per-user and per-organization rate limits. Use scoped tokens with minimum required permissions.

Encryption

TLS 1.3 for all API traffic. Encrypt model weights and sensitive data at rest. Use hardware security modules (HSMs) for key management in high-security environments.

Network segmentation

Isolate AI infrastructure from general corporate networks. Use dedicated VPCs for model serving. Implement egress filtering to prevent data exfiltration through model outputs.

Multi-tenant isolation

In multi-tenant AI deployments, ensure strict isolation of models, data, and compute resources. Cross-tenant data leakage through shared models is a real risk.

Why red-team AI differently

Traditional penetration testing doesn't cover AI-specific attack vectors. Red-teaming for AI means testing prompt injection resilience, data extraction resistance, safety guardrail robustness, and tool abuse scenarios.

Building a testing pipeline

• Create adversarial prompt libraries covering known injection techniques
• Test against the OWASP Top 10 for LLM Applications systematically
• Automate regression testing for known vulnerabilities
• Include red-teaming in CI/CD pipelines
• Engage external security researchers through bug bounties

OWASP resources

The OWASP GenAI Security Project provides a FinBot Capture The Flag application designed specifically for practicing agentic AI security skills in a controlled environment.