Prompt Injection (c95f6f8d-b1dd-55d7-b77a-100c9e131313)
AI models, particularly large language models (LLMs), are susceptible to prompt injection attacks, where adversaries craft inputs designed to override model constraints, extract sensitive data, or manipulate system behavior.
- Meta Prompt Extraction: Attackers can manipulate prompts to reveal system instructions, policies, or proprietary data.
- Indirect Injection Attacks: If an AI model ingests untrusted external content, such as the contents or names of uploaded files, text from emails, chat inputs, or web pages, attackers can embed hidden prompts or malicious instructions within these elements. These indirect inputs can exploit the model's processing logic to alter its behavior, produce misleading responses, or trigger unauthorized actions, even without direct access to the model's interface.
- System Command Override: Specially crafted prompts could trick AI models into executing unintended actions or disclosing confidential information.
Threat-modeling question: Could the AI system be vulnerable to prompt injection attacks, leading to unauthorized access or manipulation?