Jailbreaking LLMs: A Survey of Attacks, Defenses and Evaluation

Abstract

Large Language Models (LLMs) excel at natural language understanding and generation, but deployment introduces critical security risks through jailbreak attacks that circumvent safety alignment. This survey provides the first unified rigorous systematization of the LLM security threat landscape (2022-2025), synthesizingresearch across premier security and AI venues. We introduce a comprehensive taxonomy of jailbreak vectors from elementary prompt manipulation to sophisticated multimodal exploits. We find a persistent asymmetry between attack sophistication and defensive capability: advanced automated attacks routinely achieve 90-99% success on open-weight models, while black-box attacks reach 80-94% effectiveness on proprietary models. Agent-driven multi-turn attacks demonstrate 95% success by decomposing harmful queries across conversation turns. Embodied AI vulnerabilities enable jailbreaks to trigger harmful physical actions in robotic platforms, expanding threats beyond digital domains. Defenses show fundamental limits: feedback-based attacks often retain residual success above 15% even against layered protections. Evaluation remains unreliable, with automated judge agreement varying 70-93% depending on implementation, weakening confidence in security assessments. By examining attack transferability, computational costs, and defense overhead, we identify gaps in multimodal safety protocols, agent-based security frameworks, and mechanistic interpretability. Robust LLM security requires shifting from reactive mitigation to proactive security-by-design architectures integrating constitutional AI principles with formal verification.