The Dark Side of Function Calling: Pathways to Jailbreaking Large Language Models

Function calling is a major security blind spot: While researchers have extensively studied safety risks in standard LLM chat interactions, the function calling feature—which allows LLMs to execute specific tasks or access external tools—has received minimal security scrutiny, leaving a significant gap in AI safety protections.

A new attack method succeeds over 90% of the time: The researchers demonstrated a "jailbreak function" attack that exploits misalignments in how LLMs handle function calls, achieving an alarming average success rate of over 90% across six leading models including GPT-4o, Claude-3.5-Sonnet, and Gemini-1.5-pro.

Multiple vulnerabilities enable the attacks: The jailbreak exploits three key weaknesses: alignment discrepancies (where safety training doesn't fully extend to function calling), user coercion tactics, and the absence of rigorous safety filters specifically designed for function calls.

This represents an urgent, practical threat: Since function calling is increasingly deployed in real-world applications to connect LLMs with external systems and tools, this vulnerability could enable attackers to manipulate AI systems into performing harmful actions at scale.

Defensive measures exist but require implementation: The researchers propose practical defensive strategies, including the use of defensive prompts, suggesting that enhanced security measures are both necessary and feasible—but require immediate adoption by AI developers.