SafeGesture: Evaluating Fine-Grained Hand Gesture Understanding in Vision-Language Models through Scenario-Conditioned Safety Interpretation

Vision-language models struggle with safety-critical reasoning despite strong gesture recognition: GPT-4o correctly identified hand gestures 98.4% of the time but only achieved 53.3% accuracy when asked to recommend appropriate safety actions in specific scenarios—a 45-percentage-point gap that reveals models can "see" gestures but cannot reliably reason about what actions they should trigger in context.

Current models show concerning overconfidence and rarely express uncertainty: Four of the five tested models almost never used an uncertainty label when unsure, suggesting they may confidently provide incorrect safety recommendations rather than acknowledging limitations—a critical flaw for deployment in real-world safety applications.

Scenario context is the primary challenge, not gesture recognition: Even when researchers provided models with the correct gesture as text input, performance improved by only 0.4 to 3.2 percentage points, indicating that understanding what safety action a gesture should prompt in a given situation is far harder for these models than recognizing the gesture itself.

Simple statistical patterns can outperform current AI models: A basic policy that simply guessed the most common safety action for each scenario achieved 58.3% accuracy—higher than most evaluated models—highlighting that models may be learning superficial patterns rather than genuine safety reasoning.

Visual information helps but isn't sufficient: While providing visual input improved safety accuracy by 11.2 to 30.2 percentage points compared to text-only baselines, no model exceeded 56.2% accuracy overall, suggesting that vision-language models need fundamental improvements in safety reasoning before deployment in gesture-based safety systems.