On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective

ChatGPT performs better than comparable models on adversarial and out-of-distribution tasks, but still falls short of perfection, suggesting that robustness remains a significant vulnerability in large language models that could affect their reliability in real-world applications.

The model shows concerning behavior in medical contexts, tending to provide informal suggestions rather than definitive answers for medical diagnosis tasks, which raises safety concerns if users rely on it for health-critical decisions.

Adversarial and out-of-distribution robustness are identified as "significant threats to foundation models", meaning these AI systems can produce unreliable outputs when encountering unexpected or deliberately manipulated inputs—a critical issue for responsible AI deployment.

ChatGPT demonstrates particular strength in dialogue-related tasks, outperforming other models in understanding conversational text, which may explain its popularity but also suggests its design may prioritize conversational ability over robustness in other domains.

Robustness testing is essential for responsible AI, especially for safety-critical applications, and this paper demonstrates that comprehensive evaluation of how AI systems handle unexpected inputs should be a standard part of responsible AI assessment before deployment.