Prompt Stability in Code LLMs: Measuring Sensitivity across Emotion- and Personality-Driven Variations

Performance and stability are separate concerns: The study found that across 14 code generation models, high performance on coding tasks does not guarantee stability—models that generate correct code may still produce wildly different outputs when prompts are rephrased with different emotional tones or communication styles, meaning developers cannot assume reliable behavior just because a model works well on benchmarks.

Prompt phrasing significantly affects code generation: Identical coding requirements expressed with different emotions (e.g., urgent vs. casual) or personality styles produced divergent outputs, demonstrating that code LLMs are sensitive to superficial variations in how requests are worded rather than just their technical content—a concerning finding for real-world deployment where users may phrase requests differently.

Larger models don't automatically mean more stable models: The research revealed "architectural and scale related patterns that challenge common assumptions about model robustness," suggesting that simply using bigger or newer models (tested across Llama, Qwen, and DeepSeek families) does not guarantee more predictable behavior, which has important implications for model selection decisions.

Stability should be evaluated alongside performance and fairness: The paper proposes treating prompt stability as a complementary evaluation dimension for responsible AI, meaning organizations deploying code generation tools should measure not just whether models work correctly, but whether they work consistently regardless of how requests are phrased—currently an under-examined aspect of AI safety.

New evaluation framework enables practical screening: PromptSE provides a practical tool for both researchers and practitioners to quantify stability trade-offs before deployment, supporting rapid assessment of closed-source models and enabling more informed decisions about which tools are trustworthy for software development assistance.