Is "Knowing It's Malicious Enough?" Evaluating LLMs for Fine-Grained Malware Behavior Auditing

LLMs show promise but have unclear reliability for malware analysis: The paper reveals that while Large Language Models appear promising for auditing malicious software behavior, their actual reliability and trustworthiness for this critical security task remains largely unknown—a gap that could have serious consequences if these systems are deployed without proper validation.

Current AI systems struggle to explain why malware is dangerous: Traditional automated malware detection flags suspicious files, but security analysts need more than a warning—they need clear explanations of malicious behaviors backed by specific code evidence. The paper finds that both signature-based methods and existing AI explanation techniques "often fail to provide such support in a human-interpretable form."

Verifying AI claims about code is a major challenge: A key finding is the lack of "reliable mechanisms to verify whether generated claims are grounded in code evidence." This means even when an LLM explains malware behavior, there's currently no robust way to confirm the explanation actually matches the code, creating a potential trust gap in security auditing.

Real-world constraints limit current AI capabilities: The paper identifies that actual malware codebases often exceed what current LLMs can process at once, and the lack of expert-written ground truth makes it difficult to properly test and benchmark these systems—highlighting a gap between lab conditions and real-world deployment needs.

Structured evaluation frameworks are essential for responsible deployment: The MalEval framework demonstrates that rigorous, stage-by-stage testing with expert validation is necessary before deploying LLMs in security-critical roles, suggesting that responsible AI requires moving beyond general performance metrics to domain-specific auditing.