Parameterized Argumentation-based Reasoning Tasks for Benchmarking Generative Language Models

Current AI models are not ready for legal applications: State-of-the-art generative language models "often fail" at reasoning tasks based on witness testimony, even at low complexity levels, making them unsuitable for responsible deployment in law and evidence domains where accuracy is critical.

AI reasoning is brittle and inconsistent: The models demonstrated "inconsistent performance" and made "obvious mistakes" on argumentation tasks, indicating their reasoning capabilities lack the reliability needed for high-stakes decision-making in the justice system.

Complexity reveals hidden weaknesses: Even specialized reasoning models failed when tested on higher-complexity argument structures, suggesting that current benchmarks may not adequately stress-test AI systems before they're used in real-world legal contexts.

We need better evaluation tools: The paper demonstrates that dynamically-generated, complexity-scaled benchmarks can effectively expose reasoning limitations in language models, providing a methodology for more rigorous testing before deployment in sensitive domains.

Transparency about AI limitations is essential: By formally documenting where and why these models fail at legal reasoning tasks, the research supports responsible AI adoption by preventing overconfidence in systems that cannot yet reliably handle the nuanced argumentation required in law and justice.