Sacred or Synthetic? Evaluating LLM Reliability and Abstention for Religious Questions

LLMs struggle with religious expertise and show significant accuracy gaps: The study found that Large Language Models perform inconsistently when answering Islamic jurisprudence questions, with notable variation across different models, suggesting these systems lack reliable religious knowledge comparable to human scholars.

"Knowing when not to answer" is critical for responsible AI in religious contexts: While GPT-4o achieved the highest accuracy, models like Gemini and Fanar demonstrated superior "abstention behavior"—the ability to recognize uncertainty and decline to answer—which the researchers identified as crucial for preventing confidently incorrect religious rulings that could mislead users.

Language bias creates disparities in religious AI services: All tested models showed significant performance drops when answering questions in Arabic compared to English, meaning Muslims who primarily speak Arabic may receive less reliable religious guidance from these systems, raising equity concerns for global religious communities.

Religious nuance requires specialized evaluation: The benchmark revealed that LLMs often fail to distinguish between the four major schools of Islamic thought, each with different rulings on the same questions. This finding demonstrates that generic AI evaluation is insufficient—religious applications need domain-specific testing that accounts for theological complexity and legitimate diversity of opinion.

Current LLMs are not ready for independent religious decision-making: The researchers explicitly recommend "cautious deployment" of LLMs in religious applications, suggesting these tools should not replace qualified religious scholars and may require human oversight before providing guidance to believers.