AI2030 Responsible AI Tools Marketplace

Key Takeaways

LLMs show promise but have significant gaps in construction safety: GPT-4o achieved 84.6% accuracy on standardized safety exams, exceeding the BCSP benchmark, but both models tested (GPT-3.5 at 73.8% and GPT-4o) demonstrated notable weaknesses in critical areas including science, mathematics, emergency response, and fire prevention—domains where accuracy is essential for worker safety.
How you ask matters, but there's no perfect approach: Prompt engineering strategies produced substantial variations in accuracy (up to 13.5% for GPT-3.5 and 7.9% for GPT-4o), meaning the way safety questions are phrased significantly affects LLM responses. However, the research found no single prompt configuration that works universally, requiring ongoing experimentation and caution.
Four specific failure modes threaten reliability: The study identified four primary reasons LLMs make mistakes in safety contexts: lack of knowledge, reasoning flaws, memory issues, and calculation errors. These aren't random failures but systematic limitations that organizations need to understand before deploying these tools in safety-critical decisions.
Strengths don't guarantee safe deployment: While both models performed well in safety management systems and hazard identification, their uneven performance across different safety domains means they cannot be trusted as standalone decision-makers. Organizations must implement human oversight, particularly in areas where the models showed weakness.
Responsible AI requires systematic evaluation before deployment: The paper emphasizes that deploying LLMs without understanding their specific capabilities and limitations risks generating inaccurate safety information and creating false confidence among workers—highlighting why rigorous testing is essential before real-world implementation.

Abstract

Construction remains one of the most hazardous sectors. Recent advancements in AI, particularly Large Language Models (LLMs), offer promising opportunities for enhancing workplace safety. However, responsible integration of LLMs requires systematic evaluation, as deploying them without understanding their capabilities and limitations risks generating inaccurate information, fostering misplaced confidence, and compromising worker safety. This study evaluates the performance of two widely used LLMs, GPT-3.5 and GPT-4o, across three standardized exams administered by the Board of Certified Safety Professionals (BCSP). Using 385 questions spanning seven safety knowledge areas, the study analyzes the models' accuracy, consistency, and reliability. Results show that both models consistently exceed the BCSP benchmark, with GPT-4o achieving an accuracy rate of 84.6% and GPT-3.5 reaching 73.8%. Both models demonstrate strengths in safety management systems and hazard identification and control, but exhibit weaknesses in science, mathematics, emergency response, and fire prevention. An error analysis identifies four primary limitations affecting LLM performance: lack of knowledge, reasoning flaws, memory issues, and calculation errors. Our study also highlights the impact of prompt engineering strategies, with variations in accuracy reaching 13.5% for GPT-3.5 and 7.9% for GPT-4o. However, no single prompt configuration proves universally effective. This research advances knowledge in three ways: by identifying areas where LLMs can support safety practices and where human oversight remains essential, by offering practical insights into improving LLM implementation through prompt engineering, and by providing evidence-based direction for future research and development. These contributions support the responsible integration of AI in construction safety management toward achieving zero injuries.

29 pages, 5 figures

Key Takeaways

LLMs show promise but have significant gaps in construction safety: GPT-4o achieved 84.6% accuracy on standardized safety exams, exceeding the BCSP benchmark, but both models tested (GPT-3.5 at 73.8% and GPT-4o) demonstrated notable weaknesses in critical areas including science, mathematics, emergency response, and fire prevention—domains where accuracy is essential for worker safety.
How you ask matters, but there's no perfect approach: Prompt engineering strategies produced substantial variations in accuracy (up to 13.5% for GPT-3.5 and 7.9% for GPT-4o), meaning the way safety questions are phrased significantly affects LLM responses. However, the research found no single prompt configuration that works universally, requiring ongoing experimentation and caution.
Four specific failure modes threaten reliability: The study identified four primary reasons LLMs make mistakes in safety contexts: lack of knowledge, reasoning flaws, memory issues, and calculation errors. These aren't random failures but systematic limitations that organizations need to understand before deploying these tools in safety-critical decisions.
Strengths don't guarantee safe deployment: While both models performed well in safety management systems and hazard identification, their uneven performance across different safety domains means they cannot be trusted as standalone decision-makers. Organizations must implement human oversight, particularly in areas where the models showed weakness.
Responsible AI requires systematic evaluation before deployment: The paper emphasizes that deploying LLMs without understanding their specific capabilities and limitations risks generating inaccurate safety information and creating false confidence among workers—highlighting why rigorous testing is essential before real-world implementation.

Abstract

29 pages, 5 figures

Responsible AI in Construction Safety: Systematic Evaluation of Large Language Models and Prompt Engineering

Key Takeaways

Abstract

Paper Info

Responsible AI Topics

Links

Responsible AI in Construction Safety: Systematic Evaluation of Large Language Models and Prompt Engineering

Key Takeaways

Abstract

Paper Info

Responsible AI Topics

Links