Year-long AI Surveillance Pilot in Two South Australian Aged Care Facilities Reportedly Overwhelmed Staff with False Positives
Description
Between March 2021 and March 2022, an AI-enabled video and audio monitoring system was trialed in two South Australian aged care facilities. According to an independent audit commissioned by South Australia Health, the system produced over 12,000 false alerts, overwhelming staff and contributing to at least one missed real incident. The report concluded the system did not reach accuracy levels acceptable to staff or management during the pilot.
Actors
Alleged Deployers
- south-australia-health
Alleged Developers
- sturdie-trade-services
Harmed Parties
- residents-and-staff-of-northgate-house
- residents-and-staff-of-mount-pleasant-district-hospital
- northgate-house
- mount-pleasant-district-hospital
Classification
Risk Domain
AI Safety & FailuresRisk Subdomain
7.3. Lack of capability or robustness
MIT Risk Profile
Topics
Source
Related Incidents (AI Safety & Failures)
View all →Research Papers on This Topic
View all →Concrete Problems in AI Safety
A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI
Representation Engineering: A Top-Down Approach to AI Transparency
Explainable artificial intelligence (XAI) in deep learning-based medical image analysis
Mitigation Tools
Tools related to this risk domain, via shared responsible AI topics.