Unsafe at any AUC: Unlearned Lessons from Sociotechnical Disasters for Responsible AI

Known risks are often ignored due to organizational and economic pressures: Major disasters like Chernobyl, Three Mile Island, and the Challenger disaster weren't caused by unforeseeable accidents—hazards were identified beforehand but not acted upon because of social, political, and economic factors. The same pattern threatens AI systems when organizations prioritize speed or cost over addressing known safety concerns.

Technical excellence alone is insufficient for safety: The paper argues that focusing only on building "reliably designed" AI components misses critical failure points. Safety requires analyzing the entire sociotechnical system—including how organizations use the technology, who makes decisions about it, and what pressures they face—not just the algorithm's performance metrics.

Organizations need better risk communication and perception: Historical disasters reveal that risks weren't always unknown to technical experts; they were often poorly communicated within organizations or dismissed by decision-makers. AI development requires improved systems for surfacing, discussing, and acting on identified risks at the organizational level.

Responsibility and traceability must be built into AI systems from the start: The paper emphasizes that we need clear tracking of requirements, decisions, and accountability throughout AI development and deployment. Without this, it becomes unclear who is responsible when problems emerge—a pattern that repeated across major industrial disasters.

Social and organizational dynamics must be treated as engineering priorities, not afterthoughts: Responsible AI requires integrating human factors, organizational structure, and institutional incentives into the core design process, rather than treating them as secondary concerns.