Context and Symmetry in Auditing: A Case Study of Skeleton Inference in Motion Capture

Ground truth is often assumed rather than verified: The paper reveals that AI audits typically rely on "ground truth" measurements without questioning whether these reference points are actually accurate or appropriate. In motion capture systems, what counts as correct skeleton inference may depend on context and purpose rather than being objectively determinable, meaning audits could validate systems against flawed benchmarks.

Context matters for understanding AI system performance: The authors argue that auditing AI systems requires understanding the real-world practices and contexts in which they operate, not just testing them in isolation. This contextual approach can reveal hidden assumptions—for example, motion capture systems may perform differently depending on factors like clothing, body type, or movement style that standard audits might miss.

Symmetry can help audit systems when perfect accuracy is impossible: When ground truth cannot be definitively established (which is common for systems observing human behavior), the paper suggests using "symmetry"—examining whether a system's errors are consistent and predictable across similar situations. This provides a practical auditing approach when perfection isn't achievable.

Current AI auditing methods may have blind spots: Standard auditing approaches risk validating systems that work well in test conditions but fail in real-world deployment. The motion capture case study suggests auditors should explicitly question what serves as ground truth and whether audit conditions reflect actual usage.

Auditors must be transparent about their assumptions: The paper emphasizes that responsible AI auditing requires auditors to openly acknowledge what they're measuring against, why they chose those measures, and what limitations exist—rather than presenting audit results as objective facts.