Towards Lightweight Data Integration using Multi-workflow Provenance and Data Observability

Data tracking enables responsible AI development: The paper demonstrates that by automatically monitoring data flows across complex AI systems without slowing them down (near-zero overhead), researchers can better understand how their models work and make decisions—a key requirement for developing AI responsibly and ensuring reproducibility.

Scientific collaboration requires better data visibility: Modern scientific discoveries involve multiple teams using different tools and computing systems (supercomputers, cloud services, edge devices). MIDA's approach of creating a unified view of all data moving through these systems helps researchers from different disciplines work together more effectively and catch potential issues early.

Transparency doesn't require sacrificing performance: The system successfully tracked up to 100,000 computational tasks across thousands of processor cores with minimal slowdown, showing that comprehensive data observability—essential for understanding and auditing AI systems—can be achieved without compromising the speed needed for large-scale scientific computing.

Automated monitoring reduces human error in complex workflows: By intercepting and recording data automatically "in the background," the approach eliminates the need for researchers to manually instrument their code, reducing opportunities for incomplete or incorrect documentation of how data moves through AI pipelines—a critical issue for reproducibility and accountability.