Model LineUpper: Supporting Interactive Model Comparison at Multiple Levels for AutoML

Model selection requires more than performance metrics: Current AutoML systems only show performance numbers when users choose between candidate models, but the research found that practitioners actually evaluate models using multiple criteria, including how predictions are made and what types of errors occur—suggesting that AutoML tools need to provide richer information for informed decision-making.

Multi-level comparison is essential for understanding model differences: The paper identifies that users need to compare models at various levels of detail, from broad error patterns down to how individual predictions are generated, indicating that a one-size-fits-all comparison view is insufficient for responsible model selection.

Explainable AI techniques can support better model choices: By integrating explainability and visualization methods into the model comparison process, the research demonstrates that data scientists can make more informed decisions about which AutoML-generated model to deploy, potentially leading to more trustworthy and appropriate model selections.

Data scientists have unique needs in AutoML workflows: The user study revealed that the final step of AutoML—human selection from dozens of candidates—requires specialized support tailored to how practitioners actually think about and evaluate models, rather than treating it as a simple ranking problem.