Linking convolutional kernel size to generalization bias in face analysis CNNs

Architecture choices matter as much as training data: While researchers typically focus on dataset bias when explaining algorithmic fairness problems, this paper demonstrates that how a neural network is designed—specifically the size of convolutional filters—can independently cause biased performance across demographic groups, even with balanced training data.

Kernel size affects what features the network learns: Smaller or larger convolutional kernels cause neural networks to focus on different types of visual information (high-frequency vs. low-frequency details). This architectural choice disproportionately impacts different demographic groups, suggesting that some groups' facial features may be better captured by certain kernel sizes than others.

Small architectural changes have outsized fairness consequences: Modifying the kernel size in even a single layer of a face recognition system significantly altered how the network generalized across racial and gender subgroups, indicating that seemingly minor technical decisions can have substantial fairness implications.

Current bias audits may be incomplete: Since most fairness testing focuses on dataset composition rather than architectural design choices, existing bias evaluations of face analysis systems may miss important sources of algorithmic discrimination that stem from how the network is structured.

Responsible AI requires examining the full pipeline: Developing fair AI systems requires scrutinizing not just what data is used for training, but also the technical design decisions embedded in the model architecture itself.