Symmetry breaking in neural network optimization: insights from input dimension expansion
Junjie Zhang, Nan Cheng, Fupeng Li, Xiucheng Wang, Jiannan Chen, Long-Gang Pang, Deyu Meng · npj Artificial Intelligence · 2025
Understanding how neural networks learn and optimize remains a central point in machine learning, with implications for designing better models. While techniques like dropout and batch normalization are widely used, the underlying principles driving their success-such as symmetry breaking, a concept in physics-are underexplored. We propose the symmetry breaking hypothesis, showing that breaking symmetries during training (e.g., via input expansion) substantially improves performance across tasks. We develop a metric to quantify symmetry breaking in networks, revealing its role in common optimization methods and its connection to properties like equivariance. This metric offers a practical tool to evaluate architectures without exhaustive training or full datasets, enabling more efficient design choices. Our work positions symmetry breaking as a unifying principle behind optimization techniques, bridging theoretical gaps and providing actionable insights for improving model efficiency.