An Empirical Study of Symmetry-Induced Parameter Sharing as an Inductive Bias in Neural Networks

authors: JuHwan Kim
affiliations: Seoul Science High School
abstract: This study empirically investigates whether parameter sharing based on task-inherent symmetry can serve as an inductive bias in neural networks. Although standard Multi-Layer Perceptrons (MLPs) are highly expressive, they are also prone to learning unnecessary dependencies on input order or structure. To address this issue, prior work has typically relied on data augmentation or structurally constrained architectures to improve generalization. In this work, we propose a method that restricts the function space by enforcing weight sharing according to task symmetry, thereby inducing a struc- tural inductive bias. To evaluate the effectiveness of this approach, we compare standard MLPs, data augmentation-based methods, and symmetry-aware parameter-sharing models on three toy tasks, an- alyzing their performance in terms of symmetric robustness, sample efficiency, parameter efficiency, and generalization. Experimental results show that models with symmetry-based parameter sharing achieve more stable performance and improved generalization, particularly in settings with limited data and restricted model capacity. These findings suggest that parameter sharing is not merely a model compression technique, but can also function as a structural inductive bias that promotes generalization in neural networks.
keywords: symmetry; parameter sharing; inductive bias; robustness; efficiency; neural networks

중간제출 기간에 맞추어 급박하게 완성한거라 Discussion과 Conclusion & Recommendation이 상당히 빈약합니다. Preprint 수준이라 생각하시면 좋을 것 같고 추후 7월 중으로 수정이 이루어질 듯합니다.