Use of localized gating in mixture of experts networks
Viswanath Ramamurti, Joydeep Ghosh · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · 1998
The 'mixture-of-experts (MOE)' is a popular architecture for function approximation. In the standard architecture, each expert is gated via a softmax function, and its domain of application is not very localized. This paper summarizes several recent results showing the advantages of using localized gating instead. These include a natural framework for model selection/adaptation by growing and shrinking the number of experts, modeling of non-stationary environments, improving the generalization performance and obtaining confidence intervals of network outputs. These results substantially increase the scope and power of MOE networks. Several simulation results are presented to support the theoretical arguments.