Flat Channels to Infinity in Neural Loss Landscapes

PAPER 📝 We discover channels of slowly decreasing loss in network loss landscapes that lead to minima at infinite parameter norm. In the limit, these solutions implement Gated Linear Units using standard neurons. These channels are parallel to lines of saddle points generated by permutation symmetries. Read the paper: Flat Channels to Infinity in Neural Loss Landscapes.