On Scale-Invariant Neural Networks and Ideal Gases.

In this seminar, we will revisit the high-school thermodynamics curriculum and discuss the model of an ideal gas. Surprisingly, ideal gases have a rather precise analogue in the world of stochastic optimization: scale-invariant (SI) neural networks.

Unlike previous work exploring connections between thermodynamics and optimization, which has focused primarily on the notion of temperature (T), we will show that SI networks admit natural definitions of pressure (p) and volume (V), and even give rise to an analogue of the ideal gas law, (pV/T = \mathrm{const}) (under certain assumptions on the covariance matrix of stochastic gradient noise). This analogy provides a deeper understanding of how training hyperparameters, such as the learning rate and weight decay, affect the stationary distribution of stochastic gradient descent.

Above all, this talk is about mathematical beauty: discovering essentially the same structure in two seemingly unrelated worlds—thermodynamics and optimization.