Bilkent University
Department of Computer Engineering

---

CS690 SEMINAR

 

SPECTRAL NORMALIZATION ON ORDINARY DIFFERENTIAL EQUATION NEURAL NETWORKS

 

Saeid Karimi
PhD Student
(Supervisor: Asst. Prof. Dr. Hamdi Dibeklioğlu)
Computer Engineering Department
Bilkent University

Ordinary differential equation neural networks (ODE nets) are new family of deep neural network models. Instead of specifying a discrete sequence of hidden layers. They parameterize the derivative of the hidden state using a neural network. The output of the network is computed using a blackbox differential equation solver. These continuous-depth models have constant memory cost and adapt their evaluation strategy to each input. They demonstrate these properties in continuous-depth residual networks and continuous-time latent variable models. In this research, we analyze the effect of normalizing methods (spectral normalization and orthogonal normalization) in Ordinary differential equation neural networks (ODEnets) for two metrics: loss and number of evaluation functions (NFEs). We found that by applying these normalization methods NFEs decrease, which is an important factor in training time of ODE-nets, however, we encountered a bit increase in the training loss of the networks. So, by decreasing the NFEs and hence, training time, the ODE nets are more suitable to be used in generative models, since ode nets are reversible networks.

 

DATE: 03 May 2021, Monday @ 15:30