Optimization Methods in Deep Learning

A review of optimization methods in DL

Image credit: Google

Content

This post is to sort out the optimization methods in deep learning from the most basic idea about what is gradient descent to the most popular optimizer - Adam. Moreoever, the SGD, SGDM, Adagrad, PMSProp and some recent advances such as AdamW are introduced. Hope this post could give me some hints about how to choose different optimizers to train the neural network.

Reference: Optimization slides, Tutorial.

Trulli
Page 1
Trulli
Page 2
Trulli
Page 3
Trulli
Page 4
Trulli
Page 5
Trulli
Page 6
Avatar
Xiaohong Liu
Ph.D. Candidate

My research interests include image and video processing, computer vision, machine learning and deep learning.