How to choose an optimizer for your Neural Network?

Cross Validated Asked by White on January 1, 2022

My question boils down to this : Which optimizer should I use to train my neural network.

I understand this question depends on the problem.

However, for instance it seems that ADAM optimizer generally outperforms SGD. So maybe we can find some rationality in the choice of optimizers (and their learning rate) ?

Maybe it won’t be mathematically correct, but maybe some of you came accross a standard method for choosing an optimizer ?

neural networks optimization

One Answer

There is no science behind which optimiser to use in a model. It mostly boils down to heuristics and type of model in use.

ADAM is not generally better than SGD: This blog post reviews an article about SGD being a better optimiser than ADAM. https://shaoanlu.wordpress.com/2017/05/29/sgd-all-which-one-is-the-best-optimizer-dogs-vs-cats-toy-experiment/

There are pros and cons of each optimiser and no optimiser works best

https://ruder.io/optimizing-gradient-descent/ This link compares different gradient descent optimisers.

Answered by Vivek on January 1, 2022

How to choose an optimizer for your Neural Network?

One Answer

Add your own answers!

Related Questions

Error metric to compare ratios derived from a binary prediction task

Machine learning to select features to create predictive model

Summary statistics to collect for data that is too large in volume

Endogeneity testing using correlation test

find upper and lower bounds on average causal effect $theta$

How to correct for unequal observation time in poisson regression

On gradient boosting and types of encodings

RFE: Pre-define a specificity threshold

Best software for image segmentation for time-series images?

semantic value of a convolution neural network

Getting Different R-squared for Same Model Equation

How to maximize the steady state transition probability for a state in a Markov chain by altering that state’s outgoing transition probabilities?

Find out if increase in a series of values is statistically significant

Covariance between sample mean of two simple random samples

Distribution of the maximum of two correlated normal variables

How to formulate a classification problem with time series element

Asymmetric error measure for forecasts

How to model variance of a heteroskedastic dataset

What analysis do I do? (Wage, education, gender)

Is random seed a hyper-parameter to tune in training deep neural network?

Ask a Question