How to choose an optimizer for your Neural Network?

Cross Validated Asked by White on January 1, 2022

My question boils down to this : Which optimizer should I use to train my neural network.

I understand this question depends on the problem.

However, for instance it seems that ADAM optimizer generally outperforms SGD. So maybe we can find some rationality in the choice of optimizers (and their learning rate) ?

Maybe it won’t be mathematically correct, but maybe some of you came accross a standard method for choosing an optimizer ?

neural networks optimization

One Answer

There is no science behind which optimiser to use in a model. It mostly boils down to heuristics and type of model in use.

ADAM is not generally better than SGD: This blog post reviews an article about SGD being a better optimiser than ADAM. https://shaoanlu.wordpress.com/2017/05/29/sgd-all-which-one-is-the-best-optimizer-dogs-vs-cats-toy-experiment/

There are pros and cons of each optimiser and no optimiser works best

https://ruder.io/optimizing-gradient-descent/ This link compares different gradient descent optimisers.

Answered by Vivek on January 1, 2022

How to choose an optimizer for your Neural Network?

One Answer

Add your own answers!

Related Questions

Why does a class weight fraction improve precision compared to undersampling approach where precision drops?

Is it possible to implement an activation function or layer in Keras that uses two distinct sets of weights?

Expert Knowledge Acquisition and Machine learning

Improving F1 scores using models with good precision and recall

Is ROCR applied to training data or testing data?

Is the plot “White noise”?

Multilabel Tweet Classification

Propagating uncertainty through nested random forest models

Identify a contaminated distribution

How do we generate the samples of hidden root nodes in the Bayes network (Sigmoid Belief Networks) of a generative model

Which monotone transformations give a very loose confidence interval in transformed space?

Restricted standard deviation of survival time

What does the term episode mean in meta-learning?

Classification technique to classify categories in two variables when dateset has larger number of numerical variables and few data points

difference partial dependence and feature weights

Interpreting HRs from stratified cox survival analysis in R

Compare two datasets and whether they agree

Handling daily time series data for better accuracy

Model tuning in the presence of incorrect training labels

Term for the error in machine learning as a direct result of incorrectly labelled data?

Ask a Question