How to decide if gradients are vanishing?
Data Science Asked on December 18, 2020
I am trying to debug a neural network. I am seeing gradients close to zero. How can I decide whether these gradients are vanishing or not? Is there some threshold to decide on vanishing gradient by looking at the values? I am getting values close to 4 decimal places(0.0001) and in some cases close to 5 decimal places (0.00001). The network seems not to be learning since the histogram of weight is also quite similar in all epochs. I am using RELU activation and Adam optimizer. What could be the reason for the vanishing gradient in case of RELU activation? Is possible please point me to some resources that might be helpful. Thanks in advance.
One Answer
Vanishing gradients are the updates to the weights (not the weight themselves) over the layers after each training batch.
You should check the update signal between the top-most layer and lowest layer after a single batch.
Answered by Brian Spiering on December 18, 2020
Add your own answers!
Ask a Question
Get help from others!
Recent Questions
- How can I transform graph image into a tikzpicture LaTeX code?
- How Do I Get The Ifruit App Off Of Gta 5 / Grand Theft Auto 5
- Iv’e designed a space elevator using a series of lasers. do you know anybody i could submit the designs too that could manufacture the concept and put it to use
- Need help finding a book. Female OP protagonist, magic
- Why is the WWF pending games (“Your turn”) area replaced w/ a column of “Bonus & Reward”gift boxes?
Recent Answers
- haakon.io on Why fry rice before boiling?
- Jon Church on Why fry rice before boiling?
- Peter Machado on Why fry rice before boiling?
- Joshua Engel on Why fry rice before boiling?
- Lex on Does Google Analytics track 404 page responses as valid page views?