Timeline for Can the vanishing gradient problem be solved by multiplying the input of tanh with a coefficient?
Current License: CC BY-SA 4.0
18 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Oct 7, 2019 at 11:05 | comment | added | Neil Slater | Not really, but search for "1.7159 tanh" and you will see it is a commonly used variant | |
| Oct 7, 2019 at 10:49 | comment | added | zephyr | @NeilSlater - any references you would suggest looking at? | |
| Oct 7, 2019 at 7:50 | comment | added | Neil Slater | There are quite a few resources that suggest using $1.7159 \text{tanh}(\frac{2}{3}x)$ . . . I have always assumed that was a similar idea to batch normalisation - about creating closer to ideal distribution of features in hidden layers, including the effect on gradients. | |
| S May 17, 2019 at 7:35 | history | bounty ended | zephyr | ||
| S May 17, 2019 at 7:35 | history | notice removed | zephyr | ||
| May 15, 2019 at 8:42 | vote | accept | zephyr | ||
| May 15, 2019 at 9:34 | |||||
| May 14, 2019 at 16:30 | history | edited | Juan Esteban de la Calle | CC BY-SA 4.0 |
added 1 character in body
|
| May 14, 2019 at 11:01 | vote | accept | zephyr | ||
| May 15, 2019 at 8:07 | |||||
| May 14, 2019 at 11:01 | answer | added | zephyr | timeline score: 0 | |
| May 11, 2019 at 4:47 | comment | added | Tuyen | Well, if you use mini-batches - as common now - in training DNN, then probability/statistics/randomness will enter definitely. One easy-to-understand source for this is Nielsen's book Neural Networks and Deep Learning. How about you run several experiments to verify this yourself and report what you found here after? | |
| May 10, 2019 at 19:07 | answer | added | Esmailian | timeline score: 11 | |
| S May 10, 2019 at 11:13 | history | bounty started | zephyr | ||
| S May 10, 2019 at 11:13 | history | notice added | zephyr | Authoritative reference needed | |
| May 8, 2019 at 4:02 | answer | added | Tuyen | timeline score: 1 | |
| May 7, 2019 at 13:43 | history | edited | Green Falcon | CC BY-SA 4.0 |
edited body; edited tags; edited title
|
| May 7, 2019 at 13:40 | answer | added | Green Falcon | timeline score: 4 | |
| May 7, 2019 at 13:10 | review | First posts | |||
| May 7, 2019 at 18:57 | |||||
| May 7, 2019 at 13:07 | history | asked | zephyr | CC BY-SA 4.0 |