Timeline for Why ReLU is better than the other activation functions
Current License: CC BY-SA 3.0
14 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jun 14, 2022 at 2:42 | answer | added | Youjun Hu | timeline score: 1 | |
| Dec 27, 2017 at 10:13 | history | edited | Green Falcon |
edited tags
|
|
| S Oct 6, 2017 at 5:46 | history | suggested | Maxim |
add relevant tags
|
|
| Oct 5, 2017 at 21:26 | review | Suggested edits | |||
| S Oct 6, 2017 at 5:46 | |||||
| Oct 4, 2017 at 21:38 | history | edited | Green Falcon | CC BY-SA 3.0 |
edited title
|
| Oct 3, 2017 at 19:32 | vote | accept | Green Falcon | ||
| S Oct 3, 2017 at 18:37 | history | suggested | Maxim | CC BY-SA 3.0 |
spelling fix
|
| Oct 3, 2017 at 18:37 | answer | added | Maxim | timeline score: 32 | |
| Oct 3, 2017 at 18:10 | review | Suggested edits | |||
| S Oct 3, 2017 at 18:37 | |||||
| Oct 3, 2017 at 15:26 | comment | added | Green Falcon | yes, actually they are for somehow preventing vanishing/exploding gradients, after some iterations the outputs get larger I guess. | |
| Oct 3, 2017 at 15:15 | comment | added | noe | You are right, in not very deep CNNs it is normal not to have batch normalization. Have you considered the role of weight initial values? (e.g. He initialization) | |
| Oct 3, 2017 at 14:23 | comment | added | Green Falcon |
@ncasas But in typical CNN normalizing the output of the relu is not common? At least I have never seen that.
|
|
| Oct 3, 2017 at 14:22 | comment | added | noe | It is common to have some sort of normalization (e.g. batch normalization, layer normalization) together with ReLU. This adjusts the output range. | |
| Oct 3, 2017 at 14:17 | history | asked | Green Falcon | CC BY-SA 3.0 |