Timeline for How is weight matrix calculated in a neural network?
Current License: CC BY-SA 4.0
7 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jan 28, 2023 at 23:53 | comment | added | Robert Melikyan | (3) Initial weight settings is a huge subject on its own, but basically people played around with setting it to particular values, but then found this biased results (ie setting to 0), and found the best to be a form of random initialization, the standard being Xavier Random Initialization for sigmoid activations and He Random Init for ReLU activations. So this too, is also taken care for you usually by the framework. (4) You're correct, but the loss function can be just about anything you define to be honest, and needn't involve a target. In the classical sense though, it's like you say. | |
| Jan 28, 2023 at 23:49 | comment | added | Robert Melikyan | (2) When defining your neural network, typically an API will take care of the layer creation. If you are really interested I recommend the site Papers With Code, FastAI, and NeuroMatch to get a more solid introductory grasp. Essentially you can define a neuron as a code object, then define a layer as a group of these objects with inputs and outputs. So really this is done programmatically. However you may want to investigate a layer of neurons, which can be done using whatever framework you like (ie PyTorch/Tensorflow) to get the weights of the neurons. This is useful in vision like CNNs. | |
| Jan 28, 2023 at 23:43 | comment | added | Robert Melikyan | @TerezaTizkova clarifying here: (1) the activation function isn't to have an output of better form, but rather to allow Non-Linearities to occur, otherwise if you did NOT have activations, you would just have a chain of linear functions, which by definition would be linear. Usually people focus on the last layer/neuron's activation function as that is what can deliver an output suitable for a given regression/classification task. I will answer the rest in another comment as I have ran out of characters. | |
| Jan 22, 2023 at 18:37 | comment | added | Tereza Tizkova | This is perfect answer! I really appreciate your time explaining that. Makes sense now. Could you please clarify few things? 1) Activation function serves for having output in better form? 2) How do we tell apart layers (Which neuron is in which)? How do they usually differ? 3) How do we initially set weights and biases? Do we say e.g. "by default each weight is 1 and bias 0" and then adjust? 4) The loss function gives me the difference of target vs output, correct? | |
| Jan 22, 2023 at 10:32 | vote | accept | Tereza Tizkova | ||
| S Jan 16, 2023 at 19:07 | review | First answers | |||
| Jan 16, 2023 at 21:28 | |||||
| S Jan 16, 2023 at 19:07 | history | answered | Robert Melikyan | CC BY-SA 4.0 |