0
$\begingroup$

What kind of operation does Dense Layer perform to reduce dimemsion. So basically I have used Dense layer to compress the dimension all the time like from 10000 neurons to direct 2000 neurons or even 10 neurons for Output.

I'm not really able to understand what kind of operation does Dense layer perform to reduce the dimension from such higher number to lower number.

$\endgroup$

2 Answers 2

0
$\begingroup$

Let's stick with reducing 3 neurons to 2 neurons for simplicity (the mechanism will be the same for any number of neurons). Take the image below (taken from a StackOverflow post) as an example.

enter image description here.

Consider the transition from the second layer with 3 neurons to third layer with 2 neurons. All that happens is that the output of each of the 3 neurons of the 2nd layer is used as input to the each of the 2 neurons of the 3rd layer. To understand how this input-output mechanism works for each neuron, check this video.

$\endgroup$
2
  • $\begingroup$ so is it just the sum of previous layer neurons and there weights? $\endgroup$
    – Chris_007
    Oct 16, 2021 at 9:33
  • $\begingroup$ @Chris_007 not exactly, neurons should be considered as functions and output of the previous layer are the variables of this function. I modified the third link. That video should clarify things. $\endgroup$
    – serali
    Oct 16, 2021 at 9:54
0
$\begingroup$

A Dense layer in neural networks performs a linear operation on the layer's input vector. This operation can be summarized as a matrix multiplication followed by a bias offset.

The Dense layer takes the inputs, multiplied by the weights, and then adds the bias. This is also known as a dot product. The weights and biases are learnable parameters which the neural network adjusts during training to minimize the loss function.

The dimensionality reduction happens because the weight matrix in the Dense layer has a shape of (input_dim, output_dim). So, if you have 10000 neurons (input_dim) and you want to reduce it to 2000 neurons (output_dim), the weight matrix would have a shape of (10000, 2000). The result of the matrix multiplication would then be a vector of size 2000, effectively reducing the dimensionality of the input.

After the linear operation, the Dense layer typically applies an activation function to introduce non-linearity into the model, which allows the network to learn complex patterns.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.