1
$\begingroup$

I might be in danger of having my question closed as "not clear what I'm asking for," but here goes.

Suppose we have a simple feedforward network. It has a few layers, each layer has a "reasonable" number of neurons, nothing complicated. Let's say the output has size $n$, and there is no final activation function on the output.

The network will have an "easier" time training to produce some outputs relative to others. In particular, outputs close to 0, that is, closer to the origin in the $\mathbb{R}^n$ output space, will be easier. But this is an intuition I have; I'm not sure if this is actually true. (By "easier" I think what I really mean is in fewer iterations.)

I haven't found a source for this, but there is a lot of advice on the internet to normalize one's data which seems to have a similar motivation.

Is there any theoretical basis for this notion of the effort a network needs to produce an output? Is it meaningful at all to talk about without making assumptions about the function we're trying to learn?

And if this idea is accurate in some way, is the "effort" independent of direction? Is it like an $n$-variate Gaussian distribution where the density of that point corresponds to the effort the network has to produce that point? Or is the "distribution" spikier, making it easier to output e.g. $[0,0,0,1]$ relative to $[\frac{1}{2},\frac{1}{2},\frac{1}{2},\frac{1}{2}]$? (Or vice versa?)

$\endgroup$

1 Answer 1

1
$\begingroup$

You asked several questions, I'll answer the one about which specific distributions are easier to learn. Information Theory would predict that $[0,0,0,1]$ would far easier to learn than $[\frac{1}{2},\frac{1}{2},\frac{1}{2},\frac{1}{2}]$.

$[\frac{1}{2},\frac{1}{2},\frac{1}{2},\frac{1}{2}]$ is a uniform discrete distribution which has maximum entropy. $[0,0,0,1]$ has far less entropy.

$\endgroup$
1
  • $\begingroup$ Interesting perspective! But I don't quite follow - how is [1/2, 1/2, 1/2, 1/2] a distribution? In my example, it's just a point, moreover the sum of the values is 2, not 1. Still, I think you have a point there. If we use softmax on the raw output to create a distribution, then there are a lot more raw points that correspond to low entropy distributions than high entropy distributions. $\endgroup$
    – Elliot Way
    Jul 2, 2021 at 20:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.