2
$\begingroup$

I have the last couple of month worked with an regression problem, of turning a framed audio file into a set of mfcc features, for a speech recognition application

I tried a lot different network structures, Cnn, different normalisation techniques, different optimizer, adding more layers and so on..

but finally i've got some decent result, but i don't understand why.. What i did was i added a linear layer as output, and somehow that minimised the error tremendeosly, and bit puzzled why a linear layer would have that much effect?...

I mean am still tried to fit the actual output to the desired output?.. Why would the activation function matter here?... I mean the weight are being adjusted based on the error, so why is the neural network better at adjusting for the error when the output is linear rather than non-linear (such as: tanh, Relu).. ?

Improve this question
$\endgroup$
1
  • $\begingroup$ As an aside - why use NN to calculate MFCC features? Calculation of MFCC is already well-defined, and it is going to be faster and more accurate to use the usual route. Only reason not to is if you cannot run the FFT etc software on your target installation where the NN needs to run. $\endgroup$ – Neil Slater Jan 26 '17 at 14:03
2
$\begingroup$

If you are performing regression, you would usually have a final layer as linear.

Most likely in your case - although you do not say - your target variable has a range outside of (-1.0, +1.0). Many standard activation functions have restricted output values. For example a sigmoid activation can only output values in range (0.0, 1.0) and a ReLU activation can only output positive values. If the target value is outside of their range, they will never be able to match it and loss values will be high.

The lesson to learn here is that it is important which activation function you use in the output layer. The function must be able to output the full range of values of the target variable.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for the response.. About the ranges is something i've also considered. tanh didn't seem to work well.. it takes a while (if ever (stopped it pretty early)) for it reach the same loss level as using relu (being the one i ended up using). What is the range for the linear activation?.. the range of the target variable is within (-100 , +100), I didn't normalize the output, only the input... $\endgroup$ – Loser Jan 26 '17 at 13:57
  • $\begingroup$ @Loser: A linear activation is not bound to a range other than limitations of floating point. In practice it might be limited by constraints on weights, so if you are using L1 or L2 regularisation on weight values you may want to make it less restrictive on the output layer. $\endgroup$ – Neil Slater Jan 26 '17 at 14:43
  • $\begingroup$ And as you hinted at, one possible solution is to map your target variable to a smaller range e.g. divide by 100, and then you could use tanh. $\endgroup$ – Neil Slater Jan 26 '17 at 14:44
0
$\begingroup$

There is a short article here regarding hidden and output activation functions.

Why use activation functions?

One of my regression models actually performs best with linear hidden and output activations. Like the article states, a linear function of linear functions is still linear, but in my experience sometimes the data is best modeled as such.

Also, MIT's Deep Learning Book has a good breakdown of the various activation functions in Chapter 6 starting on page 181.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.