1
$\begingroup$

I am struggling to understand why I am getting such a high loss/val_loss rate on my training. I am training a regression network. I've normalized the input data to range between -1 to 1, and left the output data unaltered, its range is approximately between -100 and 100.

I chose to normalize the input so that I could use tanh as the activation function since it outputs within this range.

enter image description here

enter image description here

The neural network consists of 3 layers.

  print "Model definition!"
    model = Sequential()

    #act = PReLU(init='normal', weights=None)
    model.add(Dense(output_dim=400,input_dim=400, init="normal",activation=K.tanh))

    #act1 = PReLU(init='normal', weights=None)
    model.add(Dense(output_dim=400,input_dim=400, init="normal",activation=K.tanh))

    #act2 = PReLU(init='normal', weights=None)
    #model.add(Dense(output_dim=400, input_dim=400, init="normal",activation=K.tanh))

    act4=ELU(100)
    model.add(Dense(output_dim=13, input_dim=400, init="normal",activation=act4))

The mapping between the input and output consists of mapping audio samples into MFCC features. The samples are the ones i've normalized to the aforementioned range.

Why am I getting these results?

Am I doing anything that is unclear?

Normalizing the output_range +-1:

enter image description here Why is the val_loss loss lower than the loss?

Improve this question
$\endgroup$
4
  • $\begingroup$ As I said in comments to previous question, you are taking on a tough challenge. Your title question "Why is my loss so high?" is a very broad question, and there are loads of things to investigate for your project. As you are exploring an unusual prediction goal, it is doubtful that anyone can give you better than guesswork or pointers to things to try. $\endgroup$
    – Neil Slater
    Commented Nov 27, 2016 at 13:08
  • 1
    $\begingroup$ As Neil said it's hard to say what could be wrong, there's no obvious mistake in what you've done. My suggestion is to try a higher initial learning rate, this can help when you have so many parameters. Worst case scenario is that the learning rate is too high and your LR scheduling will kick in to reduce it. $\endgroup$
    – Hugh
    Commented Nov 27, 2016 at 18:16
  • $\begingroup$ @Hugh Great idea.. I guess i will try it out. I tried normalizing the output aswell.. The dataset i had ranged between +-100.. So i scaled it down to +-1 and used a tanh as a activation function for the output layer, which shrinked the loss to around 0.01.. $\endgroup$
    – Carlton Banks
    Commented Nov 28, 2016 at 8:24
  • $\begingroup$ why create the input layer multiple times? It should be created only once. what is your training set and I will show you how to setup a dense network $\endgroup$
    – Golden Lion
    Commented Aug 19, 2022 at 20:47

1 Answer 1

Reset to default
1
$\begingroup$

As your two training curves show, neural networks are sensitive to a range of both features and targets. Normalizing both features and targets can facilitate learning.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.