0
$\begingroup$

I have a neural net that's generating an average 15% error across the three outputs it gives. My problem is two of the three really have about a 2% error while the third has around 40%.

I was wondering if anyone has used results from a correlation matrix on the raw data to set starting parameters for what features are valued more relative to the labels. If this is possible, or logical, how would you initialize this for two input features if your model's first layer has say eight nodes - or at least more than two? I'm using Keras btw, but even a theoretical explanation would be helpful.

If it doesn't make sense to follow this path, does anyone have recommendations at correctly the imbalance?

|improve this question
$\endgroup$
0
$\begingroup$

The way I see it is that NN can learn nonlinear features, so I wonder: does your correlation matrix on the raw data take into account nonlinear connections between features to see the full picture? Probably, it doesn't. As the topic is not about the single-layer nn (which would be an example of linear regression), the final connections (values of each weight) will be created by the contribution of many layers. Therefor it seems to me, that setting start weights on the proposed way wouldn't be high relevant.

Speaking about imbalancing, I'd recommend to experiment with loss functions. You can set higher penalty for mistakes on a certain class.

|improve this answer
$\endgroup$
  • $\begingroup$ This is good advice, makes sense about non-linear connections in the correlation matrix. I'm going to start looking into the penalizing loss functions, this is an interesting idea. Do you know if the Generalized Dice, or any others, are able to work with both regression or classification problems? I'd imagine not $\endgroup$ – 55thSwiss Aug 24 '19 at 16:50
  • $\begingroup$ @55thSwiss just an example gist.github.com/wassname/ce364fddfc8a025bfab4348cf5de852d You can find some implementation of weighted categorical cross entropy with Keras library. Generalised dice is for two classes problem, I'll delete it from the answer) $\endgroup$ – Lana Aug 24 '19 at 16:56
  • $\begingroup$ that's great, I am dealing with a regression problem, I will look thru this link you sent $\endgroup$ – 55thSwiss Aug 24 '19 at 17:08
  • $\begingroup$ You initial advice was good, although that link was for a categorical loss function $\endgroup$ – 55thSwiss Aug 25 '19 at 0:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.