3
$\begingroup$

This may seem to be a silly question. But I just wonder why the MAE doesn't reduce to values close to 0.

MAE

It's the result of an MLP with 2 hidden layers and 6 neurons per hidden layer, trying to estimate one outputvalue depending on three input values.

Why is the NN (simple feedforward and backprop, nothing special) not able to maybe even overfit and meet the desired training values?

Costfunction = $0.5 (Target - Modeloutput)^2$

EDIT:

Indeed I found an inconsistency in the inputdata.

2Inputdata

Already cheering, I was hoping to see a better result, after fixing the issue with the input data. But what I got is this:

not_fixed!

I'm using a Minibatch-SGD and now I think it might get trapped in a local minimum. I read about Levenberg-Marquardt algorithm, which is said to be stable and fast. Is it a better algorithm for global minimum detection?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ If the model is not able to overfit on the data it is likely that the model does not have enough capacity, i.e. does not have enough free parameters. Try using a larger model with more hidden layers and/or more neurons per layer. $\endgroup$
    – Oxbowerce
    Apr 16, 2021 at 8:20
  • $\begingroup$ It doesn't necessarily make sense to try to obtain perfect performance in my opinion. Of course it depends what is the task and the data, but in general I would say that the goal is to obtain the best possible performance given a realistic context for the task, and quite often a realistic context involves noisy/inconsistent data. The risk with pushing performance too far is to overfit the model, at the end it would perform very well but only for exactly the same kind of "clean" data. A robust model which doesn't perform perfectly but can handle noisy data is often more useful. $\endgroup$
    – Erwan
    Apr 16, 2021 at 22:13

1 Answer 1

Reset to default
5
$\begingroup$

There can be other reasons related to the model but the most simple explanation is that the data contains contradicting patterns: if the same features correspond to different target values, there is no way for any model to achieve perfect performance.

Let me illustrate with a toy example:

x   y
0   4
0   4
1   3
1   4
1   3
1   3
2   8
2   7
2   7 
2   6

The best a model can do with this data is to associate x=0 with y=4, x=1 with y=3 and x=2 with y=7. Applying this best model to the training set would lead to 3 errors and the MAE would be 0.3.

Improve this answer
$\endgroup$
6
  • $\begingroup$ I wouldn't call this "contradicting patterns", but simply "noise". $\endgroup$
    – Stephan Kolassa
    Apr 16, 2021 at 17:51
  • $\begingroup$ @StephanKolassa ..well, I guess that technically speaking, not all noise leads to contradictions in the data, or you could have problems where that is not considered noise. $\endgroup$
    – jjmontes
    Apr 16, 2021 at 18:13
  • $\begingroup$ @StephanKolassa I agree that most of the time this is noise, but one can imagine cases where the data is perfectly reliable but the features are simply not informative enough. For instance if one tries to predict a person's age from their height and weight I don't think I would call this "noise". $\endgroup$
    – Erwan
    Apr 16, 2021 at 22:05
  • $\begingroup$ @Erwan: to clarify, "noise" to a statistician is "residual variance" or "residual variation", i.e., any variation in the target variable that cannot be explained from predictors alone. Except for some extremely few cases in classical physics, residual variation is everywhere. I have to admit that I am a little aghast that the OP knows how to fit an MLP but apparently believes that it should fit their data perfectly, i.e., yield a zero MAE. Lack of understanding of residual variation leads to magical thinking about the possible performance of DS/AI/ML. $\endgroup$
    – Stephan Kolassa
    Apr 17, 2021 at 8:03
  • $\begingroup$ @StephanKolassa thanks for the clarification. Indeed, there can be a lot of confusion in some of the questions around here. $\endgroup$
    – Erwan
    Apr 17, 2021 at 16:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.