0
$\begingroup$

I'm modeling a regression problem. An initial attempt yields the following:

labels.mean(): 0.00018132978443886167
labels.std(): 0.013450786078937208

predictions.mean(): 0.0005549060297198594
predictions.std(): 0.00430255476385355

As you can see, the mean is off, and the standard deviation is totally different. I wonder what does it indicate?

My guess: does it mean that my features are not discriminative enough, so that the model see examples w/ positive and negative labels alike, hence the small variance in the output?

I'm running the regression using XGBRegressor, with early-stopping. I have 1M training examples, 100K validation examples (for early-stopping), and another 100K for testing purpose (for which the mean and the standard deviation are shown above).

I also checked that the label distribution of the three sets are mostly basically the same.

$\endgroup$
2
$\begingroup$

The difference in standard deviation is nothing suspicious. It is only to be expected, if you have a weak correlation.

Suppose the regression is y ~ x, i.e., y = ax + b. Suppose that x only explains a small fraction of the variability in y, i.e., y is scattered all over the place and the least regression line only weakly fits the data. Suppose also that the line is nearly horizontal (i.e., a is small). Then the standard deviation of the predicted y-values will be small (since the line is nearly horizontal) but the standard deviation of the actual y-values might be large.

But really, the way to figure out what is going on is to visualize the data. You should always start by visualizing the data. Plot a scatterplot, and superimpose the least squares fit line on top of it. I bet you'll immediately have a better sense of what might be going on.

$\endgroup$
  • $\begingroup$ Marking this as an answer not because it directly answers my question (which I realize is hard to nail on the spot), but because your answer provides a good way for anyone who shares the question to approach the problem and to find out the answer by themselves. Appreciate it. $\endgroup$ – Roy May 4 '17 at 21:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.