10
$\begingroup$

I've been analyzing a data set of ~400k records and 9 variables The dependent variable is binary. I've fitted a logistic regression, a regression tree, a random forest, and a gradient boosted tree. All of them give virtual identical goodness of fit numbers when I validate them on another data set.

Why is this so? I'm guessing that it's because my observations to variable ratio is so high. If this is correct, at what observation to variable ratio will different models start to give different results?

$\endgroup$
7
$\begingroup$

This results means that whatever method you use, you are able to get reasonably close to the optimal decision rule (aka Bayes rule). The underlying reasons have been explained in Hastie, Tibshirani and Friedman's "Elements of Statistical Learning". They demonstrated how the different methods perform by comparing Figs. 2.1, 2.2, 2.3, 5.11 (in my first edition -- in section on multidimensional splines), 12.2, 12.3 (support vector machines), and probably some others. If you have not read that book, you need to drop everything RIGHT NOW and read it up. (I mean, it isn't worth losing your job, but it is worth missing a homework or two if you are a student.)

I don't think that observations to variable ratio is the explanation. In light of my rationale offered above, it is the relatively simple form of the boundary separating your classes in the multidimensional space that all of the methods you tried have been able to identify.

$\endgroup$
  • $\begingroup$ I'll ask my boss if I can get the company to pay for it. $\endgroup$ – JenSCDC Aug 19 '14 at 18:39
  • 1
    $\begingroup$ ESL is 'free' as a pdf from their homepage...also worth downloading is ISL (by many of same authors) - more practical www-bcf.usc.edu/~gareth/ISL $\endgroup$ – seanv507 Aug 20 '14 at 12:02
4
$\begingroup$

its worth also looking at the training errors.

basically I disagree with your analysis. if logistic regression etc are all giving the same results it would suggest that the 'best model' is a very simple one (that all models can fit equally well - eg basically linear).

So then the question might be why is the best model a simple model?: It might suggest that your variables are not very predictive. Its of course hard to analyse without knowing the data.

$\endgroup$
1
$\begingroup$

As @seanv507 suggested, the similar performance may simply be due to the data being best separated by a linear model. But in general, the statement that it is because the "observations to variable ratio is so high" is incorrect. Even as your ratio of sample size to number of variables goes to infinity, you should not expect different models to perform nearly identically, unless they all provide the same predictive bias.

$\endgroup$
  • $\begingroup$ I just edited my question to add that the dependent variable is binary. Hence, a linear model isn't suitable. $\endgroup$ – JenSCDC Aug 18 '14 at 17:36
  • $\begingroup$ " you should not expect different models to perform nearly identically, unless they all provide the same predictive bias." I used MAE and the ratio of actual to predicted outcomes as validation measures and the ratios were very close. $\endgroup$ – JenSCDC Aug 18 '14 at 17:40
  • 1
    $\begingroup$ Andy, I would include logistic regression (and linear SVM) as 'linear' model. They are all just separating the data by a weighted sum of the inputs. $\endgroup$ – seanv507 Aug 18 '14 at 17:59
  • 1
    $\begingroup$ @seanv507 Exactly - the decision boundary is still linear. The fact that binary classification is being performed doesn't change that. $\endgroup$ – bogatron Aug 18 '14 at 18:07
  • $\begingroup$ What about trees? They really don't seem linear to me. $\endgroup$ – JenSCDC Aug 19 '14 at 14:52
0
$\begingroup$

I'm guessing that it's because my observations to variable ratio is so high.

I think this explanation makes perfect sense.

If this is correct, at what observation to variable ratio will different models start to give different results?

This will probably depend very much on your specific data (for instance, even whether your nine variables are continuous, factors, ordinary or binary), as well as any tuning decisions you made while fitting your model.

But you can play around with the observation-to-variable ratio - not by increasing the number of variables, but by decreasing the number of observations. Randomly draw 100 observations, fit models and see whether different models yield different results. (I guess they will.) Do this multiple times with different samples drawn from your total number of observations. Then look at subsamples of 1,000 observations... 10,000 observations... and so forth.

$\endgroup$
  • 1
    $\begingroup$ Hm why is that? more observations seems to increase the chance that the decision boundary is more complex -- i.e. definitely not linear. And these models do different things in complex cases, and tend to do the same in simple ones. $\endgroup$ – Sean Owen Aug 20 '14 at 9:53
  • $\begingroup$ @SeanOwen: I think I'm not understanding your comment. What part of my answer does "why is that" refer to? The OP said nothing about using linear decision boundaries - after all, he might by transforming predictors in some way. $\endgroup$ – S. Kolassa - Reinstate Monica Aug 20 '14 at 10:31
  • $\begingroup$ Why would more observations make different classifiers give more similar decisions? my intuition is the opposite. Yes, I'm not thinking of just linear decision boundaries. The more complex the optimal boundary the less likely they will all fit something similar to that boundary. And the boundary tends to be more complex with more observations. $\endgroup$ – Sean Owen Aug 20 '14 at 10:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.