-1
$\begingroup$

I found the following very surprising. I trained different machine learning classifiers on a data set that includes $15$ attributes using $k$-fold cross validation. Then test on a different validation data set. The result were fine, accuracy was about $81\%$ and the f1-measure was about $83\%$.

Out of curiosity, I tried to train and test on one single attribute, and I found that there is one attribute that outperform the previous result, accuracy was about $85\%$ and the f1-measure was about $86\%$.

This results of the one attribute was always better than all attribute when using different binary classifiers.

How common is this? Is there any interpretations for the reasons behind this?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Sounds like a multicollinearity issue. Try using some strong regularization on the full model. You might get results that are similar to the one var case. But I would just run with a polynomial of that one feature and see what happens to your prediction. $\endgroup$
    – Ryan
    May 17, 2017 at 17:09

2 Answers 2

Reset to default
1
$\begingroup$

If all the attributes are highly correlated with the best attribute that you down-selected to, I guess that could happen as having too many correlated variables is known to cause problems in certain cases. The model might have settled on a local maximum and you happened to know the global maximum.

Improve this answer
$\endgroup$
1
$\begingroup$

What different classfiers did you use? Because rf and gbm are pretty robust to multicollinearity(Not always tho!) . Also it may be because that one variable has much more predictive power than the others(e.g Others might have less variance in them or highly imbalanced classes in cases of factor featues). Did you check variable importance using information gain or chi squared tests? Do these and check if that one variable you used is much better scored than the rest. That may be a reason.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.