3
$\begingroup$

I'm new to datascience. I was wondering how one should treat an extremely dominant feature.

For example, one of the features is "on"/"off", and when it's "off", none of the other features matter and the output will just always be 0. So should I drop all rows where it's "off" in my train/test data sets? I feel like I would get a better fit that way.

If I delete those rows, I'm concerned about how I would handle those rows in the test set. For example, I'd have to write code to loop through the data and put a 0 in the prediction column for those rows, as well as make sure everything else lines up. (This is all Kaggle related, so the training set is several columns of features and a y_column, whereas the test set doesn't have the y_column and we're supposed to predict it.)

I'm using Python and Scikit Learn's random forest, if that matters.

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Are you using the RandomForest model? $\endgroup$
    – Dawny33
    Dec 14, 2015 at 11:18
  • $\begingroup$ Yes! I am using RandomForest. But I would wonder the same thing for other algorithms, like GBRT. Sorry for the late reply, was away for a few weeks. Thanks! $\endgroup$
    – gunit
    Jan 2, 2016 at 2:56

4 Answers 4

Reset to default
3
$\begingroup$

Actually, it shouldn't really matter what classification algorithm you use. The whole point of machine learning is that the algorithm learns how to combine the available features to achieve the desired result. If one feature has the ability to 'turn the others off,' the algorithm will learn that (It'll also learn lots of things that you probably aren't aware of).

So in short, no, modifying the data this way probably won't affect classification performance. Not needing to incorporate these kinds of things into the training set is part of what makes machine learning so cool!

Improve this answer
$\endgroup$
4
$\begingroup$

Rather than discarding the dominant feature (which will discard information), try reducing the number of features randomly selected when making each partition. In scikit's syntax this is max_features (mtry in R's randomForest). By default this is set to compare the square root of all features (http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html), instead try much smaller (perhaps even 2 aka decision stump). You'll probably also need many more trees vs. higher settings. This will allow you to extract information from more subtle features without losing all that great information provided by the dominant feature.

Improve this answer
$\endgroup$
2
$\begingroup$

If you are using RandomForest, then I am sure it will pick up this rule given that $\text{off} \rightarrow 0$ provides complete node purity. However, your intuition is correct, it is not uncommon to preprocess data. You can think of this rule as a single-level decision tree or decision stump, (aka 1-rule algorithm). Basically, you would remove these records from test and training to reduce noise. During classification you would also preprocess, if the input vector matches the rule then classify as 0 otherwise classify input vector with your model.

Improve this answer
$\endgroup$
2
$\begingroup$

You seem to be using the Random Forest model.

I don't see how that feature would influence the model. It actually doesn't make a difference, as random forest divides the sample space iteratively, and your sample space would be divided as switch = 0 and switch = 1.

So, the presence of those sample points do not affect the model.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.