Is it necessary to drop noisy features (eg column of random numbers) from tree features? I think it's not. sometimes it may benefit but will never cause any harm to the model. Because at each split model is checking which feature will reduce the impurity. Sometimes by chance random numbers may be the one.
1 Answer
This is not a direct answer to your question, but more like an experiment. I created a simple script in Python where I ran multiple times the Iris dataset with the regular columns and also with 4 extra columns with random numbers. Then I store the difference of the accuracy between the two models and plot the distribution of it. If you try it yourself as well, you will see that there are cases that the "clean" dataset has better accuracy, even if the majority of the results are exactly the same.
from sklearn.datasets import load_iris
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
import numpy as np
import warnings
import seaborn as sns
import random as random
warnings.filterwarnings("ignore")
iris = load_iris()
df = pd.DataFrame(data= np.c_[iris['data'], iris['target']], columns= iris['feature_names'] + ['target'])
df['rand1'] = np.random.randint(0, 2, df.shape[0])
df['rand2'] = np.random.randint(0, 2, df.shape[0])
df['rand3'] = np.random.randint(0, 2, df.shape[0])
df['rand4'] = np.random.randint(0, 2, df.shape[0])
all_inputs = df[iris['feature_names']].values
all_inputs_with_random = df[iris['feature_names']+['rand1', 'rand2', 'rand3','rand4']].values
all_classes = df['target'].values
dif = []
for i in range(100):
a = random.randint(0,1000)
(train_inputs, test_inputs, train_classes, test_classes) = train_test_split(all_inputs, all_classes, train_size=0.7, random_state = a)
dtc1 = DecisionTreeClassifier()
dtc1.fit(train_inputs, train_classes)
a1 = dtc1.score(test_inputs, test_classes)
(train_inputs, test_inputs, train_classes, test_classes) = train_test_split(all_inputs_with_random, all_classes, train_size=0.7, random_state = a)
dtc2 = DecisionTreeClassifier()
dtc2.fit(train_inputs, train_classes)
a2 = dtc2.score(test_inputs, test_classes)
dif.append(a1-a2)
sns.distplot(dif)
-
$\begingroup$ Thanks for the reply. I got your point. I was wondering how can I theoretically explain this ? $\endgroup$– tamFeb 6, 2019 at 19:50
-
2$\begingroup$ It's always related to the rest of the dataset. If you have several strong correlated features, then it's quite sure that your "real" features will end up to pure tree nodes and the noised features will not be used at all. Feature engineering and selection is something that you should always do with your domain knowledge $\endgroup$– TasosFeb 7, 2019 at 6:56