6
$\begingroup$

Is it necessary to drop noisy features (eg column of random numbers) from tree features? I think it's not. sometimes it may benefit but will never cause any harm to the model. Because at each split model is checking which feature will reduce the impurity. Sometimes by chance random numbers may be the one.

$\endgroup$

1 Answer 1

3
$\begingroup$

This is not a direct answer to your question, but more like an experiment. I created a simple script in Python where I ran multiple times the Iris dataset with the regular columns and also with 4 extra columns with random numbers. Then I store the difference of the accuracy between the two models and plot the distribution of it. If you try it yourself as well, you will see that there are cases that the "clean" dataset has better accuracy, even if the majority of the results are exactly the same.

from sklearn.datasets import load_iris
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
import numpy as np
import warnings
import seaborn as sns
import random as random

warnings.filterwarnings("ignore")

iris = load_iris()

df = pd.DataFrame(data= np.c_[iris['data'], iris['target']], columns= iris['feature_names'] + ['target'])

df['rand1'] = np.random.randint(0, 2, df.shape[0])
df['rand2'] = np.random.randint(0, 2, df.shape[0])
df['rand3'] = np.random.randint(0, 2, df.shape[0])
df['rand4'] = np.random.randint(0, 2, df.shape[0])

all_inputs = df[iris['feature_names']].values
all_inputs_with_random = df[iris['feature_names']+['rand1', 'rand2', 'rand3','rand4']].values
all_classes = df['target'].values

dif = []

for i in range(100):
    a = random.randint(0,1000)

    (train_inputs, test_inputs, train_classes, test_classes) = train_test_split(all_inputs, all_classes, train_size=0.7, random_state = a)

    dtc1 = DecisionTreeClassifier()
    dtc1.fit(train_inputs, train_classes)

    a1 = dtc1.score(test_inputs, test_classes)

    (train_inputs, test_inputs, train_classes, test_classes) = train_test_split(all_inputs_with_random, all_classes, train_size=0.7, random_state = a)

    dtc2 = DecisionTreeClassifier()
    dtc2.fit(train_inputs, train_classes)

    a2 = dtc2.score(test_inputs, test_classes)

    dif.append(a1-a2)


sns.distplot(dif)

Difference in accuracy between clean and random-included datasets

$\endgroup$
2
  • $\begingroup$ Thanks for the reply. I got your point. I was wondering how can I theoretically explain this ? $\endgroup$
    – tam
    Feb 6, 2019 at 19:50
  • 2
    $\begingroup$ It's always related to the rest of the dataset. If you have several strong correlated features, then it's quite sure that your "real" features will end up to pure tree nodes and the noised features will not be used at all. Feature engineering and selection is something that you should always do with your domain knowledge $\endgroup$
    – Tasos
    Feb 7, 2019 at 6:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.