I have a dataframe with a feature selection problem. I want to get the variables explaining the variance within each segment of the following dataset:

    Do you agree    Gender  Age     ...  City          Urban/Rural  Output
0   Yes             Female  25-34   ...  Madrid        Urban        Will buy
1   No              Male    18-25   ...  Fès-Meknès  Rural        Won't
2   ...             ...     ...     ...  ...      ...               Undecided

The target being Output.

I've been told a decision tree could be a way so after googling a bit I did:

# Feature Importance with Extra Trees Classifier
from sklearn.ensemble import ExtraTreesClassifier

# feature extraction
model = ExtraTreesClassifier(n_estimators=10)
model.fit(X, y)

Which returns:

[0.         0.00473011 0.00716472 0.00778101 0.0051573  0.00139121
 0.02045262 0.00791912 0.         0.00222593 0.00173901 0.00417362
 0.00222593 0.00313295 0.00565095 0.00652543 0.         0.00527774
 0.         0.         0.00601354 0.         0.         0. ...

This looks exactly what I am looking for, but I don't get how we obtained this output. It look a bit like a black box. I am eager to learn and to do it myself to understand.Can you explain to me or refer me some resoures that would help me implement it myself?

Annex: data preparation

def load_dataset():
    connection = psycopg2.connect(user = "user",
                                  password = "password",
                                  host = "host",
                                  port = "5432",
                                  database = "database")
#     connection.set_client_encoding('UNICODE')
    sql = "select * from capi limit 10;"
    # load the table
    df = pd.read_sql_query(sql, connection)
    # retrieve numpy array
    dataset = df.values

    # split into input (X) and output (y) variables
    filtered_cols = ['Output']
    cols = [col for col in cols if col not in filtered_cols]

    X = df.loc[:, cols]  #independent columns
    X = X.astype(str)
    y = df['Output']    #target column i.e price range
    return X.values, y.values

X,y = load_dataset()
  • $\begingroup$ Feature importance != feature selection. All that feature importance tells you is what RF thinks the most important features were in the context of that model. In addition, random forests (used to?) might still have the property where there is a bias towards continuous over categorical variables leading to misleading splits. $\endgroup$
    – Victor Ng
    Feb 5, 2020 at 18:07


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.