# Using random forest for selecting variables returns the entire dataframe

I am in the process of dimensionality reduction. I am using Random Forest to find the columns with the highest level of correlation with the target SalePrice column.

The problem is that the output is too large. Definitely not what I want from it. It is returning 259 columns. Some of these columns are a result of one-hot-encoding the categorical variables and adding them back into the dataframe, which logically increases the dimension of the dataset. However, I only wanted to return the columns with the highest correlation to the target variable 'SalePrice'. Not the whole damn dataframe.

Here is the output:

       0   1     2      3     4    5    6    ... 252 253 254 255 256 257 258
0        1  RL  65.0   8450  Pave  NaN  Reg  ...   0   1   0   0   1   0   1
1        2  RL  80.0   9600  Pave  NaN  Reg  ...   0   1   0   0   1   0   1
2        3  RL  68.0  11250  Pave  NaN  IR1  ...   0   1   0   0   1   0   1
3        4  RL  60.0   9550  Pave  NaN  IR1  ...   0   0   0   0   1   0   1
4        5  RL  84.0  14260  Pave  NaN  IR1  ...   0   1   0   0   1   0   1
...    ...  ..   ...    ...   ...  ...  ...  ...  ..  ..  ..  ..  ..  ..  ..
1455  1456  RL  62.0   7917  Pave  NaN  Reg  ...   0   1   0   0   1   0   1
1456  1457  RL  85.0  13175  Pave  NaN  Reg  ...   0   1   0   0   1   0   1
1457  1458  RL  66.0   9042  Pave  NaN  Reg  ...   0   1   0   0   1   0   1
1458  1459  RL  68.0   9717  Pave  NaN  Reg  ...   0   1   0   0   1   0   1
1459  1460  RL  75.0   9937  Pave  NaN  Reg  ...   0   1   0   0   1   0   1

[1460 rows x 259 columns]


Here is my code:

import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_selection import SelectFromModel
from sklearn.model_selection import train_test_split

categorical_columns = ['MSSubClass', 'MSZoning', 'LotShape', 'LandContour', 'LotConfig', 'Neighborhood', 'Condition1',
'Condition2', 'BldgType', 'HouseStyle', 'RoofStyle', 'RoofMatl', 'Exterior1st', 'Exterior2nd',
'Foundation', 'Heating', 'Electrical', 'Functional', 'GarageType', 'PavedDrive', 'Fence',
'MiscFeature', 'SaleType', 'SaleCondition', 'Street', 'CentralAir']

ranked_columns = ['Utilities', 'LandSlope', 'ExterQual', 'ExterCond', 'BsmtQual', 'BsmtCond', 'BsmtExposure',
'BsmtFinType1', 'BsmtFinType2', 'HeatingQC', 'KitchenQual', 'FireplaceQu', 'GarageQual', 'GarageCond',
'PoolQC', 'OverallQual', 'OverallCond']

numerical_columns = ['LotArea', 'LotFrontage', 'YearBuilt', 'YearRemodAdd', 'MasVnrArea', 'BsmtFinSF1', 'BsmtFinSF2',
'BsmtUnfSF','TotalBsmtSF', '1stFlrSF', '2ndFlrSf', 'LowQualFinSF', 'GrLivArea', 'BsmtFullBath',
'BsmtHalfBath', 'FullBath', 'HalfBath', 'Bedroom', 'Kitchen', 'TotRmsAbvGrd', 'Fireplaces',
'GarageYrBlt', 'GarageCars', 'GarageArea', 'WoodDeckSF', 'OpenPorchSF', 'EnclosedPorch',
'3SsnPorch', 'ScreenPorch', 'PoolArea', 'MiscVal', 'MoSold', 'YrSold']

def feature_encoding(df, categorical_list):

# take one-hot encoding
OHE_sdf = pd.get_dummies(df[categorical_list])

# drop the old categorical column from original df
df.drop(columns = categorical_list, inplace = True)

# attach one-hot encoded columns to original dataframe
df = pd.concat([df, OHE_sdf], axis = 1, ignore_index = True)

# Integer Encoding
df['Utilities'] = df['Utilities'].replace(['AllPub', 'NoSeWa'], [2, 1])  # Utilities
df['ExterQual'] = df['ExterQual'].replace(['Ex', 'Gd', 'TA', 'Fa'], [4, 3, 2, 1])  # Exterior Quality
df['LandSlope'] = df['LandSlope'].replace(['Gtl', 'Mod', 'Sev'], [3, 2, 1])  # Land Slope
df['ExterCond'] = df['ExterCond'].replace(['Ex', 'Gd', 'TA', 'Fa', 'Po'], [4, 3, 2, 1, 0])  # Exterior Condition
df['HeatingQC'] = df['HeatingQC'].replace(['Ex', 'Gd', 'TA', 'Fa', 'Po'], [4, 3, 2, 1, 0])  # Heating Quality and Condition
df['KitchenQual'] = df['KitchenQual'].replace(['Ex', 'Gd', 'TA', 'Fa'], [3, 2, 1, 0])  # Kitchen Quality

# Replacing the NA values of each column with XX to avoid pandas from listing them as NaN
na_data = ['Alley', 'BsmtQual', 'BsmtCond', 'BsmtExposure', 'BsmtFinType1', 'BsmtFinType2', 'FireplaceQu',
'GarageType', 'GarageFinish', 'GarageQual', 'GarageCond', 'PoolQC', 'Fence', 'MiscFeature']

for i in na_data:
df[i] = df[i].fillna('XX')

# Replaced the NaN values of LotFrontage and MasVnrArea with the mean of their column
df['LotFrontage'] = df['LotFrontage'].fillna(df['LotFrontage'].mean())
df['MasVnrArea'] = df['MasVnrArea'].fillna(df['MasVnrArea'].mean())

x_train, x_test, y_train, y_test = train_test_split(df, df['SalePrice'], test_size = 0.3, random_state = 42)

sel = SelectFromModel(RandomForestClassifier(n_estimators = 100), threshold = 300 * "mean")
sel.fit(x_train, y_train)
sel.get_support()

selected_feat = x_train.columns[sel.get_support()]

return selected_feat

print(feature_encoding(train, categorical_columns))


The code for Random Forest is right after the train-test-split.

Update

After changing the code to the above version, I am getting the following error:

Traceback (most recent call last):
File "C:\Users\security\AppData\Roaming\Python\Python37\site-packages\pandas\core\indexes\base.py", line 2657, in get_loc
return self._engine.get_loc(key)
File "pandas\_libs\index.pyx", line 108, in pandas._libs.index.IndexEngine.get_loc
File "pandas\_libs\index.pyx", line 129, in pandas._libs.index.IndexEngine.get_loc
File "pandas\_libs\index_class_helper.pxi", line 91, in pandas._libs.index.Int64Engine._check_type
KeyError: 'Utilities'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
print(feature_encoding(train, categorical_columns))
df['Utilities'] = df['Utilities'].replace(['AllPub', 'NoSeWa'], [2, 1])  # Utilities
File "C:\Users\security\AppData\Roaming\Python\Python37\site-packages\pandas\core\frame.py", line 2927, in __getitem__
indexer = self.columns.get_loc(key)
File "C:\Users\security\AppData\Roaming\Python\Python37\site-packages\pandas\core\indexes\base.py", line 2659, in get_loc
return self._engine.get_loc(self._maybe_cast_indexer(key))
File "pandas\_libs\index.pyx", line 108, in pandas._libs.index.IndexEngine.get_loc
File "pandas\_libs\index.pyx", line 129, in pandas._libs.index.IndexEngine.get_loc
File "pandas\_libs\index_class_helper.pxi", line 91, in pandas._libs.index.Int64Engine._check_type
KeyError: 'Utilities'

• Hi, I see you are working on the Kaggle House Prices. Just one question, did I get it right, that you want to get the correlation between the SalePrice and all columns? Wouldn't you just have to use df.drop("SalePrice", axis=1).apply(lambda x: x.corr(df.SalePrice)) to get it (I mean after one-hot-encoding)? – jottbe Sep 13 '19 at 8:54

You don’t need the for loop at all.

def feature_encoding(df, categorical_list):

# One Hot Encoding the columns gathered in categorical_columns
# take one-hot encoding
OHE_sdf = pd.get_dummies(df[categorical_list])

# drop the old categorical column from original df
df.drop(columns=categorical_list, inplace = True)

# attach one-hot encoded columns to original dataframe
df = pd.concat([df, OHE_sdf], axis = 1, ignore_index = True)

• Why did you remove the axis parameter? – Onur Ozbek Aug 12 '19 at 5:28
• There is no need for it if you specify what you are removing is columns, axis is then implied – MichaelD Aug 12 '19 at 14:31
• Using ignore_index=True kills all the column names, and makes the rest of the cleaning steps fail. (I checked; removing it works, but there are several categorical columns that aren't in OP's list and need to be accounted for. And also filling with 'XX' fails since sklearn cannot work with strings. @OnurOzbek, you'll need to fill with something else, presumably depending on the column type.) – Ben Reiniger Aug 12 '19 at 19:25
• Yea but now I'm getting a KeyError for my Utilities column. I've posted the entire error above. – Onur Ozbek Aug 13 '19 at 9:58
• I don’t think the issue raised in your last comment has anything to do with the changes. Check your sources and make sure that the column names match. Also, you can encode ordinal values (I.e. ordered categories) by specifying the column type as ‘(categorical’)[pandas.pydata.org/pandas-docs/stable/user_guide/… – MichaelD Aug 13 '19 at 21:22

I guess the problem is in the for loop you have used

for col in categorical_list:

# take one-hot encoding
OHE_sdf = pd.get_dummies(df[categorical_list])

# drop the old categorical column from original df
df.drop(col, axis = 1, inplace = True)

# attach one-hot encoded columns to original dataframe
df = pd.concat([df, OHE_sdf], axis = 1, ignore_index = True)

return df


You have provided a return statement. This is the reason it is returning a dataframe when the function is called. So just remove the return statement in the for loop.

For getting the columns with the highest correlation to the target variable you can zip the lists of the features and the correlation values that you get and sort them in descending order with respect to the correlation values.

• I don't think so! If I don't return that df, the OHE columns are not added back into the dataframe. I tried your approach and now its giving me a KeyError with the names of every column that was OHE. Btw, that for loop is at the beginning of the function! There is another return statement at the bottom of the function that totally overrides the first return statement. – Onur Ozbek Aug 11 '19 at 15:25
• @OnurOzbek, that's not how return statements work. The first one encountered will exit the function. – Ben Reiniger Aug 11 '19 at 19:20
• @OnurOzbek, as for the KeyErrors, that's probably from the fill_na step. You don't have those columns anymore (and shouldn't need to fill them anyway) after using get_dummies. Finally, 300*mean might be too aggressive. – Ben Reiniger Aug 11 '19 at 19:26
• @BenReiniger That first one is indented within the for loop tho. Why would it exit the function? (Not doubting you. Just confused). – Onur Ozbek Aug 12 '19 at 0:19
• @OnurOzbek: In particular, return is a feature only for functions. For-loops (and other control structures) are unaffected (but have their own sorts of short-circuits, e.g. break, continue, else). For-loops also don't have their own local variables, so in your code the dataframe changes persist past the loop, so just removing that return statement should do what you want. – Ben Reiniger Aug 12 '19 at 3:28