Using random forest for selecting variables returns the entire dataframe

Question

I am in the process of dimensionality reduction. I am using Random Forest to find the columns with the highest level of correlation with the target SalePrice column.

The problem is that the output is too large. Definitely not what I want from it. It is returning 259 columns. Some of these columns are a result of one-hot-encoding the categorical variables and adding them back into the dataframe, which logically increases the dimension of the dataset. However, I only wanted to return the columns with the highest correlation to the target variable 'SalePrice'. Not the whole damn dataframe.

Here is the output:

       0   1     2      3     4    5    6    ... 252 253 254 255 256 257 258
0        1  RL  65.0   8450  Pave  NaN  Reg  ...   0   1   0   0   1   0   1
1        2  RL  80.0   9600  Pave  NaN  Reg  ...   0   1   0   0   1   0   1
2        3  RL  68.0  11250  Pave  NaN  IR1  ...   0   1   0   0   1   0   1
3        4  RL  60.0   9550  Pave  NaN  IR1  ...   0   0   0   0   1   0   1
4        5  RL  84.0  14260  Pave  NaN  IR1  ...   0   1   0   0   1   0   1
...    ...  ..   ...    ...   ...  ...  ...  ...  ..  ..  ..  ..  ..  ..  ..
1455  1456  RL  62.0   7917  Pave  NaN  Reg  ...   0   1   0   0   1   0   1
1456  1457  RL  85.0  13175  Pave  NaN  Reg  ...   0   1   0   0   1   0   1
1457  1458  RL  66.0   9042  Pave  NaN  Reg  ...   0   1   0   0   1   0   1
1458  1459  RL  68.0   9717  Pave  NaN  Reg  ...   0   1   0   0   1   0   1
1459  1460  RL  75.0   9937  Pave  NaN  Reg  ...   0   1   0   0   1   0   1

[1460 rows x 259 columns]

Here is my code:

import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_selection import SelectFromModel
from sklearn.model_selection import train_test_split

train = pd.read_csv("https://raw.githubusercontent.com/oo92/Boston-Kaggle/master/train.csv")
test = pd.read_csv("https://raw.githubusercontent.com/oo92/Boston-Kaggle/master/test.csv")

categorical_columns = ['MSSubClass', 'MSZoning', 'LotShape', 'LandContour', 'LotConfig', 'Neighborhood', 'Condition1',
                       'Condition2', 'BldgType', 'HouseStyle', 'RoofStyle', 'RoofMatl', 'Exterior1st', 'Exterior2nd',
                       'Foundation', 'Heating', 'Electrical', 'Functional', 'GarageType', 'PavedDrive', 'Fence',
                       'MiscFeature', 'SaleType', 'SaleCondition', 'Street', 'CentralAir']

ranked_columns = ['Utilities', 'LandSlope', 'ExterQual', 'ExterCond', 'BsmtQual', 'BsmtCond', 'BsmtExposure',
                  'BsmtFinType1', 'BsmtFinType2', 'HeatingQC', 'KitchenQual', 'FireplaceQu', 'GarageQual', 'GarageCond',
                  'PoolQC', 'OverallQual', 'OverallCond']

numerical_columns = ['LotArea', 'LotFrontage', 'YearBuilt', 'YearRemodAdd', 'MasVnrArea', 'BsmtFinSF1', 'BsmtFinSF2',
                     'BsmtUnfSF','TotalBsmtSF', '1stFlrSF', '2ndFlrSf', 'LowQualFinSF', 'GrLivArea', 'BsmtFullBath',
                     'BsmtHalfBath', 'FullBath', 'HalfBath', 'Bedroom', 'Kitchen', 'TotRmsAbvGrd', 'Fireplaces',
                     'GarageYrBlt', 'GarageCars', 'GarageArea', 'WoodDeckSF', 'OpenPorchSF', 'EnclosedPorch',
                     '3SsnPorch', 'ScreenPorch', 'PoolArea', 'MiscVal', 'MoSold', 'YrSold']


def feature_encoding(df, categorical_list):

    # take one-hot encoding
    OHE_sdf = pd.get_dummies(df[categorical_list])

    # drop the old categorical column from original df
    df.drop(columns = categorical_list, inplace = True)

    # attach one-hot encoded columns to original dataframe
    df = pd.concat([df, OHE_sdf], axis = 1, ignore_index = True)

    # Integer Encoding
    df['Utilities'] = df['Utilities'].replace(['AllPub', 'NoSeWa'], [2, 1])  # Utilities
    df['ExterQual'] = df['ExterQual'].replace(['Ex', 'Gd', 'TA', 'Fa'], [4, 3, 2, 1])  # Exterior Quality
    df['LandSlope'] = df['LandSlope'].replace(['Gtl', 'Mod', 'Sev'], [3, 2, 1])  # Land Slope
    df['ExterCond'] = df['ExterCond'].replace(['Ex', 'Gd', 'TA', 'Fa', 'Po'], [4, 3, 2, 1, 0])  # Exterior Condition
    df['HeatingQC'] = df['HeatingQC'].replace(['Ex', 'Gd', 'TA', 'Fa', 'Po'], [4, 3, 2, 1, 0])  # Heating Quality and Condition
    df['KitchenQual'] = df['KitchenQual'].replace(['Ex', 'Gd', 'TA', 'Fa'], [3, 2, 1, 0])  # Kitchen Quality

    # Replacing the NA values of each column with XX to avoid pandas from listing them as NaN
    na_data = ['Alley', 'BsmtQual', 'BsmtCond', 'BsmtExposure', 'BsmtFinType1', 'BsmtFinType2', 'FireplaceQu',
               'GarageType', 'GarageFinish', 'GarageQual', 'GarageCond', 'PoolQC', 'Fence', 'MiscFeature']

    for i in na_data:
        df[i] = df[i].fillna('XX')

    # Replaced the NaN values of LotFrontage and MasVnrArea with the mean of their column
    df['LotFrontage'] = df['LotFrontage'].fillna(df['LotFrontage'].mean())
    df['MasVnrArea'] = df['MasVnrArea'].fillna(df['MasVnrArea'].mean())

    x_train, x_test, y_train, y_test = train_test_split(df, df['SalePrice'], test_size = 0.3, random_state = 42)

    sel = SelectFromModel(RandomForestClassifier(n_estimators = 100), threshold = 300 * "mean")
    sel.fit(x_train, y_train)
    sel.get_support()

    selected_feat = x_train.columns[sel.get_support()]

    return selected_feat


print(feature_encoding(train, categorical_columns))

The code for Random Forest is right after the train-test-split.

Update

After changing the code to the above version, I am getting the following error:

Traceback (most recent call last):
  File "C:\Users\security\AppData\Roaming\Python\Python37\site-packages\pandas\core\indexes\base.py", line 2657, in get_loc
    return self._engine.get_loc(key)
  File "pandas\_libs\index.pyx", line 108, in pandas._libs.index.IndexEngine.get_loc
  File "pandas\_libs\index.pyx", line 129, in pandas._libs.index.IndexEngine.get_loc
  File "pandas\_libs\index_class_helper.pxi", line 91, in pandas._libs.index.Int64Engine._check_type
KeyError: 'Utilities'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/security/Downloads/AP/Boston-Kaggle/Boston.py", line 66, in <module>
    print(feature_encoding(train, categorical_columns))
  File "C:/Users/security/Downloads/AP/Boston-Kaggle/Boston.py", line 37, in feature_encoding
    df['Utilities'] = df['Utilities'].replace(['AllPub', 'NoSeWa'], [2, 1])  # Utilities
  File "C:\Users\security\AppData\Roaming\Python\Python37\site-packages\pandas\core\frame.py", line 2927, in __getitem__
    indexer = self.columns.get_loc(key)
  File "C:\Users\security\AppData\Roaming\Python\Python37\site-packages\pandas\core\indexes\base.py", line 2659, in get_loc
    return self._engine.get_loc(self._maybe_cast_indexer(key))
  File "pandas\_libs\index.pyx", line 108, in pandas._libs.index.IndexEngine.get_loc
  File "pandas\_libs\index.pyx", line 129, in pandas._libs.index.IndexEngine.get_loc
  File "pandas\_libs\index_class_helper.pxi", line 91, in pandas._libs.index.Int64Engine._check_type
KeyError: 'Utilities'

Answer 1

You don’t need the for loop at all.

def feature_encoding(df, categorical_list):

    # One Hot Encoding the columns gathered in categorical_columns
    # take one-hot encoding
    OHE_sdf = pd.get_dummies(df[categorical_list])

     # drop the old categorical column from original df
    df.drop(columns=categorical_list, inplace = True)

     # attach one-hot encoded columns to original dataframe
     df = pd.concat([df, OHE_sdf], axis = 1, ignore_index = True)

Answer 2

I guess the problem is in the for loop you have used

for col in categorical_list:

        # take one-hot encoding
        OHE_sdf = pd.get_dummies(df[categorical_list])

        # drop the old categorical column from original df
        df.drop(col, axis = 1, inplace = True)

        # attach one-hot encoded columns to original dataframe
        df = pd.concat([df, OHE_sdf], axis = 1, ignore_index = True)

        return df

You have provided a return statement. This is the reason it is returning a dataframe when the function is called. So just remove the return statement in the for loop.

For getting the columns with the highest correlation to the target variable you can zip the lists of the features and the correlation values that you get and sort them in descending order with respect to the correlation values.