# When to perform feature selection, how, and how does data affect choosing the predictive model?

Some background:

I am attempting to predict attendance at this place using various features I have collected. I added to these features by extracting binary information from some of them. For example, temperature was expanded to temperature, temp>100, temp<60, tempbetween60and100, the seasons were expanded out from the months as well. In total, I have 237 features, which include binary, multi-level categorical, and numerical.

the output, attendance, is either numerical (by true attendance) or categorical (by binning attendance into 0-500, 501-1000, 1001-1200, 1201+ ). I am debating whether or not to further simplify the task and turn it into binary classification and do 2 bins, 0-800 and 801+

My attempts thus far:

My first approach, with much guidance on SO, was to gather as much data as possible (a max of 3 years worth of samples, with as many features as possible) and create a neural net to predict the true attendance. With some advice, I tried simpler methods, such as linear/logistic/other regression, but to no avail. Eventually, I switched to classification in hopes it would improve the accuracy and make this easier. In truth, the binning is all I would need to know, as opposed to true attendance anyway. The accuracy did improve, up to about 70%. Since then Ive tried methods such as PCA, Cross Validation, feature selection, etc.

I have mostly used SkLearn's library, but have also used the Orange gui due to ease of trying various models.

I come here now to get even more advice:

Should I perform feature selection on this data? I know I have some data that is highly correlated, as I extracted some out from others.

If yes, how? I have looked at pearsons coef between all columns, MIC between all columns, and I tried using feature importance in various ways here:

from sklearn.ensemble import RandomForestRegressor
from sklearn.ensemble import RandomForestClassifier
import numpy as np
import pandas as pd
from collections import defaultdict
from sklearn.model_selection import ShuffleSplit
from sklearn.metrics import r2_score
from sklearn.svm import LinearSVC
from sklearn.feature_selection import RFE
from sklearn.feature_selection import chi2
from sklearn.feature_selection import mutual_info_classif

rows, cols = df.shape

X = df.iloc[: , 0:(cols - 1)]
Y = df.iloc[: , cols - 1 ]
print(X.shape)
print(Y.shape)

# Places attendance into bins
Y = Y.apply(lambda x: 0 if 0 <= x <= 500 else (1 if 500 < x <= 1000 else (2 if 1000 < x <= 1200 else 3)))
print(np.unique(Y))

#Column names
names = list(X)

X_train = X.iloc[:rows-151, :]
X_test = X.iloc[rows-151:, :]

Y_train = Y.iloc[:rows-151]
Y_test = Y.iloc[rows-151:]

# # chi2 of categorical features
# feat = chi2(X_train.drop('Temperature', axis=1),Y_train)
# print(feat)

rf1 = RandomForestClassifier()
rf1.fit(X_train,Y_train)

features = []

print("Features sorted by their score classifier (gini):")
features.append(sorted(zip(map(lambda x: round(x, 4), rf1.feature_importances_), names),
reverse=True))

rf2= RandomForestClassifier(criterion='entropy')
rf2.fit(X_train,Y_train)

print("Features sorted by their score classifier (entropy):")
features.append(sorted(zip(map(lambda x: round(x, 4), rf2.feature_importances_), names),
reverse=True))

svm = LinearSVC()
# create the RFE model for the svm classifier
# and select attributes
rfe1 = RFE(svm, 20)
rfe1 = rfe1.fit(X_train, Y_train)
# print summaries for the selection of attributes
features.append([x for x,y in zip(list(X_train),rfe1.support_) if y==True ])

rf3 = RandomForestClassifier()

rfe2 = RFE(rf3, 20)
rfe2 = rfe2.fit(X_train, Y_train)
# print summaries for the selection of attributes
features.append([x for x,y in zip(list(X_train),rfe2.support_) if y==True ])

rf4= RandomForestClassifier(criterion='entropy')

rfe3 = RFE(rf4, 20)
rfe3 = rfe3.fit(X_train, Y_train)
# print summaries for the selection of attributes
features.append([x for x,y in zip(list(X_train),rfe3.support_) if y==True ])

all_feats = {}
for a,b,c,d,e in zip(features[0], features[1], features[2], features[3], features[4]):
if a[1] not in all_feats.keys():
all_feats[a[1]] = 1
else:
all_feats[a[1]] += 1
if b[1] not in all_feats.keys():
all_feats[b[1]] = 1
else:
all_feats[b[1]] += 1
if c not in all_feats.keys():
all_feats[c] = 1
else:
all_feats[c] += 1
if d not in all_feats.keys():
all_feats[d] = 1
else:
all_feats[d] += 1
if e not in all_feats.keys():
all_feats[e] = 1
else:
all_feats[e] += 1
print(sorted(all_feats, key=all_feats.get, reverse=True)[:10])


My goal here was to extract the top ten features that each method chose. This didnt improve performance from testing in Orange

At this point, I am not sure what is left besides picking a better model and tuning the parameters, because even after feature selection I seem to be stuck at 70%. Is there a better way to look at my data and decide

1. Feature selection? Y/n
2. If yes, which methods?
3. PCA or similar methods? Y/n
4. If yes, which method?
5. Which model? Linear, or not?
6. How to tune the model?

From all of my testing and various models used, I believe I need to learn how to better understand this data and use that to guide the steps, but I cant seem to get a handle on it.. any suggestions?