Some background:

I am attempting to predict attendance at this place using various features I have collected. I added to these features by extracting binary information from some of them. For example, temperature was expanded to temperature, temp>100, temp<60, tempbetween60and100, the seasons were expanded out from the months as well. In total, I have 237 features, which include binary, multi-level categorical, and numerical.

the output, attendance, is either numerical (by true attendance) or categorical (by binning attendance into 0-500, 501-1000, 1001-1200, 1201+ ). I am debating whether or not to further simplify the task and turn it into binary classification and do 2 bins, 0-800 and 801+

My attempts thus far:

My first approach, with much guidance on SO, was to gather as much data as possible (a max of 3 years worth of samples, with as many features as possible) and create a neural net to predict the true attendance. With some advice, I tried simpler methods, such as linear/logistic/other regression, but to no avail. Eventually, I switched to classification in hopes it would improve the accuracy and make this easier. In truth, the binning is all I would need to know, as opposed to true attendance anyway. The accuracy did improve, up to about 70%. Since then Ive tried methods such as PCA, Cross Validation, feature selection, etc.

I have mostly used SkLearn's library, but have also used the Orange gui due to ease of trying various models.

I come here now to get even more advice:

Should I perform feature selection on this data? I know I have some data that is highly correlated, as I extracted some out from others.

If yes, how? I have looked at pearsons coef between all columns, MIC between all columns, and I tried using feature importance in various ways here:

from sklearn.ensemble import RandomForestRegressor
from sklearn.ensemble import RandomForestClassifier
import numpy as np
import pandas as pd
from collections import defaultdict
from sklearn.model_selection import ShuffleSplit
from sklearn.metrics import r2_score
from sklearn.svm import LinearSVC
from sklearn.feature_selection import RFE
from sklearn.feature_selection import chi2
from sklearn.feature_selection import mutual_info_classif

df = pd.read_excel('Attempt1.xlsx', index_col=0, header=0)
rows, cols = df.shape

X = df.iloc[: , 0:(cols - 1)]
Y = df.iloc[: , cols - 1 ]

# Places attendance into bins
Y = Y.apply(lambda x: 0 if 0 <= x <= 500 else (1 if 500 < x <= 1000 else (2 if 1000 < x <= 1200 else 3)))

#Column names
names = list(X)

X_train = X.iloc[:rows-151, :]
X_test = X.iloc[rows-151:, :]

Y_train = Y.iloc[:rows-151]
Y_test = Y.iloc[rows-151:]

# # chi2 of categorical features
# feat = chi2(X_train.drop('Temperature', axis=1),Y_train)
# print(feat)

rf1 = RandomForestClassifier()

features = []

print("Features sorted by their score classifier (gini):")
features.append(sorted(zip(map(lambda x: round(x, 4), rf1.feature_importances_), names),

rf2= RandomForestClassifier(criterion='entropy')

print("Features sorted by their score classifier (entropy):")
features.append(sorted(zip(map(lambda x: round(x, 4), rf2.feature_importances_), names),

svm = LinearSVC()
# create the RFE model for the svm classifier
# and select attributes
rfe1 = RFE(svm, 20)
rfe1 = rfe1.fit(X_train, Y_train)
# print summaries for the selection of attributes
features.append([x for x,y in zip(list(X_train),rfe1.support_) if y==True ])

rf3 = RandomForestClassifier()

rfe2 = RFE(rf3, 20)
rfe2 = rfe2.fit(X_train, Y_train)
# print summaries for the selection of attributes
features.append([x for x,y in zip(list(X_train),rfe2.support_) if y==True ])

rf4= RandomForestClassifier(criterion='entropy')

rfe3 = RFE(rf4, 20)
rfe3 = rfe3.fit(X_train, Y_train)
# print summaries for the selection of attributes
features.append([x for x,y in zip(list(X_train),rfe3.support_) if y==True ])

all_feats = {}
for a,b,c,d,e in zip(features[0], features[1], features[2], features[3], features[4]):
    if a[1] not in all_feats.keys():
        all_feats[a[1]] = 1
        all_feats[a[1]] += 1
    if b[1] not in all_feats.keys():
        all_feats[b[1]] = 1
        all_feats[b[1]] += 1
    if c not in all_feats.keys():
        all_feats[c] = 1
        all_feats[c] += 1
    if d not in all_feats.keys():
        all_feats[d] = 1
        all_feats[d] += 1
    if e not in all_feats.keys():
        all_feats[e] = 1
        all_feats[e] += 1
print(sorted(all_feats, key=all_feats.get, reverse=True)[:10])

My goal here was to extract the top ten features that each method chose. This didnt improve performance from testing in Orange

At this point, I am not sure what is left besides picking a better model and tuning the parameters, because even after feature selection I seem to be stuck at 70%. Is there a better way to look at my data and decide

  1. Feature selection? Y/n
  2. If yes, which methods?
  3. PCA or similar methods? Y/n
  4. If yes, which method?
  5. Which model? Linear, or not?
  6. How to tune the model?

From all of my testing and various models used, I believe I need to learn how to better understand this data and use that to guide the steps, but I cant seem to get a handle on it.. any suggestions?

Link to full data

In the data provided, the yellow and bold data are missing values. I imputed them using their corresponding data in future years, as it is probably close. In testing, I tried this, and imputing using mean/common values, as well as eliminating all rows with missing values.

Quick edit: please excuse the messy and borderline bad code... it was written frantically attempting to test it before leaving


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.