I am trying to build a 2 level stacking model in order to tackle a multiclass classification problems with 8 classes. My base (level 1) models andd their micro f1 scores in the test set are:
- Random Forest Classifier (0.51)
- XGBoost Classifier (0.54)
- LightGBM Classifier (0.54)
- Logistic Regression (0.44)
- keras neural network (0.57)
- keras neural netqork (0.56)
As a level 2 model I use an XGBClassifier which is not tuned. I use 7 fold cross validation to produce the meta features for level 2 model. The code I use to produce the meta features for the simple classifiers is:
ntrain = X_train.shape[0]
ntest = X_test.shape[0]
seed = 0
nfolds = 7
kf = StratifiedKFold(nfolds, random_state=seed)
def get_meta(clf, Χ_train, y_train, Χ_test):
meta_train = np.zeros((ntrain,))
meta_test = np.zeros((ntest,))
for i, (train_index, test_index) in enumerate(kf.split(X_train, y_train)):
Χ_tr = X_train.iloc[train_index]
y_tr = y_train.iloc[train_index]
Χ_te = Χ_train.iloc[test_index]
clf.train(Χ_tr, y_tr)
meta_train[test_index] = clf.predict(Χ_te)
clf.fit(X_train,y_train)
meta_test = clf.predict(X_test)
return meta_train.reshape(-1, 1), meta_test.reshape(-1, 1)
and for the keras neural networks is:
def get_meta_keras(clf, Χ_train, y_train, Χ_test, epochs = 200, batch_size = 70, class_weight=class_weights):
meta_train = np.zeros((ntrain,))
meta_test = np.zeros((ntest,))
encoder = LabelEncoder()
encoder.fit(y_train)
encoded_Y = encoder.transform(y_train)
# convert integers to dummy variables (i.e. one hot encoded)
dummy_y = np_utils.to_categorical(encoded_Y)
for i, (train_index, test_index) in enumerate(kf.split(X_train, y_train)):
Χ_tr = X_train.iloc[train_index]
y_tr = dummy_y[train_index]
Χ_te = Χ_train.iloc[test_index]
clf.fit(Χ_tr, y_tr, epochs = epochs, batch_size = batch_size, class_weight=class_weights)
meta_train[test_index] = clf.predict_classes(Χ_te)
clf.fit(X_train, dummy_y, epochs = epochs, batch_size = batch_size, class_weight=class_weights)
meta_test = clf.predict_classes(X_test)
return meta_train.reshape(-1, 1), meta_test.reshape(-1, 1)
My final micro f1 score is 0.54 which is less than the best of my base models score. My models are uncorrelated (corr<0.55). I tried to add more simple models like knn, naive bayes etc but the score fell even more. Why doesn't my stacking approach improve the score ?