How to reduce RMS error value in regression analysis & predictions - feature engineering, model selection

Question

There's this dataset containing the metadata of Twitch's top 1,000 streamers of 2020. You can have the details here.

I am currently participating in a challenge to predict the values for Followers gained, by creating and training the model using the remaining features from the dataset. The kernel objective is to get the lowest RMSE (Root-Mean Squared Error) metric value from the model's predictions.

Until now, I have made numerous attempts to lower down the RMSE loss value as much as possible. My current lowest achievement is around 101,000, which I got by augmenting the dataset and training a DNN model with 7 hidden-layers.
Yet, I am trying to lower the RMSE error value to 5-digits from 6. I've tried the removal of outliers, data augmentation, feature engineering, polynomial regression, trained DNN models with more than 5 hidden-layers on average; created and trained multiple models and stacked them in order to make a final prediction (and I was heard from the community that using a stacked model is one of the keys to achieve regression predictions resulting in low error metrics.)

All the results from my models have not surpassed the threshold of 6-digits of RMSE error. Feature engineering was conducted to contain only the new variables with high correlation with the target prediction value. Nearly all hyperparameters of the models created using the Tensorflow library were adjusted to show the best performance. And yet, RMSE values doesn't seem to show reduction in its value.

Here are some of the codes I wrote explaining the procedure of feature engineering and model creation & training. These did not resulted in a lower RMSE value than around 101,000. But instead, it resulted in a higher value, nearly 110,000.

[DNN Model]

inputs = Input(shape=(7))

x1 = Dense(430, activation='selu', kernel_initializer=keras.initializers.RandomUniform())(inputs)
d1 = Dropout(0.9)(x1)

x2 = Dense(430, activation='selu', kernel_initializer=keras.initializers.RandomUniform())(d1)
d2 = Dropout(0.8)(x2)

x3 = Dense(256, activation='selu', kernel_initializer=keras.initializers.RandomUniform())(d2)
d3 = Dropout(0.7)(x3)

x4 = Dense(256, activation='selu', kernel_initializer=keras.initializers.RandomUniform())(d3)
d4 = Dropout(0.6)(x4)

x5 = Dense(128, activation='selu', kernel_initializer=keras.initializers.RandomUniform())(d4)
d5 = Dropout(0.7)(x5)

x6 = Dense(128, activation='selu', kernel_initializer=keras.initializers.RandomUniform())(d5)
d6 = Dropout(0.9)(x6)

x7 = Dense(32, activation='selu', kernel_initializer=keras.initializers.RandomUniform())(d6)
d7 = Dropout(0.9)(x7)

x8 = Dense(32, activation='selu', kernel_initializer=keras.initializers.RandomUniform())(d7)
d8 = Dropout(0.9)(x8)

outputs = Dense(1)(d8)

model = keras.Model(inputs=inputs, outputs=outputs)

def rmse(y_true, y_pred):
    return K.sqrt(mse(y_true, y_pred))

model.compile(
    loss=rmse, optimizer=keras.optimizers.Adam(learning_rate=0.001)
)

[XGBRegressor]

from xgboost import XGBRegressor

# Model generation and training
xgb_model = XGBRegressor(objective='reg:linear', 
                         n_estimators=5000, 
                         max_depth=15, 
                         eta=0.001, 
                         subsample=0.8, 
                         colsample_bytree=0.8, 
                         eval_metric='rmse')

xgb_model.fit(X_train, y_train, eval_set=[(X_val, y_val)], verbose=0)

# Make predictions
train_pred = xgb_model.predict(X_train)
test_pred = xgb_model.predict(X_test)

# Train set performance
xgb_train_evs = explained_variance_score(y_train, train_pred)
xgb_train_rmse = rmse(y_train, train_pred)

# Test set performance
xgb_test_evs = explained_variance_score(y_test, test_pred)
xgb_test_rmse = rmse(y_test, test_pred)

# Output results
xgb_results = f"""
XGBoost Train EVS: {xgb_train_evs}
XGBoost Train RMSE: {xgb_train_rmse}

XGBoost Test EVS: {xgb_test_evs}
XGBoost Test RMSE: {xgb_test_rmse}
"""

print(xgb_results)

[LGBRegressor]

from lightgbm import LGBMRegressor as lgb

# Model generation and training
lgb_model = lgb(boosting_type='gbdt', objective='regression',
                num_leaves=150, learning_rate=0.001, n_estimators=10**4)
lgb_model.fit(X_train, y_train)

# Make predictions
train_pred = lgb_model.predict(X_train)
test_pred = lgb_model.predict(X_test)

# Train set performance
lgb_train_evs = explained_variance_score(y_train, train_pred)
lgb_train_rmse = rmse(y_train, train_pred)

# Test set performance
lgb_test_evs = explained_variance_score(y_test, test_pred)
lgb_test_rmse = rmse(y_test, test_pred)

# Output results
lgb_results = f"""
LightGBM Train EVS: {lgb_train_evs}
LightGBM Train RMSE: {lgb_train_rmse}

LightGBM Test EVS: {lgb_test_evs}
LightGBM Test RMSE: {lgb_test_rmse}
"""

print(lgb_results)

[RandomForestRegressor]

from sklearn.ensemble import RandomForestRegressor

# Model generation and training
forest = RandomForestRegressor(n_estimators=350, verbose=1)
forest.fit(X_train, y_train)

# Make predictions
train_pred = forest.predict(X_train)
test_pred = forest.predict(X_test)

# Train set performance
rf_train_evs = explained_variance_score(y_train, train_pred)
rf_train_rmse = rmse(y_train, train_pred)

# Test set performance
rf_test_evs = explained_variance_score(y_test, test_pred)
rf_test_rmse = rmse(y_test, test_pred)

# Output results
rf_results = f"""
Random Forests Train EVS: {rf_train_evs}
Random Forests Train RMSE: {rf_train_rmse}

Random Forests Test EVS: {rf_test_evs}
Random Forests Test RMSE: {rf_test_rmse}
"""

print(rf_results)

[Stacked Models]

from sklearn.ensemble import StackingRegressor
from sklearn.linear_model import LinearRegression

# Models to use
estimators = [
    ('XGBRegressor', xgb_model),
    ('LGBMRegressor', lgb_model),
    ('RFRegressor', forest)
]

# Build Stacked Model
stack_model = StackingRegressor(
    estimators=estimators, final_estimator=LinearRegression()
)

# Train Stacked Model
stack_model.fit(X_train, y_train)

# Make Predictions
sm_train_pred = stack_model.predict(X_train) 
sm_test_pred = stack_model.predict(X_test)

# Train Set Performance
sm_train_evs = explained_variance_score(y_train, sm_train_pred)
sm_train_rmse = rmse(y_train, sm_train_pred)

# Test Set Performance
sm_test_evs = explained_variance_score(y_test, sm_test_pred)
sm_test_rmse = rmse(y_test, sm_test_pred)

# Output results
sm_results = f"""
Stacked Model Train EVS: {sm_train_evs}
Stacked Model Train RMSE: {sm_train_rmse}

Stacked Model Test EVS: {sm_test_evs}
Stacked Model Test RMSE: {sm_test_rmse}
"""

print(sm_results)

[Model Predictions - in this case, the final prediction is calculated as the mean value between the DNN model's and the Stacked model's predictions.]

dnn_predictions = model.predict(test.values)
dnn_predictions = dnn_predictions.transpose()[0]

stacked_predictions = model.predict(test.values)
stacked_predictions = stacked_predictions.transpose()[0]

predictions = np.divide(np.add(dnn_predictions, stacked_predictions), 2)

Apparently, using the dataset created after feature engineering tends to result predictions with a higher error value. I'm looking for a reason why, and the opportunities of enhancement. How can an optimal feature engineering be performed in the case of this dataset? You can access the .csv file of the original dataset from the shared link above. Also, what is the most recommended model structure in regression tasks like this? I guess the more complex a model becomes, the harder it is to make predictions due to overfitting.

Answer 1

To improve a model there are multiple things you can do, not just feature engineering or using different models. For example you haven't tried hyperparameter tuning for any of the above models.

Here are some of the things you could try to improve the score (keep in mind that these are general tips for anyone who wants to improve their model):-

1.) First of all clean the data i.e deal with nan values and outliers. You can either remove nan values or impute them with a statistical value. For outliers you have to be careful as not all outliers are redundant. Try to see if they make sense and if so keep them. Else remove them. Also remove duplicate values from your dataset.

2.) The next step would be feature engineering. Try to see if 2 or more features can be combined into one. This will help in reducing the dimensionality (although you say this results in lower performance).

3.) Feature selection where you keep useful features only and remove the redundant features. The best tool you can use for this purpose is domain knowledge. Use your domain knowledge to see if any feature is worth removing. If you don't have enough domain knowledge, only then go for feature selection techniques like filter based, wrapper based, hybrid etc. (RFECV and BorutaShap are 2 of the best ones out there imo)

4.) Hyperparameter tuning! You can use any of the techniques out there for this purpose (go for Optuna if you ask me!).

5.) Try more models! I don't know how many models you have tried but here you mention only a few. There are more than 36 Regression models out there so try as many as you can to see which give best results. Here is an extensive list!

6.) Simply get more data. In your case this will probably not be of much use since you took the dataset from Kaggle!