Random forest vs. XGBoost vs. MLP Regressor for estimating claims costs

Question

Context

I'm building a (toy) machine learning model estimate the cost of an insurance claim (injury related). Aim is to teach myself machine learning by doing. I have settled on three algorithms to test: Random forest, XGBoost and a multi-layer perceptron.

Data set

The data set has the following columns:

cols = [ 'AGE_RANGE', 'GENDER', 'TOTAL_PAID', 'INDUSTRY_DESCRIPTION', 'WORKER_AGE', 'NATURE_CODE', 'ACCIDENT_TYPE_CODE', 'INJURY_NATURE']

'TOTAL_PAID' is the label ($s). The rest are features. The majority of features are categorical:

categoricals = [ 'AGE_RANGE', 'GENDER', 'INDUSTRY_DESCRIPTION', 'NATURE_CODE', 'ACCIDENT_TYPE_CODE', 'INJURY_NATURE']

Code:

Import:

import pandas as pd
import numpy as np

cols = [ 'AGE_RANGE', 'GENDER', 'TOTAL_PAID', 'INDUSTRY_DESCRIPTION', 'WORKER_AGE', 'NATURE_CODE', 'ACCIDENT_TYPE_CODE', 'INJURY_NATURE']
features = pd.read_csv('gs://longtailclaims2/filename.csv', usecols = cols, header=0, encoding='ISO-8859-1')
categoricals = [ 'AGE_RANGE', 'GENDER', 'INDUSTRY_DESCRIPTION', 'NATURE_CODE', 'ACCIDENT_TYPE_CODE', 'INJURY_NATURE']

First I turn categorical values into 0s and 1s:

features2 = pd.get_dummies(features, columns = categoricals)

Then I isolate features from labels (TOTAL_PAID):

labels = np.array(features['TOTAL_PAID'])
features = features2.drop('TOTAL_PAID', axis = 1)
feature_list = list(features.columns)
feature_list_no_facts = list(features.columns)

Spit into SK Learn training and test set:

# Using Skicit-learn to split data into training and testing sets
from sklearn.model_selection import train_test_split

# Split the data into training and testing sets
train_features, test_features, train_labels, test_labels = train_test_split(features, labels, test_size = 0.25, random_state = 42)
test_features.head(5)
test_features.head(5)

Then I explore the data:

print('Training Features Shape:', train_features.shape)
print('Training Labels Shape:', train_labels.shape)
print('Testing Features Shape:', test_features.shape)
print('Testing Labels Shape:', test_labels.shape)

Output:

Training Features Shape: (128304, 337)
Training Labels Shape: (128304,)
Testing Features Shape: (42768, 337)
Testing Labels Shape: (42768,)

1. XGBRegressor

First we try training XGBoost model. I needed so push # estimates above 10,000 to get a decent accuracy (R2 > 0.94)

# Import the model we are using
from xgboost import XGBRegressor
from sklearn.ensemble import RandomForestRegressor
from sklearn import ensemble 
from sklearn.metrics import mean_squared_error

x = XGBRegressor(random_state = 44, n_jobs = 8, n_estimators = 10000, max_depth=10, verbosity = 3)
x.fit(train_features, train_labels)
print('xgboost train score: ', x.score(train_features, train_labels))
predictions = x.predict(test_features)
print('xgboost test score: ', x.score(test_features, test_labels))

Here, train and test score: R2 ~0.94.

2. Random Forest Regressor

Then we try Random Forest model. After some fiddling it appears 100 estimators is enough to get a pretty good accuracy (R2 > 0.94)

# Instantiate model with 100 decision trees
rf = RandomForestRegressor(n_estimators = 100, criterion='mse', verbose=1, random_state = np.random.RandomState(42), n_jobs = -1)
# Train the model on training data
rf.fit(train_features, train_labels);
print('random forest train score: ', rf.score(train_features, train_labels))
predictions = rf.predict(test_features)
print('random forest test score: ', rf.score(test_features, test_labels))

In this case, train and test score R2 at ~0.94.

3. MLP Neural Network

Finally I wanted to compare performance to an MLP Regressor. According to the books I have been reading on deep learning, a neural network should be able to outperform any shallow learning algorithm given enough time and horse power. I am using a pretty beefy machine with 8 cores and 30 GB RAM on Google Cloud.

from sklearn.neural_network import MLPRegressor
nn = MLPRegressor(hidden_layer_sizes=(338, 338, 50), 
                  activation='relu', solver='adam', max_iter = 100, random_state = 56, verbose = True)
nn.fit(train_features, train_labels)
nn_predictions = nn.predict(test_features)
print('nn train score: ', nn.score(train_features, train_labels))
print('nn test score: ', nn.score(test_features, test_labels))

R2 around 0.4.

I've used 338 neurons in the input layer as this is the exact number of columns. The neural network stalls at 82 iterations and doesn't go any further. Running with 50 iterations I get accuracy at R2 < 0.5, which is not impressive.

My questions:

1. Do I handle the management of categorical features correctly? With the scores above, does it look like model #1 and #2 are overfitting? R2 > 0.94 is pretty good and both test and training accuracy look good and in same ballpark, so I don't think it is overfitting

2. Why does the neural network not perform that well?

3. Should I consider a different type of neural network for regression?

4. Why do I have to add so many more estimators to XGBoost (10,000) to get the same performance as Random Forest (100)?

5. What would be a fit for purpose neural network to solve this problem with deep learning? I am concerned the ensemble methods may not be appropriate or that I am doing something wrong.

Answer 1

Some ideas:

1. Handling categorical features correctly: using one-hot encoding is one valid approach. Other approaches include target encoding (or mean encoding), and the hashing trick. There's no real hard and fast rule about when to choose which method.
2. Poor performance of neural network: I don't have much experience with neural networks, but I have read that inputs into neural networks should be scaled in some way - either standardised, or to lie within some narrow and consistent interval. You could also look at other layer structures - e.g. have you tried the default values from Scikit-Learn?
3. Considering different kind of network: Judge based on (2) above
4. More estimators in xgboost: xgboost has many parameters to fine tune. You should also consider that xgboost uses linear regression as a default regression task, which implies that your target insurance losses are normally distributed. This is not usually the case in the real world, where we see that insurance losses usually follow a Tweedie distribution. xgboost offers Tweedie regression capability.
5. Optimal neural network for this problem: Unsure as my experience with neural networks is limited.

Answer 2

Regarding the MLPRegressor, you should use the lbfgs optimizer for better results with small datasets. Accuracy should be up to 0.99.