I'm building a (toy) machine learning model estimate the cost of an insurance claim (injury related). Aim is to teach myself machine learning by doing. I have settled on three algorithms to test: Random forest, XGBoost and a multi-layer perceptron.
The data set has the following columns:
cols = [ 'AGE_RANGE', 'GENDER', 'TOTAL_PAID', 'INDUSTRY_DESCRIPTION', 'WORKER_AGE', 'NATURE_CODE', 'ACCIDENT_TYPE_CODE', 'INJURY_NATURE']
'TOTAL_PAID' is the label ($s). The rest are features. The majority of features are categorical:
categoricals = [ 'AGE_RANGE', 'GENDER', 'INDUSTRY_DESCRIPTION', 'NATURE_CODE', 'ACCIDENT_TYPE_CODE', 'INJURY_NATURE']
import pandas as pd import numpy as np cols = [ 'AGE_RANGE', 'GENDER', 'TOTAL_PAID', 'INDUSTRY_DESCRIPTION', 'WORKER_AGE', 'NATURE_CODE', 'ACCIDENT_TYPE_CODE', 'INJURY_NATURE'] features = pd.read_csv('gs://longtailclaims2/filename.csv', usecols = cols, header=0, encoding='ISO-8859-1') categoricals = [ 'AGE_RANGE', 'GENDER', 'INDUSTRY_DESCRIPTION', 'NATURE_CODE', 'ACCIDENT_TYPE_CODE', 'INJURY_NATURE']
First I turn categorical values into 0s and 1s:
features2 = pd.get_dummies(features, columns = categoricals)
Then I isolate features from labels (TOTAL_PAID):
labels = np.array(features['TOTAL_PAID']) features = features2.drop('TOTAL_PAID', axis = 1) feature_list = list(features.columns) feature_list_no_facts = list(features.columns)
Spit into SK Learn training and test set:
# Using Skicit-learn to split data into training and testing sets from sklearn.model_selection import train_test_split # Split the data into training and testing sets train_features, test_features, train_labels, test_labels = train_test_split(features, labels, test_size = 0.25, random_state = 42) test_features.head(5) test_features.head(5)
Then I explore the data:
print('Training Features Shape:', train_features.shape) print('Training Labels Shape:', train_labels.shape) print('Testing Features Shape:', test_features.shape) print('Testing Labels Shape:', test_labels.shape)
Training Features Shape: (128304, 337) Training Labels Shape: (128304,) Testing Features Shape: (42768, 337) Testing Labels Shape: (42768,)
First we try training XGBoost model. I needed so push # estimates above 10,000 to get a decent accuracy (R2 > 0.94)
# Import the model we are using from xgboost import XGBRegressor from sklearn.ensemble import RandomForestRegressor from sklearn import ensemble from sklearn.metrics import mean_squared_error x = XGBRegressor(random_state = 44, n_jobs = 8, n_estimators = 10000, max_depth=10, verbosity = 3) x.fit(train_features, train_labels) print('xgboost train score: ', x.score(train_features, train_labels)) predictions = x.predict(test_features) print('xgboost test score: ', x.score(test_features, test_labels))
Here, train and test score: R2 ~0.94.
2. Random Forest Regressor
Then we try Random Forest model. After some fiddling it appears 100 estimators is enough to get a pretty good accuracy (R2 > 0.94)
# Instantiate model with 100 decision trees rf = RandomForestRegressor(n_estimators = 100, criterion='mse', verbose=1, random_state = np.random.RandomState(42), n_jobs = -1) # Train the model on training data rf.fit(train_features, train_labels); print('random forest train score: ', rf.score(train_features, train_labels)) predictions = rf.predict(test_features) print('random forest test score: ', rf.score(test_features, test_labels))
In this case, train and test score R2 at ~0.94.
3. MLP Neural Network
Finally I wanted to compare performance to an MLP Regressor. According to the books I have been reading on deep learning, a neural network should be able to outperform any shallow learning algorithm given enough time and horse power. I am using a pretty beefy machine with 8 cores and 30 GB RAM on Google Cloud.
from sklearn.neural_network import MLPRegressor nn = MLPRegressor(hidden_layer_sizes=(338, 338, 50), activation='relu', solver='adam', max_iter = 100, random_state = 56, verbose = True) nn.fit(train_features, train_labels) nn_predictions = nn.predict(test_features) print('nn train score: ', nn.score(train_features, train_labels)) print('nn test score: ', nn.score(test_features, test_labels))
R2 around 0.4.
I've used 338 neurons in the input layer as this is the exact number of columns. The neural network stalls at 82 iterations and doesn't go any further. Running with 50 iterations I get accuracy at R2 < 0.5, which is not impressive.
Do I handle the management of categorical features correctly? With the scores above, does it look like model #1 and #2 are overfitting? R2 > 0.94 is pretty good and both test and training accuracy look good and in same ballpark, so I don't think it is overfitting
Why does the neural network not perform that well?
Should I consider a different type of neural network for regression?
Why do I have to add so many more estimators to XGBoost (10,000) to get the same performance as Random Forest (100)?
What would be a fit for purpose neural network to solve this problem with deep learning? I am concerned the ensemble methods may not be appropriate or that I am doing something wrong.