1
$\begingroup$

Background

I'm trying to predict the value of website visitors. Only a small fraction of the visitors actually make a purchase, so ~97% of the visits has the value of 0, while about 2-3% has values between 1 and 1000 (median at about 50, 95% percentile at 150). The number of samples are a few millions.

The kind of feature variables that I have are things like country, device info (e.g. iphone, android, desktop computer), browser version, time of visit, day of week, which page they entered on, etc. From these variables it's impossible to predict, with a high degree of certainty, if the user is going to make a purchase, so the predicted value for each visitor should be a fairly low value (somewhere between 0.2 and 4). However by looking at the data it's clear that the feature variables does affect the average value of the visitors (e.g. users on desktop are 25% more likely to make a purchase than mobile users). The predicted values are used in ad bidding.

I started out with a "model" where I simply predicted each visitor's value to the average visitor value. From there I moved on to an extremely basic, naive non ML model where I simply just modify the average value based on just a few feature variables that I've seen has a large impact (on average). E.g. make the desktop visitors worth 25% more than mobile visitors, and increase the value of visitors on week-ends by 10%. I empirically saw clear improvements from using this naive model instead of just using the mean value for each visitor. However, this obviously has problems since the different features might be correlated, and I would also like to be able to use as many features as possible that might improve the predictions.

Therefore I figured some kind of ML-model should be able to perform better than my very naive model would. I decided to start with a Random Forest regression model, since it seemed like a good model to start with (supposed to work fairly good for many different problems, has few hyper parameters to tune, works with highly correlated features, and requires little preprocessing).

The question(s)

I've done quite a bit of feature engineering and I'm now able to train a random forest model. However, I'm not sure how I can best evaluate the performance of my trained model (except for testing it out empirically, which takes quite a bit of time so it's not feasible just for tuning hyper parameters).

Since the predicted values are quite low (around 1) while the actual, rare purchases are high (around 50), the R2 score that I get for my test set is extremely low. After some hyper parameter tuning (n_estimators=300, min_samples_leaf=100, max_features=3, min_samples_split=5) the best R2 score I get for the test set is ~0.006. The R2 score for my naive "model" for the same test set is ~0.003.

  • Does this mean that my RF model performs much better than the naive one?
  • Does it make any sense to look at R2 score when it's so low?
  • Is there any other method of evaluating the model that would better suit my case?

I'd think an R2 score of 0.003 would indicate that the model is quite garbage, however I know empirically that my naive model yields significantly better results for our use-case than just predicting the mean.

I've tried to provide as much (possibly) useful info as possible. I'm a data science novice.

$\endgroup$
  • $\begingroup$ Have you seen this scikit-learn.org/stable/tutorial/machine_learning_map/… essentially are you sure you want to be usign a RF, wouldn't you be better off with a regressor? $\endgroup$ – Tasty213 Aug 30 '19 at 13:59
  • $\begingroup$ @Tasty213 Thanks for the reply! Yes, I have seen that graph. I think my main question on how to best evaluate a trained model (does it make sense to look at R2 score?) would still be as relevant no matter the ML algorithm I use. I decided to start with Random Forest since it's supposed to work fairly good for many different problems, has few hyper parameters to tune, works with highly correlated features, and requires less preprocessing (no scaling, one hot encoding etc.). $\endgroup$ – fibble Aug 30 '19 at 15:16
  • $\begingroup$ $R^2$ is a measure of the correlation between your predictions and the true values, so a score of 0.006 compared to 0.003 does imply that your new model is performing better than the naive one, but only just. There are other metrics you can use to evaluate (like RMSE for instance), but the main improvements are in their explanatory value rather than telling you anything different about the goodness of fit. 0.003 is, however, pretty darn low; I'd be surprised if that can't be increased. As an interesting exercise, have you calculated the $R^2$ score for just predicting the mean? $\endgroup$ – Dan Scally Sep 13 '19 at 8:49
  • $\begingroup$ @DanScally Thanks for the reply! Just predicting the mean gives me the R2 score 0.0. $\endgroup$ – fibble Sep 17 '19 at 11:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.