I'm trying to predict the value of website visitors. Only a small fraction of the visitors actually make a purchase, so ~97% of the visits has the value of 0, while about 2-3% has values between 1 and 1000 (median at about 50, 95% percentile at 150). The number of samples are a few millions.
The kind of feature variables that I have are things like country, device info (e.g. iphone, android, desktop computer), browser version, time of visit, day of week, which page they entered on, etc. From these variables it's impossible to predict, with a high degree of certainty, if the user is going to make a purchase, so the predicted value for each visitor should be a fairly low value (somewhere between 0.2 and 4). However by looking at the data it's clear that the feature variables does affect the average value of the visitors (e.g. users on desktop are 25% more likely to make a purchase than mobile users). The predicted values are used in ad bidding.
I started out with a "model" where I simply predicted each visitor's value to the average visitor value. From there I moved on to an extremely basic, naive non ML model where I simply just modify the average value based on just a few feature variables that I've seen has a large impact (on average). E.g. make the desktop visitors worth 25% more than mobile visitors, and increase the value of visitors on week-ends by 10%. I empirically saw clear improvements from using this naive model instead of just using the mean value for each visitor. However, this obviously has problems since the different features might be correlated, and I would also like to be able to use as many features as possible that might improve the predictions.
Therefore I figured some kind of ML-model should be able to perform better than my very naive model would. I decided to start with a Random Forest regression model, since it seemed like a good model to start with (supposed to work fairly good for many different problems, has few hyper parameters to tune, works with highly correlated features, and requires little preprocessing).
I've done quite a bit of feature engineering and I'm now able to train a random forest model. However, I'm not sure how I can best evaluate the performance of my trained model (except for testing it out empirically, which takes quite a bit of time so it's not feasible just for tuning hyper parameters).
Since the predicted values are quite low (around 1) while the actual, rare purchases are high (around 50), the R2 score that I get for my test set is extremely low. After some hyper parameter tuning (n_estimators=300, min_samples_leaf=100, max_features=3, min_samples_split=5) the best R2 score I get for the test set is ~0.006. The R2 score for my naive "model" for the same test set is ~0.003.
- Does this mean that my RF model performs much better than the naive one?
- Does it make any sense to look at R2 score when it's so low?
- Is there any other method of evaluating the model that would better suit my case?
I'd think an R2 score of 0.003 would indicate that the model is quite garbage, however I know empirically that my naive model yields significantly better results for our use-case than just predicting the mean.
I've tried to provide as much (possibly) useful info as possible. I'm a data science novice.