Interpreting Results of Multivariable Regression / how to transform variables to improve results

Question

I am working on a project that predicts the Market Cap (value) of different crypto-currencies. My data is very small (51 observations) and I initially have 18 X-variables. I was hoping to get feedback on my modeling approach and results, and suggestions on improving the model (particularly by transforming variables / with a non-linear regression) technique. I will do my best to keep the post clear and brief, and hope it can be helpful to others working on a similar analysis. The post may seem long, but a lot of it is images. Also, I am doing this in R, and can share any code on request.

1.) My first action was to log-transform the Market Cap (Y-variable) into log(Market Cap). Here are 2 graphs, of Market Cap and of log(Market Cap) of my 50 observations:

...the outlier point, with a $190B market cap, is bitcoin, and the model badly overfit to this data point if I did not log-transform. For this reason, I think this first action of log-transforming the Market Cap is a good action.

2.) 12 of the 50 observations were missing values for 6 of the X-variables. Rather than throwing these observations away, I predict missing values for these 6 X-variables by fitting, for each of these 6 X-variables, a simple linear regression between the (a) the X-variable with missing values, and (b) one of the remaining 12 X-variables with no missing values. For (b), I chose the X-variable with the highest pearson correlation to the missing-value-X-variable. I use this simple linear regression to predict missing values.

3.) I then fit an initial linear model with all 18 variables. However, since 18 variables will almost certainly overfit to 50 data points, and because there is some multicollinearity to the X-variables, I then use stepwise regression with backward elimination to fit a model with fewer variables. The R summary output of the model returned using backward elimination, with 9 variables, is here:

and the R diagnostics plots from this model are:

4.) Lastly, here is a grid with variable distributions and correlations for these 9 predictor variables + the log(Market Cap):

as well as a table of Variance Inflation Factors

My thoughts on next steps are to fix the remaining multicollinearity issue by removing additional variables, and also remove some outlier data points indicated in the Residuals vs. Leverage graph. I also plan to test the model on a test data-set (I did a 40 / 10 split), 5 times using 5-fold cross validation.

I am interested in anyone's thoughts on transforming the X-variables, which I currently do none of. The grid of histograms / correlations / scatter plots shows that many variables are right-skewed. Additionally, I have graphed the simple linear regression fit between log(Market Cap) and each of the 9 remaining variables, and received these plots (for 4 of the 9):

And noticed for the most part that none of the X-variables seem to have a great linear-fit with log(Market Cap).

Any thoughts on next-steps on my model building would be greatly, greatly appreicated. Apologies again for the long post, but I felt that thorough / lots of images / plots would be helpful here. Thanks!

Answer 1

Your VIF values are extreme, and your residuals plot is a right-opening megaphone (vice a null plot, which is what you want). It looks like the residual plot is being strongly affected by BTC.

If I were working this I would look into Principal Component Analysis (PCA) for two reasons:

1) It can help mitigate your multicollinearity and drive down your VIFs

2) It can reduce the number of predictor variables you're using (linear regression is better on fewer variables)

I would also consider removing the BTC data point just to see what happens. This can give you some insights that can help you build the model. Good luck!

Answer 2

Given,

1. You have a very small training sample which does not let you test on a hold out to a large extent
2. You are looking to do some feature selection
3. You are looking to reduce multi-collinearity

This could be a good use case for LASSO. It would help with removal of multi-collinearity in your models along with assisting in feature selection. You could try a 5-fold cross validation on the complete sample. As the model performs coefficient shrinkage, it should also generalize better on hold-out samples in case your other models seem to be over-fitting.