In no way is this going to be an exhaustive answer, but it will definitely give you a starting point in
Pandas Profiling. It will give you HTML reports of your variables. If the quality of the data is good, it will provide some insights into the fill rate, depending upon the variable type some statistics for each variable
The pandas profiling report includes the coorelation matrix. But if you are looking to compute by hand, use
pd.corr(). You can vary parameters to get different correlation metrics like
‘pearson’, ‘kendall’, ‘spearman’
Dimension Reduction -> PCA (Dimension reduction)
There are many ways to do this. Keep in mind if you are looking for accuracies only and don't care about how
X is influencing
y, (1) is an optional step (applies to (2) as well).
- Analyse the correlation matrix and use
VIF to dump variables with high correlation
- Factor Analysis / PCA for dimensionality reduction
- Use LASSO to fit a model, check the coefficients and the ones that are
0 or going to
0 can be thought of as weak indicators and can be eliminated.
- Keep all 50, and use Ridge Regression and vary the alpha parameter to fine-tune accuracy (or whatever metric you are trying to optimize)
- If the model still doesn't seem to be stable, try to cook non-linear features with sklearn's Polynomial Features, regularize and repeat.
- Probably the most important in the real world, ask the domain expert on what he/she thinks might be the important variables
Basic Linear Regression technique
- Playing with hyperparamters to get good cross-validation/test score is the key here for a basic Linear Regression model.
- Try as many techniques as you can from here and here