I am a newbie to data science and statistics. I came across this problem, which has 50 independent variables and one dependent variable and trying to identify the good regression technique to start with. The following is the flow chart that I executed:

Data Exploration -> Correlational matrix -> dimension reduction -> PCA (Dimension reduction) -> Basic Linear Regression technique.

Can someone guide me, if there is any other better technique or procedure.


In no way is this going to be an exhaustive answer, but it will definitely give you a starting point in Python -

Data Exploration

Start with Pandas Profiling. It will give you HTML reports of your variables. If the quality of the data is good, it will provide some insights into the fill rate, depending upon the variable type some statistics for each variable

Correlational matrix

The pandas profiling report includes the coorelation matrix. But if you are looking to compute by hand, use pd.corr(). You can vary parameters to get different correlation metrics like ‘pearson’, ‘kendall’, ‘spearman’

Dimension Reduction -> PCA (Dimension reduction)

There are many ways to do this. Keep in mind if you are looking for accuracies only and don't care about how X is influencing y, (1) is an optional step (applies to (2) as well).

  1. Analyse the correlation matrix and use VIF to dump variables with high correlation
  2. Factor Analysis / PCA for dimensionality reduction
  3. Use LASSO to fit a model, check the coefficients and the ones that are 0 or going to 0 can be thought of as weak indicators and can be eliminated.
  4. Keep all 50, and use Ridge Regression and vary the alpha parameter to fine-tune accuracy (or whatever metric you are trying to optimize)
  5. If the model still doesn't seem to be stable, try to cook non-linear features with sklearn's Polynomial Features, regularize and repeat.
  6. Probably the most important in the real world, ask the domain expert on what he/she thinks might be the important variables

Basic Linear Regression technique

  1. Playing with hyperparamters to get good cross-validation/test score is the key here for a basic Linear Regression model.
  2. Try as many techniques as you can from here and here
  • $\begingroup$ Thanks @Vivek. The answer gives a clear picture and a good start point. $\endgroup$ – user86752 Nov 7 '18 at 16:18

Not the answer you're looking for? Browse other questions tagged or ask your own question.