Handling Covariate Shift and Multi-collinearity in same Dataset

Question

The problem is related to Regression problem.

I am getting batches of data from a source of experiment which has approx 3k columns. However, I observed that almost 99% of the columns are highly correlated to each other.

The dataset looks something like this, where I have to predict the Y-variable based on the numerical features provided:

ID	1	2	3	....	3000	3001	Y
1	500	510	520	....	67800	68900	0.12
2	700	710	720	....	72800	76900	0.13
3	950	960	967	....	74800	78900	0.52
4	989	992	999	....	87800	88900	0.44

However, the correlation between the variables are extremely high. For ex. columns 100-500 will have a correlation of 0.9997-0.9998 or even 1 between them and again 1000-2000 will have very high correlation and so on.

Interesting fact is as the columns keep on increasing the higher valued column name will have lower correlation with lower valued column name, i.e. 100 will have very high correlation till 1500 let's say but will have low correlation with columns like 2800/3000 and similar for higher columns, i.e. columns 2800/3000 will have very high correlation with columns 2000/2500 but lower correlation with extreme left columns.

I am using the following piece of code to remove the columns and solve multi-collinearity:

def correlation(dataset, threshold):
    col_corr = set() # Set of all the names of deleted columns
    corr_matrix = dataset.corr()
    for i in range(len(corr_matrix.columns)):
        for j in range(i):
            if (corr_matrix.iloc[i, j] >= threshold) and (corr_matrix.columns[j] not in col_corr):
                colname = corr_matrix.columns[i] # getting the name of column
                col_corr.add(colname)
                if colname in dataset.columns:
                    del dataset[colname] # deleting the column from the dataset

    return dataset

Ref: https://stackoverflow.com/questions/29294983/how-to-calculate-correlation-between-all-columns-and-remove-highly-correlated-on

When I am running this piece of code, the total no. of feature columns reduce to 3-4 from 3000.

But, the problem is, when I am running this function on my train set, the no. of columns that I get is different from the columns I am getting from my validation set. So, when I try to keep the same columns from my train in validation, I see that the column values distribution has changed by a lot (covariate shifting) and also the correlation has changed.

For ex. after running the above piece of code, let's say I am left with columns 100, 1700 and 3000, the correlation between these variables in my train set is completely different from my validation set. In my train set the correlation between column 100 and 3000 is 0.29 where as in my validation set it is -0.13.

I really need some help in understanding how can I tackle these type of problems, because this is something completely new to me.

Answer 1

What happens in validation has a good chance to happen in production.

That's why reducing the features to the minimum might not be the best solution, even if it is mathematically valid, because of the distribution problem you've mentioned.

I suggest correcting the covariate shift by applying the ratio ri=pi(x)/qi(x) for each column and applying it to your loss function.

The idea is to have similar distributions in order to reach eventually similar correlations.