# Handling Covariate Shift and Multi-collinearity in same Dataset

The problem is related to Regression problem.

I am getting batches of data from a source of experiment which has approx 3k columns. However, I observed that almost 99% of the columns are highly correlated to each other.

The dataset looks something like this, where I have to predict the Y-variable based on the numerical features provided:

ID 1 2 3 .... 3000 3001 Y
1 500 510 520 .... 67800 68900 0.12
2 700 710 720 .... 72800 76900 0.13
3 950 960 967 .... 74800 78900 0.52
4 989 992 999 .... 87800 88900 0.44

However, the correlation between the variables are extremely high. For ex. columns 100-500 will have a correlation of 0.9997-0.9998 or even 1 between them and again 1000-2000 will have very high correlation and so on.

Interesting fact is as the columns keep on increasing the higher valued column name will have lower correlation with lower valued column name, i.e. 100 will have very high correlation till 1500 let's say but will have low correlation with columns like 2800/3000 and similar for higher columns, i.e. columns 2800/3000 will have very high correlation with columns 2000/2500 but lower correlation with extreme left columns.

I am using the following piece of code to remove the columns and solve multi-collinearity:

def correlation(dataset, threshold):
col_corr = set() # Set of all the names of deleted columns
corr_matrix = dataset.corr()
for i in range(len(corr_matrix.columns)):
for j in range(i):
if (corr_matrix.iloc[i, j] >= threshold) and (corr_matrix.columns[j] not in col_corr):
colname = corr_matrix.columns[i] # getting the name of column
if colname in dataset.columns:
del dataset[colname] # deleting the column from the dataset

return dataset


When I am running this piece of code, the total no. of feature columns reduce to 3-4 from 3000.

But, the problem is, when I am running this function on my train set, the no. of columns that I get is different from the columns I am getting from my validation set. So, when I try to keep the same columns from my train in validation, I see that the column values distribution has changed by a lot (covariate shifting) and also the correlation has changed.

For ex. after running the above piece of code, let's say I am left with columns 100, 1700 and 3000, the correlation between these variables in my train set is completely different from my validation set. In my train set the correlation between column 100 and 3000 is 0.29 where as in my validation set it is -0.13.

I really need some help in understanding how can I tackle these type of problems, because this is something completely new to me.

• Is it possible to have a larger validation dataset? Does the validation data set have the same high correlation levels as the training one? Jul 7, 2022 at 18:20
• It is not possible to have a higher validation because of certain limitations while doing the experiment. However, the correlation between the variables for both train and validation are highly correlated amongst each other, i.e. multi-collinearity exists. However, the distribution of the value changes. For ex. if for col A I have the values ranging b/w 0-100 then in validation they are changing to 500-1000. This is for other columns as well, hence multi-collinearity is consistent but no distribution Jul 7, 2022 at 18:34