1
$\begingroup$

The problem is related to Regression problem.

I am getting batches of data from a source of experiment which has approx 3k columns. However, I observed that almost 99% of the columns are highly correlated to each other.

The dataset looks something like this, where I have to predict the Y-variable based on the numerical features provided:

ID 1 2 3 .... 3000 3001 Y
1 500 510 520 .... 67800 68900 0.12
2 700 710 720 .... 72800 76900 0.13
3 950 960 967 .... 74800 78900 0.52
4 989 992 999 .... 87800 88900 0.44

However, the correlation between the variables are extremely high. For ex. columns 100-500 will have a correlation of 0.9997-0.9998 or even 1 between them and again 1000-2000 will have very high correlation and so on.

Interesting fact is as the columns keep on increasing the higher valued column name will have lower correlation with lower valued column name, i.e. 100 will have very high correlation till 1500 let's say but will have low correlation with columns like 2800/3000 and similar for higher columns, i.e. columns 2800/3000 will have very high correlation with columns 2000/2500 but lower correlation with extreme left columns.

I am using the following piece of code to remove the columns and solve multi-collinearity:

def correlation(dataset, threshold):
    col_corr = set() # Set of all the names of deleted columns
    corr_matrix = dataset.corr()
    for i in range(len(corr_matrix.columns)):
        for j in range(i):
            if (corr_matrix.iloc[i, j] >= threshold) and (corr_matrix.columns[j] not in col_corr):
                colname = corr_matrix.columns[i] # getting the name of column
                col_corr.add(colname)
                if colname in dataset.columns:
                    del dataset[colname] # deleting the column from the dataset

    return dataset

Ref: https://stackoverflow.com/questions/29294983/how-to-calculate-correlation-between-all-columns-and-remove-highly-correlated-on

When I am running this piece of code, the total no. of feature columns reduce to 3-4 from 3000.

But, the problem is, when I am running this function on my train set, the no. of columns that I get is different from the columns I am getting from my validation set. So, when I try to keep the same columns from my train in validation, I see that the column values distribution has changed by a lot (covariate shifting) and also the correlation has changed.

For ex. after running the above piece of code, let's say I am left with columns 100, 1700 and 3000, the correlation between these variables in my train set is completely different from my validation set. In my train set the correlation between column 100 and 3000 is 0.29 where as in my validation set it is -0.13.

I really need some help in understanding how can I tackle these type of problems, because this is something completely new to me.

Improve this question
$\endgroup$
2
  • $\begingroup$ Is it possible to have a larger validation dataset? Does the validation data set have the same high correlation levels as the training one? $\endgroup$
    – Nicolas Martin
    Commented Jul 7, 2022 at 18:20
  • $\begingroup$ It is not possible to have a higher validation because of certain limitations while doing the experiment. However, the correlation between the variables for both train and validation are highly correlated amongst each other, i.e. multi-collinearity exists. However, the distribution of the value changes. For ex. if for col A I have the values ranging b/w 0-100 then in validation they are changing to 500-1000. This is for other columns as well, hence multi-collinearity is consistent but no distribution $\endgroup$
    – Debadri Dutta
    Commented Jul 7, 2022 at 18:34

1 Answer 1

Reset to default
0
$\begingroup$

What happens in validation has a good chance to happen in production.

That's why reducing the features to the minimum might not be the best solution, even if it is mathematically valid, because of the distribution problem you've mentioned.

I suggest correcting the covariate shift by applying the ratio ri=pi(x)/qi(x) for each column and applying it to your loss function.

The idea is to have similar distributions in order to reach eventually similar correlations.

Improve this answer
$\endgroup$
4
  • $\begingroup$ what is pi(x) and qi(x), unaware of this! can you please explain? $\endgroup$
    – Debadri Dutta
    Commented Jul 8, 2022 at 4:24
  • $\begingroup$ They are your probability distributions. For instance: p3(720<x<1200) = 0.2. datacamp.com/tutorial/probability-distributions-python $\endgroup$
    – Nicolas Martin
    Commented Jul 8, 2022 at 6:30
  • $\begingroup$ so implementation wise do you mean to say I need to apply this factor as weights to each column? $\endgroup$
    – Debadri Dutta
    Commented Jul 8, 2022 at 11:04
  • $\begingroup$ Sorry, I don’t know exactly because it is just an idea to find a good solution. $\endgroup$
    – Nicolas Martin
    Commented Jul 8, 2022 at 14:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.