Which data set to use to find correlation between Predictor and response variables? Test data set? Training data set? or the entire data set?

Question

I have a dataset with 4 predictor variables X1, X2, X3, X4, and one response variable Y. I have been asked to check the correlation between these variables and see how they are related and then use linear model to fit them.

No split of training set:test set is given. I have one data set with 10000 samples. I was planning of splitting this data set in the ratio 80:20 for training and testing respectively.

Now I have a doubt on whether the correlation should be found after the data has been split or is it better to check correlation with the entire dataset? Which is the std way of doing?

P.S. I am going to use the R programming to do the same.

Answer 1

This depends on your research question. Do you want to make predictions? - Then you need to split your data set in training and testsamples.

However, if you are more interested in answering the following question: What impact does X1, X2, X3 and X4 have on Y, respectively? Then you are interested in estimation. For this, you do not need to split your sample, but have to test your data set for underlying model assumptions (e.g. heteroscedasticity, autocorrellation of the residuals etc.) to get unbiased/accurate estimations.

For OLS (linear regression) these model assumptions follow the Gauss-Markov-Theorem.

Most statistical tests for model assumptions are already implemented in R:

• Test for autocorrelation - Breusch-Godfrey - bgtest
• Test for heteroscedasticity - White's Test - het.test
• Test for normality - Jarque Bera Test - jarque.bera.test

However, even the tests make some assumptions about your data - but these tests are the most common ones.

UPDATE - Example for clarification

According to your comment. Imagine this real-life problem. You want to build a model that can predict the chance of cancer (y), using variables like age, blood pressure, weight, cholesterol value etc.. (X1, X2, X3, X4)

To assess the nature of the variables and their relationships you do descriptive analysis (Mean, Variance etc.) + Correlation analysis.

Let's have a look at the following data. (Yellow records would be your test samples - the data you DON'T have in real life, as those patients haven't seen the doctor yet).

As you can see the mean and variance can differ already a lot depending on the fact if you include test samples or not in your calculations.

Now we are coming to the correlation part - the relationship between the independent variables or also called predictors

I highlighted the relationship between weight and cholesterol. Without the test sample, it seems like weight and cholesterol are uncorrelated or have only a slightly positive relationship. If we add the test data it turns out that the correlation turns negative.

Question: If the correlation between your variables would have an impact on your choice of variable selection for your model, would it make sense to include the test-sample in your correlation analysis? Especially, knowing that this data is not available in real life yet.

Remember Model Estimation: if you build a multivariate regression model mean, variance and covariance are used to find the best parameters to estimate your dependent variable (Cancer). So a model that was already trained with all available data is likely to make better predictions as it has already seen the data it should predict.

Summary

Whenever you plan to make predictions with a model you should work as closely to real-life assumptions. So you split your data in train - test data. You ònly use the training data to perform all your tests and checks and you will not touch the test set before making a prediction.

Answer 2

I believe you want to evaluate the association between variables so it's better to perform correlation on the full dataset before splitting. Also, it will help to select the feature variable to prevent data leakage. https://towardsdatascience.com/preventing-data-leakage-in-your-machine-learning-model-9ae54b3cd1fb