1
$\begingroup$

I have a dataset with 4 predictor variables X1, X2, X3, X4, and one response variable Y. I have been asked to check the correlation between these variables and see how they are related and then use linear model to fit them.

No split of training set:test set is given. I have one data set with 10000 samples. I was planning of splitting this data set in the ratio 80:20 for training and testing respectively.

Now I have a doubt on whether the correlation should be found after the data has been split or is it better to check correlation with the entire dataset? Which is the std way of doing?

P.S. I am going to use the R programming to do the same.

$\endgroup$
2
$\begingroup$

This depends on your research question. Do you want to make predictions? - Then you need to split your data set in training and testsamples.

However, if you are more interested in answering the following question: What impact does X1, X2, X3 and X4 have on Y, respectively? Then you are interested in estimation. For this, you do not need to split your sample, but have to test your data set for underlying model assumptions (e.g. heteroscedasticity, autocorrellation of the residuals etc.) to get unbiased/accurate estimations.

For OLS (linear regression) these model assumptions follow the Gauss-Markov-Theorem.

Most statistical tests for model assumptions are already implemented in R:

  • Test for autocorrelation - Breusch-Godfrey - bgtest
  • Test for heteroscedasticity - White's Test - het.test
  • Test for normality - Jarque Bera Test - jarque.bera.test

However, even the tests make some assumptions about your data - but these tests are the most common ones.

UPDATE - Example for clarification

According to your comment. Imagine this real-life problem. You want to build a model that can predict the chance of cancer (y), using variables like age, blood pressure, weight, cholesterol value etc.. (X1, X2, X3, X4)

To assess the nature of the variables and their relationships you do descriptive analysis (Mean, Variance etc.) + Correlation analysis.

Let's have a look at the following data. (Yellow records would be your test samples - the data you DON'T have in real life, as those patients haven't seen the doctor yet).

As you can see the mean and variance can differ already a lot depending on the fact if you include test samples or not in your calculations.

Data Set + Descriptive Analysis

Now we are coming to the correlation part - the relationship between the independent variables or also called predictors

Correlation Analysis

I highlighted the relationship between weight and cholesterol. Without the test sample, it seems like weight and cholesterol are uncorrelated or have only a slightly positive relationship. If we add the test data it turns out that the correlation turns negative.

Question: If the correlation between your variables would have an impact on your choice of variable selection for your model, would it make sense to include the test-sample in your correlation analysis? Especially, knowing that this data is not available in real life yet.

Remember Model Estimation: if you build a multivariate regression model mean, variance and covariance are used to find the best parameters to estimate your dependent variable (Cancer). So a model that was already trained with all available data is likely to make better predictions as it has already seen the data it should predict.

Summary

Whenever you plan to make predictions with a model you should work as closely to real-life assumptions. So you split your data in train - test data. You ònly use the training data to perform all your tests and checks and you will not touch the test set before making a prediction.

| improve this answer | |
$\endgroup$
  • $\begingroup$ I want to find the association/relationship between the predictors (x1,x2,x3,x4) as well as its effect on the response y. To do the same, I'd like to use correlation. My question is whether to use the full data set or use it after splitting. (Training80:Testing20). After this, my end goal is to eventually build a prediction model. But currently I was just asking about which dataset to use for correlation. Hope I am clear. $\endgroup$ – Selvam Feb 20 at 13:09
  • $\begingroup$ As SUN and I pointed out - if you make predictions you always should seperate training (estimation of your model) from testing (forecasting) - If you perform correlation analysis to determine the relationship between your variables and use this analysis to base your decision if you ex- or include certain variables it is important to seperate the data sets as you do not know (e.g.) the mean and variance for the whole population beforehand. So you will use the training set for your correlation analysis. $\endgroup$ – Maeaex1 Feb 20 at 16:06
  • $\begingroup$ Sorry, I'm a newbie to this ML thingy. Can you please clarify why I won't know the mean and variance for whole population beforehand? I have been given the entire dataset using which I should be able to find the mean and variance right? Also, The suggestion you and SUN are providing seem to be contradicting. SUN has suggested that I should consider the full data set for correlation analysis. $\endgroup$ – Selvam Feb 21 at 5:19
2
$\begingroup$

I believe you want to evaluate the association between variables so it's better to perform correlation on the full dataset before splitting. Also, it will help to select the feature variable to prevent data leakage. https://towardsdatascience.com/preventing-data-leakage-in-your-machine-learning-model-9ae54b3cd1fb

| improve this answer | |
$\endgroup$
  • $\begingroup$ Yes, you understood my requirements correctly. But is it how its usually done? Assume I am anyways going to use all the variables. Not going to exlude any variable based on correlation (Assuming all are well correlated variables). What would you suggest in this case? $\endgroup$ – Selvam Feb 21 at 5:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.