I have a question about data cleaning. I am a novice and have just started learning in this field so please pardon my ignorance. Suppose there are two columns and based on some samples taken from both the columns you find the correlation coefficient to be high. Now for the values that aren't there, can you use linear regression to predict or find them out, by using the values you know as training data?
1
-
1$\begingroup$ Yes, as explained by Donald. Also, check - KNNImputer $\endgroup$– 10xAIJun 29, 2020 at 3:42
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
5
Hi Soumyadeep and welcome to Data Science/Stack Exchange
What you are describing is called regression imputation, and it is a valid method to use on missing data. However, if the data is sparse (lots of missing values), this issue will be more difficult to handle.
In general, missing data can be handled in several ways (row deletion, imputation, substitution, etc). Regression imputation can be used if you have little or no knowledge about the data, but usually it is better to use another method. If you have some domain knowledge about the missing values, like you have an idea what the value should be, usually you can use that knowledge to fill in the missing values. Try some different methods and see which one works best.
A person pointed out that I should check for multicollinearity if the features are both independent. Does it basically mean that one feature is falling in the span of the other feature?
Definition of multicollinearity: There exist one or more exact linear relationships among some of the variables
References: https://en.wikipedia.org/wiki/Multicollinearity https://stats.stackexchange.com/questions/234870/is-multicollinearity-the-issue-here
-
$\begingroup$ Hello Donald, thank you for the answer. Greetings! Now I have a follow up question. A person pointed out that I should check for multicollinearity if the features are both independent. Does it basically mean that one feature is falling in the span of the other feature? $\endgroup$ Jul 22, 2020 at 0:32
-
$\begingroup$ For the independent features, you should check for multicollinearity of the 2 columns you mention in your question. This does not necessarily mean the features will fall in the span (think you are referring to range of values) of the other feature. High multicollinearity means multiple features are highly correlated to or dependent on each other. $\endgroup$– Donald SJul 22, 2020 at 4:20
-
$\begingroup$ More importantly, for imputation, you want a high correlation so the imputation can be more accurate. However, when you build your model, you want to use some L1 or L2 regularization to counteract the issues that this multicollinearity or correlation will cause. If the correlation is high, you may need a higher regularization factor to avoid overfitting. Many models will include regularization(s) internally, so you only need to adjust the value of the regularization factors. $\endgroup$– Donald SJul 22, 2020 at 4:20
-
$\begingroup$ Thank you for your answer, again! Given my limited knowledge I am yet to completely understand the idea that you are trying to express. I am glad that you helped me out in this regards. I guess I was under the impression that multicollinearity was somewhat analogous to the vectors being coplanar. $\endgroup$ Jul 22, 2020 at 15:32
-
$\begingroup$ I added some feedback in my answer above about your comment on multicollinarity and if vectors are coplanar - they should be collinear. $\endgroup$– Donald SJul 23, 2020 at 1:08