Suppose that our data matrix X has a duplicated column, i.e, there is a duplicated feature and the matrix is not full column rank. What happpens?

I guess that we can not find a unique solution because that's the case for the close form in linear regression, but I do not see how to show that intuitively, or even if it is true or not.

  • $\begingroup$ In what context? What kinds of model? $\endgroup$ – Ben Reiniger Jan 26 at 18:23
  • $\begingroup$ Linear regression $\endgroup$ – vicase98 Jan 26 at 18:39

In 'Efficient Backprop' by Lecun and others (http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf), they explain why correlated variables are bad (§ 4.3 normalizing the inputs).

Duplicated data is a special case of linear dependence, which is a special case of correlation. Say you have duplicated variables $X1 = X2$, so the network output is constant over the weight lines $w_2 = c - w_1$, where c is a constant. It means that the gradient of the error is 0 along those lines : moving along those lines has no effect on the learning. In that sense the solution won't be unique.

It's bad because you could be solving a problem of lesser dimension. Removing one of the input will leave you with a network of lesser dimension.

Other than the dimension problem, for simple learner I don't think it will affect the learning process that much. For more complex learning processes (learning rate depending on time / on variables), it might get more complex.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.