Suppose that our data matrix X has a duplicated column, i.e, there is a duplicated feature and the matrix is not full column rank. What happpens?
I guess that we can not find a unique solution because that's the case for the close form in linear regression, but I do not see how to show that intuitively, or even if it is true or not.
In 'Efficient Backprop' by Lecun and others (http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf), they explain why correlated variables are bad (§ 4.3 normalizing the inputs).
Duplicated data is a special case of linear dependence, which is a special case of correlation. Say you have duplicated variables $X1 = X2$, so the network output is constant over the weight lines $w_2 = c - w_1$, where c is a constant. It means that the gradient of the error is 0 along those lines : moving along those lines has no effect on the learning. In that sense the solution won't be unique.
It's bad because you could be solving a problem of lesser dimension. Removing one of the input will leave you with a network of lesser dimension.
Other than the dimension problem, for simple learner I don't think it will affect the learning process that much. For more complex learning processes (learning rate depending on time / on variables), it might get more complex.
When you duplicate a feature, you may have to decrease the learning rate to avoid oscillation or divergence, even for a simple model like linear regression. If we have a one-dimensional input as X and duplicate the feature there, the gradient will be about twice as large than it would otherwise be, and if our learning rate is, say, 0.05, the new step size may be large enough to prevent gradient descent from converging.
Even if gradient descent does not diverge, training will be slower. See https://towardsdatascience.com/feature-selection-why-how-explained-part-1-c2f638d24cdb, or this answer by Winks (https://stats.stackexchange.com/a/191364/267884):
Gradient Descent works 'best' when the direction of the gradient at each iteration points to the optimal point; that is, you could minimize each $\beta_i$ [a weight that the model learns] separately and get to a good answer. This is possible when the function to optimize is strictly convex. But when inputs are highly correlated, this is no longer the case. Obviously, it is not possible for neural networks since the function is not convex to begin with, but it has effects on reaching the local minimum as well.
