Are dimensionality reduction techniques useful in deep learning

Question

I have been working on machine learning and noticed that most of the time, dimensionality reduction techniques like PCA and t-SNE are used in machine learning, but I rarely noticed anyone doing it for deep learning projects. Is there a specific reason for not using dimensionality reduction techniques in deep learning?

Answer 1

It highly depends on your task, your data and your network. Basically, PCA is a linear transformation of the current features. Suppose your data are images or a kind of data that locality is important. If you use PCA you are throwing away those locality information. Consequently, it is clear that people usually do not use them in convolutional networks. For sequential tasks, again it highly depends on your agent whether is online or not. If it is online, you don't have the entire signal from the beginning. Even if you have that for offline tasks, by doing such diminishing transformations you are again throwing away sequential information, I have to say I've not seen the use of them. I guess their main use is in tasks where your problem can be solved using simple MLPs which you don't keep sequential or local information. In those tasks due to the fact that you can employ PCA which leads to the reduction of highly correlated features, the number of parameters of your training model can be reduced significantly.

Answer 2

Deep learning does not use dimensionality reduction because deep learning itself is a useful dimensionality reduction technique. Deep learning learns a compressed, nonlinear representation of the data through the hidden layers. Since Deep Learning can learn nonlinear mappings, it is a more flexible dimensionality reduction technique than PCA which restricted to linear mappings.

Answer 3

If you have the computing time and power, backward selection allows you to be able to measure the effect of reducing dimensionality through the removal of variables.

Though this has been helpful in some situations, I would say generally it is not advisable on its own. Usually removing variables should be a result of domain knowledge with statistical backing, not just a statistical evaluation alone.