3
$\begingroup$

I have a Dataset of around 200 features. Most of them are categorical and only a few are numerical. It seems that an artificial neural network with an Autoencoder has some problems with that kind and amount of features. Therefore, I thought to use PCA to reduce the dimensions and then apply the Autoencoder afterwards.

Does the combination of a PCA before neural nets makes sense as the neural nets also reduces the information in internal layers? Does anybody has experiences with such a combination?


Edit: There is also an interesting answer in the stats forum.

$\endgroup$
  • $\begingroup$ How many different categories do your categorical variables have? $\endgroup$ – timleathart Jan 16 '18 at 10:40
  • $\begingroup$ Most have two different categories and some have around 10 different categories. Around 10 features are numerical. $\endgroup$ – Rene B. Jan 16 '18 at 15:49
  • $\begingroup$ There is also an interesting answer in the stats forum supporting the answer from @Media: stats.stackexchange.com/a/55059 $\endgroup$ – Rene B. Apr 23 '18 at 7:12
  • $\begingroup$ @ReneB. maybe it's worth to mention it in your question. $\endgroup$ – Media Apr 26 '18 at 12:57
3
$\begingroup$

Neural networks are actually extremely effective at performing dimensionality reduction. A great example is word2vec, which applies a shallow neural network to reduce inputs on the order of several million features (i.e. unnormalized text) to a 30-150 dimensional vector via a process that is mathematically analogous to matrix factorization (which is the class of techniques PCA belongs to). Autoencoders function very similarly to word2vec: if you're planning on using an autoencoder to learn an embedding of your data, I wouldn't expect you'd gain anything from applying PCA first rather than using an autoencoder (or something similar) to learn a better embedding for your data that isn't constrained by the assumptions of PCA.

$\endgroup$
3
$\begingroup$

PCA is used to abandon having redundant features. It expands directions which your data is highly distributed in those directions. During this process, it does not care about the labels of your data. In those directions your data will be highly distributed and none of features have correlation. PCA just does the preceding things. In that feature space your data may be easily separable or not. But using this before neural nets is a way to reduce the redundant features which may cause your net have too many parameters. It is a kind of pre-processing for just reducing the correlated features, although there are different reasons that one can apply PCA, like data visualization or understanding data, or even reporting data based on e.g. three main components. I recommend you using that because you may find that in the new space you would classify your data with a more smaller net.

Does the combination of a PCA before neural nets makes sense

Yes, you can do that as a pre-processing stage.

... as the neural nets also reduces the information in internal layers?

neural nets do not necessarily reduce the information in internal layers. In convolutional nets, max-pooling layer somehow reduces the unnecessary information but other usual nets such as convolutional layers or dense layers, try to change the space of the inputs of the layers or equivalently with another interpretation they try to find other features and patterns that the data can be separated in those spaces. PCA actually reduces the correlated features.

Does anybody has experiences with such a combination?

Yes. It all depends on your data. In some applications it worked for me, which I had so many correlated features. But it has happened that it does not work very well in both cases of existing or not existing of correlated features.

$\endgroup$
  • $\begingroup$ To which applications did it work? As you mention first that it worked for correlated features but then you tell that it does not work very well for correlated features if I understand correctly. Does it mean that you can apply PCA before Autoencoder but it does not work very well? Then I dont need to apply it beforehand, no? $\endgroup$ – Rene B. Jan 17 '18 at 14:09
  • $\begingroup$ @ReneB. Not really. First, I said I've had problems that had so many correlated features. After applying PCA I got so much better result with a smaller network. Second, I said although this approach is fine, PCA does not consider the label of data, It may happen that even after applying PCA you don't get appropriate result. I had problems that both correlated features and uncorrelated features existed, and after applying PCA I didn't get appropriate result. As a result, I said it is so much dependent on your data. So definitely try to apply PCA to investigate it works or not. $\endgroup$ – Media Jan 17 '18 at 14:21
  • $\begingroup$ @ReneB. As an advise I recommend you apply PCA if you have so many features. You will definitely have correlated features. Most of the time, it have worked for me. $\endgroup$ – Media Jan 17 '18 at 14:22
  • $\begingroup$ Thanks @Media I marked your information as usefull. However, David Marx answer also sound reasonable and useful. $\endgroup$ – Rene B. Jan 17 '18 at 15:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.