Dimensionality reduction for millions of features

Question

I have a dataset with 10 million observations and 1 million sparse features. I would like to build a binary classifier for predicting a particular feature of interest.

My main problem is how to deal with the million features (both from a statistical and computational point of view). I understand one can use e.g. mini-batch optimization techniques or Spark to train a classifier on very many observations, but this does not solve the problem with very many features. I also understand that for moderate size datasets, one can use classical dimensionality reduction techniques, but I am not aware of a dimensionality reduction method that can process dataset of this size.

(And how would the answer to this question change if the features were dense, not sparse?)

Answer 1

The question is very similar to this one

I think that, indeed, using kernel on top of PCA will prove itself useful.

They also discuss implementation of different techniques that proved themselves reliable on such amount of features.

Answer 2

you can use filter feature elimination which is a technique to reduce dimensions before start modeling based on correlation, my suggestion is partition your features to multiple parts and do feature filtering and from each part take the uncorrelated features until you have smaller set. Also you can do this on subset of observations not all the 10 million.