I have a dataset with 10 million observations and 1 million sparse features. I would like to build a binary classifier for predicting a particular feature of interest.
My main problem is how to deal with the million features (both from a statistical and computational point of view). I understand one can use e.g. mini-batch optimization techniques or Spark to train a classifier on very many observations, but this does not solve the problem with very many features. I also understand that for moderate size datasets, one can use classical dimensionality reduction techniques, but I am not aware of a dimensionality reduction method that can process dataset of this size.
(And how would the answer to this question change if the features were dense, not sparse?)