1
$\begingroup$

I have a dataset with 10 million observations and 1 million sparse features. I would like to build a binary classifier for predicting a particular feature of interest.

My main problem is how to deal with the million features (both from a statistical and computational point of view). I understand one can use e.g. mini-batch optimization techniques or Spark to train a classifier on very many observations, but this does not solve the problem with very many features. I also understand that for moderate size datasets, one can use classical dimensionality reduction techniques, but I am not aware of a dimensionality reduction method that can process dataset of this size.

(And how would the answer to this question change if the features were dense, not sparse?)

$\endgroup$

2 Answers 2

1
$\begingroup$

The question is very similar to this one

I think that, indeed, using kernel on top of PCA will prove itself useful.

They also discuss implementation of different techniques that proved themselves reliable on such amount of features.

$\endgroup$
0
$\begingroup$

you can use filter feature elimination which is a technique to reduce dimensions before start modeling based on correlation, my suggestion is partition your features to multiple parts and do feature filtering and from each part take the uncorrelated features until you have smaller set. Also you can do this on subset of observations not all the 10 million.

$\endgroup$
1
  • $\begingroup$ Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Jan 25 at 14:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.