I am designing a scikit learn classifier for a sequence labelling task which has 5000+ categories and training data is at least 80 million and may grow upto an additional 100 million each year. I have already tried with all the categories but it generates classifiers in the order of 100s of GBs binary file. So I think that having one classifier for each category would be helpful and would also help me to fine tune features for each category thereby improving accuracy, but this means 5k+ classifiers for each of these categories. So how to handle this large data requirements and which incremental classifiers to use for this case , considering the fact that I will keep on getting additional training data as well as may discover new categories?
The number of features are about 100+ and due to the sequence labelling task contiguous sequence of training samples share same features values. The feature values are mostly text based and most are categorical with long text based values with large cardinality i.e many features may have huge number of possible values.
The available RAM IS 32gb with 8 core CPU. On a small scale I tried Multinomial NB and linear SGD with sparse matrices which are extremely sparse. Used the scikit learns Dictvectorizer to vectorize the feature dictionary. Also will pandas dataframes help to optimize the overall configuration?
Due to the scale of data involved I have only used about 20% of the data. And the features itself are 56GB of data from the 500MB input data. For the restricted training data, The precision too is very low does not reach above 12% with a very low recall too(1-2%). Have gone through vowpal wabbit and found the SEARN task but it seems that it is now no longer available in the latest version .