I'm working with a high-dimensional dataset, and have found that my attempts at dimensionality-reduction hurt the accuracy of downstream classifiers, suggesting that I am effectively losing information in the course of feature-compression.
Furthermore, when I sort the features by importance (per estimation via random forest) and then train neural networks on different numbers of the most important features, I find that the resulting classifier accuracy stops increasing before I have exhausted all of the useful-seeming features. I suspect that simultaneous training on many thousands of features is the problem, and I'd like to simplify the problem by learning from the features incrementally, that is:
I'd like to explore training a neural network adding a few features at a time: e.g. train using the top 100 features, then add in the next 100, train again (using coefs from previous round plus random coefs for the new part of the input layer), rinse, repeat.
NOTE that this "incrementalness" is across features, not samples; it doesn't affect whether training is incremental ("online") with respect to samples.
I gather this sort of thing has been tried before with some success but I haven't been able to find implementations in the Python ML libraries I've checked, nor can I find good explanations of the details of successful algorithms.
I have about 70K examples, divided over 6 balanced classes. For most of my work I care only about distinctions between a total of 4 classes, so I merge two pairs of the initial classes and end up with a total of 4 (I end up with unbalanced training sets as a result, but since the two larger classes are the ones whose classification is most important, I'm fine with that for now). I have a total of about 28K features, but am finding that I get best classifier performance when using no more than the top 4K features (pre-sorted by importance via Random Forest).
- Does my approach seem reasonable?
- Are there accepted published detailed algorithms (e.g., is it reasonable to add the features as extensions of the original input layer, or is another approach preferred)?
- Do any Python libs, particularly any compatible with scikit-learn, implement such an algorithm?