Are there established good algorithms for incremental feature learning for a neural network? Do any python ML libraries implement such algorithms?

Question

I'm working with a high-dimensional dataset, and have found that my attempts at dimensionality-reduction hurt the accuracy of downstream classifiers, suggesting that I am effectively losing information in the course of feature-compression.

Furthermore, when I sort the features by importance (per estimation via random forest) and then train neural networks on different numbers of the most important features, I find that the resulting classifier accuracy stops increasing before I have exhausted all of the useful-seeming features. I suspect that simultaneous training on many thousands of features is the problem, and I'd like to simplify the problem by learning from the features incrementally, that is:

I'd like to explore training a neural network adding a few features at a time: e.g. train using the top 100 features, then add in the next 100, train again (using coefs from previous round plus random coefs for the new part of the input layer), rinse, repeat.

NOTE that this "incrementalness" is across features, not samples; it doesn't affect whether training is incremental ("online") with respect to samples.

I gather this sort of thing has been tried before with some success but I haven't been able to find implementations in the Python ML libraries I've checked, nor can I find good explanations of the details of successful algorithms.

I have about 70K examples, divided over 6 balanced classes. For most of my work I care only about distinctions between a total of 4 classes, so I merge two pairs of the initial classes and end up with a total of 4 (I end up with unbalanced training sets as a result, but since the two larger classes are the ones whose classification is most important, I'm fine with that for now). I have a total of about 28K features, but am finding that I get best classifier performance when using no more than the top 4K features (pre-sorted by importance via Random Forest).

1. Does my approach seem reasonable?
2. Are there accepted published detailed algorithms (e.g., is it reasonable to add the features as extensions of the original input layer, or is another approach preferred)?
3. Do any Python libs, particularly any compatible with scikit-learn, implement such an algorithm?

Answer 1

The opposite of what you describe is used more often. You start with many features and then prune the neural network at the end.

I don't believe any python neural network package does what you want.

But it's very simple to implement.

For example, using sklearn, you can just pass warm_start=True, so that it doesn't reinitialize weights every time you call fit(), and then you can increase the number of nodes in your feature layer, or any layer you wish.

from sklearn import datasets
from sklearn.neural_network import MLPClassifier
import numpy as np

X, y = datasets.load_iris(True)
X = (X-np.mean(X, 0))/np.std(X, 0)  # normalize

nhidden = 10
m = MLPClassifier(nhidden, max_iter=1000, warm_start=True).fit(X[:, :1], y)

# every 1,000 iterations, throw in a new feature
for nfeatures in range(2, X.shape[1]+1):
    m.coefs_[0] = np.r_[m.coefs_[0], np.random.randn(1, nhidden)]
    m.fit(X[:, :nfeatures], y)

# in the end, we have a neural network trained with all features
print(m.score(X, y))