3
$\begingroup$

Is it possible to have a PCA setup (or any other dimensionality reduction technique) in a way that adding new features wouldn't require retrain downstream models that were trained on that particular PCA?

The idea is that (hopefully) we could have new features that would improve PCA in a way that wouldn't require retraining every time we add new features. If this isn't possible as explained here, how to achieve this in another way?

We add new features regularly by adding new datasources, or add some new features through feature engineering. Downflow models rely on PCA of that feature space. By training models on PCA and not on features directly simplifies workflows, and also removes burden of retraining models if it was possible to have a PCA (or another feature extraction process) that maps original features to this "stable" subspace.

$\endgroup$
1
$\begingroup$

If you're prepared to use a neural net, it might be worthwhile looking into spatial pyramid pooling.

A spatial pyramid pooling layer can take inputs of varying sizes (eg from a convolutional layer) and convert them to an output of a fixed size (to pass on to a fully connected layer for instance). This is probably most useful where features are spatially related in some way.

Alternatively, you could continue simply implementing pca on your inputs, selecting the N dimensions with the highest variance for each example, but train the model continuously through reinforcement learning.

As there is more to how data is distributed along an axis than the variance, it is unlikely that any one pretrained model will perform well when your preprocessing step is constantly swapping out features, which is why a reinforcement model would be needed. Combined with the pca, hopefully that won't take up too much overhead.

| improve this answer | |
$\endgroup$
  • $\begingroup$ That seems to be specific to CNN, and image recognition in general? arxiv.org/pdf/1406.4729.pdf Also main purpose of this approach I described in the Q was to avoid retraining.. Thanks. $\endgroup$ – Tagar Sep 15 '17 at 20:26
  • 1
    $\begingroup$ Good point, I've had a think this morning and updated my answer, would be interested to see your response. $\endgroup$ – Finn Reilly Sep 17 '17 at 11:25
  • $\begingroup$ looks interesting, I will have a look at reinforcement learning. thanks for bringing up NN topic - will explore this too. although CNN and spatial pyramid pooling doesn't apply to us as our models aren't in image recognition domain. $\endgroup$ – Tagar Sep 18 '17 at 15:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.