Say I'm building a neural network to fit some classifier of some sort. To make things concrete, let's take the example of predicting housing prices using features of houses.

What should I do if one or two of my features consist of many more numbers than the other features, or even all other features combined? For example, say I have a few housing features: size in sqft, age, median income of location. These are 3 numbers. And then I have another feature, height of the roof for each square foot of the house (it's a bit contrived for this example of course) for which I would have actually "size in sqft"-numbers for this feature. So now my feature vector looks like this:

X = [1500sqft, 34 years, $54,000, 10ft, 10.1ft, 10.3ft...1497 more numbers here...]

It seems that if I just naively put this into a neural net that the first 3 features would essentially be ignored since they only account for 3/1503 features. But they might actually be important. One try might be to simply average the "height of roof" feature over all of its elements to get an "average height of the roof" feature. That makes sense for this example, but what if sometimes I don't want to take this average? Are there any industry practices on what I might try if I ran into a problem like this?


One solution is using the same technique that is used in sequence learning. In that technique we use from Recurrent Neural Network and LSTM module.

we can train recurrent neural networks to learn very specific outputs for an arbitrary sequence of inputs, which is very powerful.

To know more about this, you can follow this link.

  • $\begingroup$ My knowledge of recurrent networks is still a bit shaky at the moment, but I'll look into it, thanks! $\endgroup$ – enumaris Nov 2 '17 at 1:05

PCA - Principal Component Analysis:

I think the best option here would be to perform some dimensionality reduction using principal component analysis (PCA) prior to training your model and prior to predicting once you have a model. PCA will allow you to preserve a percentage of the total variance in the input data while significantly reducing the total number of dimensions.

In the toy example that you present, I would expect a high degree of linear correlation between the height per square foot features and a high degree of linear orthogonality between the other features. In such a case, PCA will naturally mash the correlated height terms together into 1 or 2 features while preserving 95-99% of the total original variance.

The only downside here, is that you will loose the physical meaning of your input features. I suppose you could decompose the eigenvectors and try to understand them, but loosing the physical meaning is usually not particularly hindering to training a model.

There are other methods, like lasso-ridge regression (you mention classification, but then provide a regression example e.g. the continuum variable of home price) or directly tracking the feature importance in a linear regression, but I prefer how PCA is able to project out linear dependence in input features, which seems particularly topical to your problem.

Give it a try and let us know how it goes. Hope this helps!

  • $\begingroup$ Thank you! The housing example is an extremely contrived one that I just used to try to get the idea of the problem across. The problem I'm working on is indeed a classification rather than regression problem, I messed that up when I was thinking of an analogy. I'll look into PCA. $\endgroup$ – enumaris Nov 2 '17 at 1:04

Let's sum up what you are doing:

You intend to apply a Neural Network on a dataset of X samples with 1,503 features to get the most accurate prediction of price (i.e. continuous variable). You are worrying that your engineered features may only contribute marginally to the performance.

Your concern may be unnecessary

If your overall goal is to get the most accurate prediction, I believe dimensionality reduction such as PCA (as mentioned by OP) is not the way to go here. With every feature you create you may explain a fraction of the variance, even if they are highly correlated. Worst case is, that it is useless. If your network is complex enough, i.e. your architecture can ensure that relationships between the variables can be accounted for completely, you will catch everything that's there and your chosen algorithm and setup is capable of. Any variable that's not important will not play a role in your prediction as weights will simply be (close to) zero. However, if you follow this way, ensure that you have a robust method to measure whether you are overfitting or not, such as K-fold cross validation. You may also want to look into regularization techniques suchas L1, L2 and DropOut (neural networks).

  • $\begingroup$ I used to make very similar claims, but have now changed course on this after seeing countless models with some PCA outperform those without. I think the predominant difference is due to variance in testing data that's not contained in training data or vice versa. In a perfect world, with an infinite amount of data these two would be equivalent, but practically speaking PCA tends to project out noise in a way that aligns training data with real world data and helps ML models and ANNs converge to a more accurate solution. Try it, I think you'll see more benefit than detriment. $\endgroup$ – AN6U5 Dec 6 '17 at 20:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.