# How to deal with features that have widely different dimensionalities

Say I'm building a neural network to fit some classifier of some sort. To make things concrete, let's take the example of predicting housing prices using features of houses.

What should I do if one or two of my features consist of many more numbers than the other features, or even all other features combined? For example, say I have a few housing features: size in sqft, age, median income of location. These are 3 numbers. And then I have another feature, height of the roof for each square foot of the house (it's a bit contrived for this example of course) for which I would have actually "size in sqft"-numbers for this feature. So now my feature vector looks like this:

X = [1500sqft, 34 years, \$54,000, 10ft, 10.1ft, 10.3ft...1497 more numbers here...]

It seems that if I just naively put this into a neural net that the first 3 features would essentially be ignored since they only account for 3/1503 features. But they might actually be important. One try might be to simply average the "height of roof" feature over all of its elements to get an "average height of the roof" feature. That makes sense for this example, but what if sometimes I don't want to take this average? Are there any industry practices on what I might try if I ran into a problem like this?

One solution is using the same technique that is used in sequence learning. In that technique we use from Recurrent Neural Network and LSTM module.

we can train recurrent neural networks to learn very specific outputs for an arbitrary sequence of inputs, which is very powerful.

• My knowledge of recurrent networks is still a bit shaky at the moment, but I'll look into it, thanks! – enumaris Nov 2 '17 at 1:05

PCA - Principal Component Analysis:

I think the best option here would be to perform some dimensionality reduction using principal component analysis (PCA) prior to training your model and prior to predicting once you have a model. PCA will allow you to preserve a percentage of the total variance in the input data while significantly reducing the total number of dimensions.

In the toy example that you present, I would expect a high degree of linear correlation between the height per square foot features and a high degree of linear orthogonality between the other features. In such a case, PCA will naturally mash the correlated height terms together into 1 or 2 features while preserving 95-99% of the total original variance.

The only downside here, is that you will loose the physical meaning of your input features. I suppose you could decompose the eigenvectors and try to understand them, but loosing the physical meaning is usually not particularly hindering to training a model.

There are other methods, like lasso-ridge regression (you mention classification, but then provide a regression example e.g. the continuum variable of home price) or directly tracking the feature importance in a linear regression, but I prefer how PCA is able to project out linear dependence in input features, which seems particularly topical to your problem.

Give it a try and let us know how it goes. Hope this helps!

• Thank you! The housing example is an extremely contrived one that I just used to try to get the idea of the problem across. The problem I'm working on is indeed a classification rather than regression problem, I messed that up when I was thinking of an analogy. I'll look into PCA. – enumaris Nov 2 '17 at 1:04

Let's sum up what you are doing:

You intend to apply a Neural Network on a dataset of X samples with 1,503 features to get the most accurate prediction of price (i.e. continuous variable). You are worrying that your engineered features may only contribute marginally to the performance.