# model with features of different sizes

I want to train a model (either classification or regression, doesn't matter) with features/inputs of different sizes, but I am not sure how to do it.

For example, for each data-point, feature 1 and 2 are just real numbers and feature 3 is a vector of length 100, so if I have $$n$$ data-points, then I have a $$n \times 1$$ vector for feature 1, another $$n \times 1$$ vector for feature 2, and a $$n \times 100$$ matrix for feature 3.

The first question is what kind of model I should use or try? Gradient boosting, random forest, or neural network?

The second question is how to feed the features of different sizes to the model? Just concatenate them to form a $$102 \times 1$$ vector for each data-point?

Thanks.

• It's impossible to tell a priori which approach should you take. Evaluate models build with different strategies and you'll see. Nov 27 '19 at 13:01

Several options you can try, all of them have different preconditions you need to consider.

1. Direct Ensemble (with high number of n, should be the first choice)

If n >> 103 you can directly use standard ML procedure different models and parameters with CV and find the best fit. Since most of the time its not the case, you may want to use ensemble tree models since they outperform other learners when number of observations are relatively small compare to number of features.

2. Clustering

If those 100 features are sourced by a common factor, you can cluster those features with optimal K (see silhouette score) and add those cluster labels as 3rd feature on your model. Finally your model becomes n observations, 3 features.

3. Fit Twice

In case of classification, you can first fit 100 features and use their probability score (predict_proba in sklearn) as 3rd feature on your model.

4. PCA + Fit

You can use PCA to 100 features and obtain p features. After that with p+2 features you can construct your model. Again common factor condition applies in here. (In direct approach you can apply PCA to 102 features as well.)

Final Comment:

All of above methods are should be tried and the best one should be picked. However, if you have n >> 102 observations you should directly fit the model with strong learners (xgboost, lgbm, nn's etc.) without clustering/twice fitting since those methods generates error by their nature in addition to final learner error.

• Additionally, you can try to train models, which will inform you of feature's improtance, like LGBM. Check importance plot at the end of this post. Nov 27 '19 at 13:04

When data is presented that way, it is often because all 100 values in the vector should be read as one. For example, those can be a set of 100 measurements of, say, a temperature at different locations of a system. Data in the vector is likely to be correlated and the order of sub-features may be a valuable information (for instance, in my previous examples, if they represent the temperature measurements over a range of positions along a pipe).

If data is not correlated at all, or sub-features have nothing relevant in common, you can just expand each sub-feature as a normal feature. In the process, you may get a final dataset with numerous features, this will eventually need to reduce the dimension.

If data is deemed "simply" correlated, a good option would be to reduce the dimension of the vector from 100 to a much lower value, then expand it as before. The most straightforward way to do so is PCA. Be sure to perform it only over each particular multi-sub-features feature, at first.

If data is correlated and the order of features matters (i.e. $$(1, 1, 0, 0, 0, 0)$$ is closer to $$(0, 1, 1, 0, 0, 0)$$ than to $$(0, 0, 1, 0, 0, 1)$$, for instance, you will need a more complex approach. You could perform density-based clustering based on a custom distance function such as the Wasserstein metric / earth mover's distance. The 100-dimension feature would thus be reduced to a single feature representing a class among this clustered dataset.

• There are similarities indeed. Actually I didn't see your answer while preparing mine! Nov 27 '19 at 12:09
• It happens, Highlighting correlation is a good approach. Nov 27 '19 at 12:13