Expand or compact features?

Question

I have a classification task for people with 3 categories. I want to apply machine learning for that. I have 10 sources of data, which have the same fields (say 4: age, job title, a number of organizations, a number of followers). Data is incomplete, some fields can be missing in some profiles. The training set is limited (say, 300 examples).

I have two strategies for feature engineering, and I don't know which one to use.

1. Expand features: take 40 features (Profile 1 age, Profile 1 job title, ..., Profile 10 age, Profile 10 job title).

2. Compact features: take 4 features, and apply some heuristics to merge the values from different profiles. Say, take age and job title which occur most frequently, take a maximum number of organizations, take a sum of numbers of followers.

What strategy is generally used to give best results and why?

Answer 1

The way I see it is that your 10 sources of data, they all refer to the same set of people. Depending on the attributes, some can be expanded, some can be merged ...

Attributes such as age should be unique, so it doesn't make sense to expand it to Profile 1 age, profile 2 age ... One simple way is merge them is by using the average or use max. Expanding age only add redundant data to your feature matrix, and increase its dimensionality, in most cases, this doesn't help generalization performance of your model.

On the other hand, number of followers can be expanded. Depending on the data source, a guy has 10 followers on Twitter but 1000 followers on Google+ might simply mean that he barely uses Twitter.

That being said, the way you pick your features or engineer new features should increase your model performance, so if expanding number of followers actually decrease Cross Validation or Test performance, compared to the one using sum of followers then you can simply use sum of followers.