Rank feature selection over multiple datasets

Question

Through backward elimination I get a ranking of features over multiple datasets. For example in the dataset 1 I have the following ranking, the feature in the top being the most important:

1. feat. 1
2. feat. 2
3. feat. 3
4. feat. 4.

...

, whereas for dataset 2 I have for example the following ranking:

1. feat. 3
2. feat. 1
3. feat. 2
4. feat. 4.

I want to filter out those features which end up in the top of the ranking the most (incorporating that finishing in the top is better than finishing in the 3rd place). Which kind of ranking metric can I use for this problem?

Answer 1

An easy one to try would be average ranks, where you take the mean of the ranks for each feature. For your example,

$ \begin{array}{cc} \textbf{Feature} & \textbf{Avg. Rank} \\ 1 & 1.5 \\ 3 & 2 \\ 2 & 2.5 \\ 4 & 4 \\ \end{array} $

You could also weight the ranks by the size of the dataset, if the datasets that you are testing on are not the same size.

Answer 2

A plan should be

• consider a dataset as a 'match' between features
• randomize matches' order (of insert them in time order, if any) and insert results in a rating system engine.

If you're interested in use more than in development, rankade, our free ranking system for sports, games, and more, allows matches with both 2 and 3+ factions, while Elo and Glicko works just for one-on-one (here's a comparison). In addition, rankade has a weight feature (all datasets have same impact?) that might refine your work.