3
$\begingroup$

Through backward elimination I get a ranking of features over multiple datasets. For example in the dataset 1 I have the following ranking, the feature in the top being the most important:

  1. feat. 1
  2. feat. 2
  3. feat. 3
  4. feat. 4.

...

, whereas for dataset 2 I have for example the following ranking:

  1. feat. 3
  2. feat. 1
  3. feat. 2
  4. feat. 4.

I want to filter out those features which end up in the top of the ranking the most (incorporating that finishing in the top is better than finishing in the 3rd place). Which kind of ranking metric can I use for this problem?

$\endgroup$
2
$\begingroup$

An easy one to try would be average ranks, where you take the mean of the ranks for each feature. For your example,

$ \begin{array}{cc} \textbf{Feature} & \textbf{Avg. Rank} \\ 1 & 1.5 \\ 3 & 2 \\ 2 & 2.5 \\ 4 & 4 \\ \end{array} $

You could also weight the ranks by the size of the dataset, if the datasets that you are testing on are not the same size.

$\endgroup$
  • $\begingroup$ Can you recommend a publication in which they propose this method, or in which it is applied for a certain problem? $\endgroup$ – Archie Mar 28 '17 at 15:16
  • 1
    $\begingroup$ @Archie Here's a paper where they use average ranks to compare many classifiers: sciencedirect.com/science/article/pii/S0167865509002499 $\endgroup$ – timleathart Mar 29 '17 at 0:06
  • $\begingroup$ Great, I'll have a look at it! $\endgroup$ – Archie Mar 29 '17 at 10:14
2
$\begingroup$

A plan should be

  • consider a dataset as a 'match' between features
  • randomize matches' order (of insert them in time order, if any) and insert results in a rating system engine.

If you're interested in use more than in development, rankade, our free ranking system for sports, games, and more, allows matches with both 2 and 3+ factions, while Elo and Glicko works just for one-on-one (here's a comparison). In addition, rankade has a weight feature (all datasets have same impact?) that might refine your work.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.