0
$\begingroup$

In my paper, I am saying that the accuracy of classification is $x\%$ when using the top N features.

My supervisor thinks that we should capture the classification accuracy when using N randomly selected features to show that the initial feature selection technique makes an actual difference.

Does this make sense?

I've argued that no one cares about randomly selected features so this addition doesn't make sense. It's quite obvious that randomly selecting features will provide a worse classification accuracy so there's no need to show that using any sort of feature ranking metric will be superior.

$\endgroup$
1
  • 4
    $\begingroup$ Seems reasonable to me. Maybe showing all features, randomly selected, and then your method would be a fair comparison. If feature selection is relevant to the paper, why not show a benchmark? $\endgroup$
    – Andy M
    Commented May 13, 2019 at 13:36

2 Answers 2

1
$\begingroup$

Your supervisor is right. Maybe not in the specific way to show your solution's dominance on the problem but at least in the main idea:

  • He is right because you need a benchmark to prove your feature selection is better than not-doing-anything.

  • Look for alternatives to represent why is your selection better:

    • A ranking for 5 possible selections: Yours and another four.
    • The alternative your supervisor told you: Random vs Yours.
    • The alternative your supervisor told you but improved: Select randomly features M times, select the best and show that your solution is the best.
    • Your solution vs all the features (the problem with this is that the accuracy improves with more variables many time, so you would have to use a measure which punishes using many features just as AIC)
$\endgroup$
1
  • $\begingroup$ I like the idea on M times and comparing with the best. Thanks! $\endgroup$
    – Sterls
    Commented May 13, 2019 at 19:52
1
$\begingroup$

Your supervisor is asking to use random feature selection as a baseline. Random performance is a common baseline in machine learning. For example in binary classification with equal numbers of samples from each group, a trained classification should perform better than 50% (random guessing).

$\endgroup$
2
  • $\begingroup$ Thanks for the answer. Yes a baseline does make sense. $\endgroup$
    – Sterls
    Commented May 13, 2019 at 19:51
  • $\begingroup$ I know this is 5 years old, but a "random feature baseline" is something I actually have not seen in ML papers. I had to search to find even 1-2. That being said, I completely agree with the idea and I wish this was more widespread! $\endgroup$ Commented May 1 at 14:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.