1
$\begingroup$

Let's suppose that I have a classifier which detects whether a script of code is written in Python or C++ or Java.

Therefore, my classifier has 3 outputs which give the probabilities that a certain code script is one of these 3 programming languages above.

However, unfortunately, there is the chance that the test data my contain some code scripts written in other programming languages such as Javascript, Ruby, PHP etc.

If I leave my classifier as it is then it going to classify these code scripts as being in Python or C++ or Java and this will be certainly pretty wrong and misleading.

Does it make sense to add another class called 'Other' which would contain code scripts of various other programming languages (Javascript, Ruby, PHP etc) or even irrelevant text to cope with that?

Is there any better idea?

Improve this question
$\endgroup$
2
  • $\begingroup$ I think your approach makes sense. Of course it would be best to have all possible outcomes (classes) in the training set, but this might not be possible. With the „other“ class you may at least avoid that code snippets are classified entirely wrong. Ultimately, it is about testing the approach, but I can imagine that it works well. $\endgroup$
    – Peter
    May 31, 2019 at 15:47
  • $\begingroup$ Thanks @Peter , I will listen to the opinions of others too and think about it. $\endgroup$
    – Outcast
    May 31, 2019 at 15:55

1 Answer 1

Reset to default
1
$\begingroup$

You could definitely add more samples and tag it as other with your classifier. However, realistically this might not be an option. For this exact purpose, you could use confidence values with the output of your classifier to predict the "other" class. You could bring in confidence thresholding to support this, i.e if your classifier is not confident > 0.8 (as an example) with its prediction, then you tag it "other" class.

What is the best value to threshold with, you can identify by running a number of experiments with different threshold values and plot a graph for correctness i.e classification error rate graph.

Improve this answer
$\endgroup$
9
  • $\begingroup$ Thanks for the answer. Your idea sounds pretty reasonable (and I have thought about it too) but 1) How? (Do you have any source code for this?), 2) How easy is it to do this? If it is quite time intensive then I do not know if it is worth it for a quick prototype. $\endgroup$
    – Outcast
    May 31, 2019 at 16:12
  • $\begingroup$ @PoeteMaudit I am going to assume you are using sklearn here. If yes, then you can refer to the following thread : stackoverflow.com/questions/31129592/… $\endgroup$
    – Nischal Hp
    May 31, 2019 at 16:19
  • $\begingroup$ Thanks, I have actually seen this post. However, predict_proba is not really a confidence score which will help in the case of unseen classes because in predict_proba the output probabilities always sum up to 1. It would be interesting if we could have some output probabilities that do not necessarily sum up to 1 but they may be all very low and let's say sum up to 0.3 if an irrelevant text is given as an input. $\endgroup$
    – Outcast
    May 31, 2019 at 16:31
  • $\begingroup$ @PoeteMaudit Usually predictions are probabilities, the one with the highest is chosen as the main class. As the case with probabilities, they always add up to 1. For example, in your scenario, if your model does not with 0.7 or 0.8 probability suggest the piece of code belong to a class, you can discard it and label it as other. $\endgroup$
    – Nischal Hp
    May 31, 2019 at 16:34
  • 1
    $\begingroup$ Regarding your second comment, thanks and it looks interesting (upvote overall for your effort) but see point (2) of my first comment at your post. $\endgroup$
    – Outcast
    May 31, 2019 at 16:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.