0
$\begingroup$

My goal is to recommend jobs to job seekers based on their skill set.

Currently I'm using an SVM for this, which is outputting one prediction, e.g. "software engineer at Microsoft". However, consider this: how significantly different are the skill sets of a software engineer at Microsoft and a software engineer at IBM? Probably not significantly different. Indeed, by inspection of my data set I can confirm this. Hence, the SVM struggles to discriminate in situations like this, of which there are many in my data set, and my classification accuracy is about 50%.

So I had an idea.

In SK Learn, once you've trained some model, you can compute the probability a particular input X belongs to each class.

So for each input X in my test set, I took the the top 3 most likely classifications. Then I tested whether or not the correct label was in the top 3 predictions. If it was, then I considered the prediction to be correct. In doing so, the classification accuracy increased to over 80%.

So my question is: is this a valid approach to measuring classification accuracy? If it is, then does it have a name?

In my mind, it is valid given my intended application, which is to recommend a selection of jobs to a job seeker, which are relevant to their skill set.

Cross posted from CS SE: https://cs.stackexchange.com/questions/117695/classification-accuracy-based-on-top-3-most-likely-classifications

I'm interested to know what perspective data scientists have on this.

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Please do not post the same question on multiple sites. Each community should have an honest shot at answering without anybody's time being wasted. $\endgroup$
    – D.W.
    Commented Nov 27, 2019 at 20:33
  • $\begingroup$ @D.W. I cross posted on recommendation of what is said on the linked page: Occasionally, people are interested in different perspectives on the same fundamental question. There are many Stack Exchange sites with overlapping topic spaces and it can be useful to get a "second opinion". Even then, however, it's best to tailor your question to each site. Ideally link to the question on the other site and explain what you hope to learn from asking another community. $\endgroup$
    – Data
    Commented Nov 27, 2019 at 20:59
  • 1
    $\begingroup$ That is a rare exceptional case, e.g., for after you have waited some time (a week?) after posting on one site and not gotten a useful response. It is not for asking simultaneously on multiple sites. And you did not "tailor your question to each site", cross-link both copies (the copy on CS.SE does not mention that it was cross-posted), or explain what you hope to learn by cross-linking in any depth. If everyone cross-posted every question that fits on multiple sites, the experience would be crummy for answerers, so there are reasons for this policy. $\endgroup$
    – D.W.
    Commented Nov 27, 2019 at 21:10
  • $\begingroup$ @D.W. I understand, thanks for the clarification $\endgroup$
    – Data
    Commented Nov 27, 2019 at 21:18

2 Answers 2

Reset to default
2
$\begingroup$

Yes, it's common to consider that the prediction is made of multiple answers (typically top N most relevant answers) and use a performance measure based on that.

Currently you're treating the problem as a classification problem but logically this is more like a recommendation problem or an information retrieval problem (like results from a search engine). Usually for this kind of problem the gold answer would also consists of a list of several items, but apparently your dataset contains a single answer for every instance.

Answer to comment: a couple of papers using some top N performance measures (note: it's just a quick selection based on the keyword "information retrieval")

The CLEF series of Shared Tasks have proposed many datasets and evaluation measures across the years, it's probably a good source for resources and papers... if you have a bit of time to explore it ;)

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for the reply. It doesn't surprise me that it's common, it seemed like an intuitive metric to use. However, to contradict myself, I am surprised that it's common because I had never seen it in a research paper before. I had a look around and could find one paper that mentions it. Do you know where I can see more examples? $\endgroup$
    – Data
    Commented Nov 27, 2019 at 23:39
  • 1
    $\begingroup$ @HumptyDumpty see edit, I added a few references in my answer $\endgroup$
    – Erwan
    Commented Nov 28, 2019 at 0:27
  • $\begingroup$ Excellent. The Microsoft link does not work, I think the 4 at the end should not be there, since the link works without the 4. I can't edit the post though, because the edit would change less than 6 characters. $\endgroup$
    – Data
    Commented Nov 28, 2019 at 0:41
  • $\begingroup$ Oh right, I must have typed it by mistake. It's fixed, thanks for letting me know $\endgroup$
    – Erwan
    Commented Nov 28, 2019 at 0:48
2
$\begingroup$

It is common and is called "top-k accuracy" or "top-n accuracy". You can find a description in these posts:

What is the definition of Top-n accuracy?

Evaluation & Calculate Top-N Accuracy: Top 1 and Top 5

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.