5
$\begingroup$

I have a set of resumes $R=\{{r_1,...,r_n\}}$, which I've transformed to a vector space using TF-IDF. Each resume has a label, which is the name of their current employer. Each of these labels comes from the set of possible employers $E = \{{e_1,...,e_m\}}$.

From this, I have trained a machine learning model. This model then takes some $r_i$ from the test set, and assigns a probability to each member of $E$. The results are then ranked, from highest probability to lowest probability.

E.g. $P(e_2|r_i)=0.56, P(e_{52}|r_i)=0.29, P(e_{29}|r_i)=0.14,...etc.$

The resume, $r_i$ belongs to some individual, so this ranking is used to inform the individual as to what companies the model believes are most likely to hire them, given the details of what their resume contains (their skills, past employers, education, personal summary). In this case, company $e_2$ is most likely, followed by $e_{52}$ and so on.

My question is, how do you evaluate the performance of this recommendation system? Where the information need of the user is to learn what companies their resume matches to the best.


My own ideas

My understanding from information retrieval is that we need to determine some measure of relevance. From this, it's possible to use some measure like mean average precision to measure performance. Determining relevance seems like the tricky part. For instance $e_2$ has a high probability, but is it actually relevant? Maybe $r_i$ is based on aeronautical engineering, but $e_2$ is a food store, which is clearly not relevant. My current idea is to take each $r_i$ in the training set belonging to the same label $e_j$, and then compute a single TF-IDF vector which is the average of the TF-IDF vectors belonging to each $r_i$ labelled as $e_j$.

E.g. (an unrealistic example) Suppose $r_2$ and $r_9$ are labelled as $e_4$. Now suppose $r_2$ has TF-IDF vector $[0.2, 0.1, 0.5, 0.2]$ and $r_9$ has TF-IDF vector $[0.22, 0.12, 0.44, 0.22]$. Then the average of these is $[0.21, 0.11, 0.47, 0.21]$. Repeating this process for all $e_j\in E$ results in $m$ of these vectors. From this, it's possible to compute the cosine similarity between some $e_i$ and $e_j$.

Returning to the first example, we can take the true label of $r_i$, and then find the cosine similarity between this label and each member of $E$. Then we set some threshold and evaluate whether $\text{cosineSim}(\text{true label}, e_j) < \text{some threshold}$. If the cosine similarity is above the threshold, then $e_j$ is relevant, otherwise, $e_j$ is not relevant.

I'm not sure if this is a sensible/valid approach (I wonder if it defeats the point of the machine learning, since I may as well just use the cosine similarity? That said, I cannot forgo the machine learning component in this project).

Maybe this is an over complication, and something like top k accuracy would be fine. I.e. is the true label in the top k suggestions?

I'm not sure, I'm interested to have some more informed perspective.

$\endgroup$
  • $\begingroup$ I don't understand what is the "label by their current employer"? At first sight it looks like this is the information to use for evaluation, by setting aside some of these labeled instances as test set. But maybe I misunderstood. $\endgroup$ – Erwan Dec 7 '19 at 17:20
  • $\begingroup$ @Erwan I have changed the wording in the question, I hope this clarifies? $\endgroup$ – DataBSc Dec 7 '19 at 17:36
  • $\begingroup$ Yep, thanks! posted an answer. $\endgroup$ – Erwan Dec 7 '19 at 17:54
1
$\begingroup$

To the extent possible you should try to evaluate based on your data rather than some ad-hoc measure. As you rightly noticed, there is a real risk that the ad-hoc measure would just confirm the predictions of the model, since it uses a somewhat similar method.

I would suggest that you split your data between a training set and test set (or even better use cross-validation), and indeed use top-K accuracy (or something similar) to evaluate on the test set. That would be the safe option for a proper evaluation, and then you could try to see if your ad-hoc measure correlates with it: if it does, then you have evidence that in the future it can be be used instead of a test set.

Side note: your instances don't contain any negative evidence such as resumes rejected by an employer. In case you could obtain this kind of data, it could probably improve the predictions.

$\endgroup$
  • $\begingroup$ I think your ideas are good. I just want to clarify that my understanding of top-k accuracy is correct, if you don't mind? This is what I think: Having generated the ranked probabilities of each member of $E$ for some resume $r$, we take the top-k. Then we check if the true label of $r$ is within the top-k. For instance, in the example in the question, if we take the top-2, we get $e_2$ and $e_{52}$. If the true label of $r$ is $e_2$ or $e_{52}$, then it's a success, otherwise it's a failure. We repeat this for all $r \in R$. Then $ top-k accuracy = # successes / (# successes + # failures)$. $\endgroup$ – DataBSc Dec 7 '19 at 18:26
  • $\begingroup$ Yes this is correct. good luck! $\endgroup$ – Erwan Dec 7 '19 at 18:50
  • $\begingroup$ Thanks! What do you think of using area under ROC? I read in a paper that Area under the curve (AUC) of ROC is used to measure the quality of ranking for a probabilistic classifier. I tried it out and the model scored 0.85. Whereas in top-3 accuracy, the model scores 0.5. I would expect a higher top-3 accuracy if AUC is 0.85? But my knowledge of AUC is not good at the moment, so I don't know. $\endgroup$ – DataBSc Dec 8 '19 at 0:13
  • 1
    $\begingroup$ @HumptyDumpty AUC is certainly an interesting evaluation measure for a ranking, but the AUC score is not comparable to the top N accuracy: AUC measures the probability that for any two items a,b such that a<b according to the "true" ranking, these items will be ranked the same way in the predicted ranking. In other words, AUC says how "correctly ordered" the predicted ranking is. So AUC considers the ranking as a whole, which means that it gives the same importance to the bottom of the ranking as to the top. It's useful in the sense that it avoids the needs for a binary threshold when .... $\endgroup$ – Erwan Dec 8 '19 at 1:13
  • 1
    $\begingroup$ ... taking the top N: if the right answer was in position N+1, the top N accuracy counts it a wrong answer even though it was not far. On the other hand, it's clearly more useful to know that the correct answer is around the top than to know that the bottom of the ranking is correctly ordered. So both methods have their merits, and it's normal that their scores would be very different since they don't measure the same thing. $\endgroup$ – Erwan Dec 8 '19 at 1:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.