Evaluating the performance of a machine learned recommendation system

I have a set of resumes $$R=\{{r_1,...,r_n\}}$$, which I've transformed to a vector space using TF-IDF. Each resume has a label, which is the name of their current employer. Each of these labels comes from the set of possible employers $$E = \{{e_1,...,e_m\}}$$.

From this, I have trained a machine learning model. This model then takes some $$r_i$$ from the test set, and assigns a probability to each member of $$E$$. The results are then ranked, from highest probability to lowest probability.

E.g. $$P(e_2|r_i)=0.56, P(e_{52}|r_i)=0.29, P(e_{29}|r_i)=0.14,...etc.$$

The resume, $$r_i$$ belongs to some individual, so this ranking is used to inform the individual as to what companies the model believes are most likely to hire them, given the details of what their resume contains (their skills, past employers, education, personal summary). In this case, company $$e_2$$ is most likely, followed by $$e_{52}$$ and so on.

My question is, how do you evaluate the performance of this recommendation system? Where the information need of the user is to learn what companies their resume matches to the best.

My own ideas

My understanding from information retrieval is that we need to determine some measure of relevance. From this, it's possible to use some measure like mean average precision to measure performance. Determining relevance seems like the tricky part. For instance $$e_2$$ has a high probability, but is it actually relevant? Maybe $$r_i$$ is based on aeronautical engineering, but $$e_2$$ is a food store, which is clearly not relevant. My current idea is to take each $$r_i$$ in the training set belonging to the same label $$e_j$$, and then compute a single TF-IDF vector which is the average of the TF-IDF vectors belonging to each $$r_i$$ labelled as $$e_j$$.

E.g. (an unrealistic example) Suppose $$r_2$$ and $$r_9$$ are labelled as $$e_4$$. Now suppose $$r_2$$ has TF-IDF vector $$[0.2, 0.1, 0.5, 0.2]$$ and $$r_9$$ has TF-IDF vector $$[0.22, 0.12, 0.44, 0.22]$$. Then the average of these is $$[0.21, 0.11, 0.47, 0.21]$$. Repeating this process for all $$e_j\in E$$ results in $$m$$ of these vectors. From this, it's possible to compute the cosine similarity between some $$e_i$$ and $$e_j$$.

Returning to the first example, we can take the true label of $$r_i$$, and then find the cosine similarity between this label and each member of $$E$$. Then we set some threshold and evaluate whether $$\text{cosineSim}(\text{true label}, e_j) < \text{some threshold}$$. If the cosine similarity is above the threshold, then $$e_j$$ is relevant, otherwise, $$e_j$$ is not relevant.

I'm not sure if this is a sensible/valid approach (I wonder if it defeats the point of the machine learning, since I may as well just use the cosine similarity? That said, I cannot forgo the machine learning component in this project).

Maybe this is an over complication, and something like top k accuracy would be fine. I.e. is the true label in the top k suggestions?

• I don't understand what is the "label by their current employer"? At first sight it looks like this is the information to use for evaluation, by setting aside some of these labeled instances as test set. But maybe I misunderstood. – Erwan Dec 7 '19 at 17:20
• @Erwan I have changed the wording in the question, I hope this clarifies? – Data Dec 7 '19 at 17:36
• Yep, thanks! posted an answer. – Erwan Dec 7 '19 at 17:54

• I think your ideas are good. I just want to clarify that my understanding of top-k accuracy is correct, if you don't mind? This is what I think: Having generated the ranked probabilities of each member of $E$ for some resume $r$, we take the top-k. Then we check if the true label of $r$ is within the top-k. For instance, in the example in the question, if we take the top-2, we get $e_2$ and $e_{52}$. If the true label of $r$ is $e_2$ or $e_{52}$, then it's a success, otherwise it's a failure. We repeat this for all $r \in R$. Then $top-k accuracy = # successes / (# successes + # failures)$. – Data Dec 7 '19 at 18:26