I have an nlp model that has ground truth labels and predicted labels (that belong to different group of classes). For example, the ground truth labels are [art, computer science, history] and predicted labels are [drawing, engineering, philosophy, mathematics]. To find the accuracy of my predictions, I use a cosine similarity of ground truth with predictions. I now want to quantify the quality of my predictions -- what I plan to do is find the cosine similarity between ground truth and a non-predicted label and then classify it as false negative and use something similar for false positive. Is it a good approach?

  • $\begingroup$ $100\%$ misclassification rate? $\endgroup$
    – Dave
    Commented Apr 12 at 2:48
  • $\begingroup$ Sorry, I do not understand what you mean. The model is unsupervised and was not trained on the ground truth. I want to somehow measure how well it works now that I have ground-truth but with different "labels". $\endgroup$ Commented Apr 12 at 4:31
  • $\begingroup$ And it gives a wrong label every time: $100\%$ misclassification rate. $\endgroup$
    – Dave
    Commented Apr 12 at 11:46
  • $\begingroup$ Ok, I understand what you mean. So, let me rephrase as finding the effectiveness of the new model. Since this model predicts for example "mathematics" when the ground truth is "math", it is more effective than a different model predicting "biology" for the same ground truth. How would I go about doing that? $\endgroup$ Commented Apr 12 at 11:51


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.