I've improved my text classification to topic module, from simple word2vec to piped tfidf and OneVsRestClassifier (using sklearn). It does improve the classification but with word2vec I was able to calculate the match percentage for each topic and with OneVsRestClassifier i get a match or no match to a specific topic. Is there a way to see with OneVsRestClassifier what was the percentage of the classification?

P.S. I am not talking about evaluating the performance of the training but the actual real time matching percentage.


Yes, of course.

Assuming that you have used sklearn's OneVsRestClassifier and so you have a decision function for example a Support Vector Classifier with say linear kernel. Use set_params to change probability key to True, default is False. Use this in the OneVsRestClassifier classifier and then go with the inbuilt function predict_proba like

from sklearn.multiclass import OneVsRestClassifier
from sklearn.svm import SVC
mod = OneVsRestClassifier(SVC(kernel='linear').set_params(probability=True)).fit(samples,classes)
print mod.predict_proba(np.array([your_sample_vector]).reshape(1,-1))


You can use your old LinearSVC with decision_function to find the distance from the hyperplane and convert them to probabilities like

mod = OneVsRestClassifier(LinearSVC()).fit(sample,clas)
proba = (1./(1.+np.exp(-mod.decision_function(np.array(your_test_array).reshape(1,-1)))))
proba /= proba.sum(axis=1).reshape((proba.shape[0], -1))\
print proba

Now you don't need tuning the parameters, I guess. :)

  • $\begingroup$ Thank you, it is strange that LinearSVC I used did not have this flexibility of setting probability as True. $\endgroup$ – MikeL Apr 2 '17 at 7:32
  • $\begingroup$ Also it seems that SVC(kernel='linear') performs worse than LinearSVC with default params setup. I will have to fine tune the SVC to LinearSVC if I want to use the predict_proba method. $\endgroup$ – MikeL Apr 2 '17 at 8:26
  • $\begingroup$ @MikeL See the edits. $\endgroup$ – Kiritee Gak Apr 2 '17 at 10:29
  • $\begingroup$ I am not sure if this is a different question or not but the above probability calculation gives a spread match that always sums up to 100%, whereas I was hoping to get the actual probability of relation to a specific document (like a distance). For example if I have one document only i will get always 100% of a match because it is the only document in the game, even if the sentence is not relevant to the document. $\endgroup$ – MikeL Apr 25 '17 at 11:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.