different results with MEKA vs Scikit-learn!

Question

I have a multilabel data which I've trained using different classifiers with MEKA (multilabel version of WEKA) and the evaluation results (e.g. accuracy) that MEKA gives me are different from those I get from the same classifiers of scikit-learn.

I am using Binary Relevance method (which is both in MEKA and scikit-multilearn), because of my data being multilabel. And for instance, I am using SVM which is SMO in MEKA (and WEKA) and linearSVC in sklearn.

Now the accuracy I get from 10-fold cross-validation with BR>SMO in MEKA is about 50% where I get something around 20% accuracy from BinaryRelevanvce>LinearSVC. Other evaluation measures (e.g. f1-score, hamming loss etc.) don't match as well.

I tried to give all the parameters I could find in the input of LinearSVC, the same value as those of SMO but there's no luck. My Python code would be like this:

from skmultilearn.problem_transform import BinaryRelevance
from sklearn.svm import LinearSVC

my_classifier = BinaryRelevance(
                classifier=LinearSVC(C=1, random_state=1, tol=0.001, max_iter=10000, dual=False),
                require_dense=[True, True])

I must add that using BR>SMO in MEKA to train and evaluate the model would take almost an hour on my data, but sklearn's linearSVC (with skmultilern's BinaryRelevance) takes less than 10 minutes on the same data (hence the lower accuracy).

And by the way, I'm using accuracy_score method in sklearn.metrics for accuracy.

So my question is what am I missing? How can I get the same accuracy from SVM in MEKA and scikit-learn?

Answer 1

According to this question

LinearSVC is not linear SVM, do not use it if do not have to.

and furthermore

LinearSVC [is] one of the mistakes of sklearn developers.

In this picture from the question you can observe how it is inaccurate, even compared to SVC with kernel=linear.

Further differences may be explained with differences how SMO and SVC with linear kernel are implemented.

This post compare the two, and the answer is SMO should be the fastest one, otherwise your dataset maybe is not suitable for it.

I would test SVC with linear kernel and see if things change.