1
$\begingroup$

I'm sure I probably did something stupid but I'm trying to fit a simple SVC classifier on MNIST dataset as an example, and it completely failed by only predicting result 1 (sometimes 7 depends on how I slice the data) for all testing set.

to get data

from sklearn.datasets import fetch_openml
mnist = fetch_openml('mnist_784', version=1)

X, y = mnist["data"], mnist["target"]
y = y.astype(np.uint8)
X_train, X_test, y_train, y_test = X[:60000], X[60000:], y[:60000], y[60000:]

to train a SVC data

from sklearn.svm import SVC

svm_clf = SVC(gamma="auto", random_state=45, probability = True)
svm_clf.fit(X_train[:5000], y_train[:5000]) # y_train, not y_train_5
svm_clf.predict_proba(X_test[0:100])

to save time I only trained on a small subset of training data, and tested on a small subset of testing data. however all prediction results are "1".

svm_clf.predict(X_test[0:100])
Out[14]: 
array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], dtype=uint8)

I tried to train a random forest on same training set

from sklearn.ensemble import RandomForestClassifier
forest_clf = RandomForestClassifier()
forest_clf.fit(X_train[:5000], y_train[:5000])
forest_clf.predict(X_test[0:100])

and at least I'm getting reasonable results and a reasonable accuracy rate without any fine tuning.


forest_clf.predict(X_test[0:100])
Out[39]: 
array([7, 2, 1, 0, 4, 1, 4, 9, 2, 9, 0, 1, 9, 0, 1, 5, 9, 7, 5, 4, 9, 6,
       2, 5, 7, 0, 7, 4, 0, 1, 3, 1, 3, 6, 7, 2, 7, 1, 3, 1, 1, 7, 4, 2,
       3, 5, 1, 2, 4, 4, 6, 3, 9, 5, 2, 0, 4, 1, 9, 4, 7, 2, 6, 3, 9, 5,
       5, 4, 3, 0, 7, 0, 0, 4, 1, 7, 3, 7, 9, 7, 7, 6, 2, 7, 2, 4, 7, 3,
       4, 1, 3, 6, 4, 3, 1, 6, 1, 7, 6, 9], dtype=uint8)

y_test[0:100]
Out[41]: 
array([7, 2, 1, 0, 4, 1, 4, 9, 5, 9, 0, 6, 9, 0, 1, 5, 9, 7, 3, 4, 9, 6,
       6, 5, 4, 0, 7, 4, 0, 1, 3, 1, 3, 4, 7, 2, 7, 1, 2, 1, 1, 7, 4, 2,
       3, 5, 1, 2, 4, 4, 6, 3, 5, 5, 6, 0, 4, 1, 9, 5, 7, 8, 9, 3, 7, 4,
       6, 4, 3, 0, 7, 0, 2, 9, 1, 7, 3, 2, 9, 7, 7, 6, 2, 7, 8, 4, 7, 3,
       6, 1, 3, 6, 9, 3, 1, 4, 1, 7, 6, 9], dtype=uint8)

so my question is, what am I missing here that the SVC completely failed predicting anything useful? I understand that fine-tuning is required to achieve good results but it's strange to think SVC completely failed at predicting anything and output all results regardless how I sliced trianing/testing set.

thanks

$\endgroup$
  • $\begingroup$ Did you try shuffle the input data, as the first 5000 might contain one number? $\endgroup$ – Andreas Storvik Strauman Jul 27 at 15:31
  • $\begingroup$ Also, try setting the kernel parameters. From the docs, it appears to be the gamma argument? $\endgroup$ – Andreas Storvik Strauman Jul 27 at 15:34
  • $\begingroup$ Thank you. If the first 5000 contain one number, then the RF wouldn't train correctly (even without tuning) most of the time. But no I didn't shuffle, because the original dataset is already shuffled, and first 5000 contain all training ys. $\endgroup$ – wizzo Jul 28 at 0:14
  • $\begingroup$ In addiiton, I'm expecting something fundamentally wrong with my approach in SVC (though I don't know what it is), because even without tuning, it shouldn't predict all results the same label. I'd understand if a poorly tuned vector machine has a accuracy of say 30%, instead of 10% coming from only predicting the same number. Also the default gamma is 1/feature. I wonder if anyone can reproduce the problem I'm having? $\endgroup$ – wizzo Jul 28 at 0:17
  • $\begingroup$ Honestly, I would not be very surprised if this is a result of the cost parameter being set too small. Your margin is probably too wide. Have you tried increasing the cost parameter? The cost parameter is vital for SVMs to be effective. Random forests often work okay without much tuning but this is not the same for other algorithms. $\endgroup$ – aranglol Jul 28 at 5:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.