1
$\begingroup$

I am working to create a SVM binary classifier for classification of Tweets based on news class "Crime" and "Non Crime". I have downloaded a dataset of 6400 rows from various sources and training my model on it. While I am able to achieve over 95% accuracy, what concerns me is about its performance on new datasets. The system will be deployed on live streaming tweets, so how can I put in a feedback system such as my model is continuously updates itself? As in, say for new crime type news source which were not covered in training model, how to incorporate them further?

I am asking a broad methodology question, not some specific program related question as I would like to figure out the implementation myself :)

$\endgroup$
  • 1
    $\begingroup$ Minor suggestion to check the F1 score also, since your binary classes might not be balanced. $\endgroup$ – AN6U5 Nov 12 '15 at 15:14
  • 1
    $\begingroup$ yes the binary classes are heavily skewed, so using undersampling / oversampling to mitigate it $\endgroup$ – Koustuv Sinha Nov 12 '15 at 15:33
  • 1
    $\begingroup$ But using accuracy to score highly skewed classes is incorrect. Consider a rare cancer that occurs in 1/10,000 people. I could design a test that always returns "No Cancer" and it would be 99.99 % accurate. Read about precision and recall and how F1-score is the harmonic mean of those two metrics. $\endgroup$ – AN6U5 Nov 12 '15 at 17:10
  • $\begingroup$ sorry for not giving all results. yes precision, recall and f-score is quite low for minority, i.e crime. its 0.81,0.62 and 0.70 respectively $\endgroup$ – Koustuv Sinha Nov 12 '15 at 18:07
3
$\begingroup$

Online learning for SVMs is supported only for certain SVM options (a linear SVM with SGD can easily update for example).

There are implementations out there to address the issue of online learning for all SVMs.

https://stats.stackexchange.com/questions/26041/can-svm-do-stream-learning-one-example-at-a-time

If your model isn't terribly large it also probably isn't too bad to train a new one on all data periodically.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.