# Improving SVM binary classification model on new dataset

I am working to create a SVM binary classifier for classification of Tweets based on news class "Crime" and "Non Crime". I have downloaded a dataset of 6400 rows from various sources and training my model on it. While I am able to achieve over 95% accuracy, what concerns me is about its performance on new datasets. The system will be deployed on live streaming tweets, so how can I put in a feedback system such as my model is continuously updates itself? As in, say for new crime type news source which were not covered in training model, how to incorporate them further?

I am asking a broad methodology question, not some specific program related question as I would like to figure out the implementation myself :)

• Minor suggestion to check the F1 score also, since your binary classes might not be balanced. – AN6U5 Nov 12 '15 at 15:14
• yes the binary classes are heavily skewed, so using undersampling / oversampling to mitigate it – Koustuv Sinha Nov 12 '15 at 15:33
• But using accuracy to score highly skewed classes is incorrect. Consider a rare cancer that occurs in 1/10,000 people. I could design a test that always returns "No Cancer" and it would be 99.99 % accurate. Read about precision and recall and how F1-score is the harmonic mean of those two metrics. – AN6U5 Nov 12 '15 at 17:10
• sorry for not giving all results. yes precision, recall and f-score is quite low for minority, i.e crime. its 0.81,0.62 and 0.70 respectively – Koustuv Sinha Nov 12 '15 at 18:07