0
$\begingroup$

I would like to ask you how to use classifier and determine accuracy of models. I have my dataset and I already cleaned the text (remove stopwords, punctuation, removed empty rows,...). Then I split it into train and test. Since I want to determine if an email is spam or not, I have used the common classifiers, I.e. Naive Bayes, SVM and logistic regression. Here I just included my train and test datasets: nothing else! I am using Python to run this analysis. My question is: should it be enough or should I implement new algorithms?

If you could provide me with an example of how an already existing algorithm was improved it would be also good.

I read a lot of literature regarding accuracy of text classification and in all the papers the authors use SVM, Naïve Bayes, logistic regression to classify spam. But I do not know if they built their own classifier or just used the existing one in Python.

Any experience on this?

$\endgroup$
1
$\begingroup$

The question mixes two different notions: models (or algorithms) and accuracy. Let me clarify them.

  1. Model (or Algorithm) is a classification technique and 'Accuracy' is one of the ways to evaluate the performance of the models.

  2. You can choose any models(Naive bayes, SVM or other deep learning techniques) to implement your classifier. They are independent of 'Accuracy' or 'F1' or any other measures by which you want to test the performance.

  3. At first, you shall pre-process the text (remove stopwords, punctuation, etc.) , and pre-processing is a choice that says how the data should look like before getting into the model. They do influence the model's performance, but not to a great extent when done right. Usually, pre-processing is applied on both the train and test set.

  4. Model performance: Once you implement you model, you may want to see how well it generalises (performamce on unseen data). So, you shall split the dataset into two halves: training and test set. (Usually most of the authors split into 3 portions: training set, validation set(to avoid over-fitting) and test set). You shall train the model with training set, and the test set is used to evaluate the performance of the model.

  5. Model evaluation: Once the model is trained on training data, predict the labels on the test set. So, you have two set of labels on test set: 1: ground truth (the actual labels indicated by the test set) and 2: predicted labels (the labels predicted by the model). Now, use the evaluation metric of your choice (Let's assume that you want to choose 'accuracy' as evaluation metric). Accuracy can simply be calculates as: (#Number of correctly predicted samples / #Total number of samples) * 100 . Where #Number of correctly predicted samples is the count of samples for which the ground truth label and predicted labels are the same.

$\endgroup$
7
  • 1
    $\begingroup$ Thank you so much Ashwin. So after preparing the dataset, cleaning it and split it, the only thing to do is just to apply algorithms. I do not need to built a new algorithm from scratch? $\endgroup$ – Math Aug 14 '20 at 15:29
  • $\begingroup$ My doubt was the following: when I read about how people determine spam/not spam, fake l/not fake news, all of them mention only algorithms (SVM, Naïve Bayes,..) without specifying anything. Then they mention the accuracy of test. So what I would like to understand is if the only thing which is important is to clean/prepare data and adding manually labels first as data usually are got without labels. $\endgroup$ – Math Aug 14 '20 at 15:36
  • $\begingroup$ Usually, if you go with standard datasets, they are already labelled. Cleaning/Preparing the data is required, but it is also dependent on the algorithm or the features for algorithm that you would like to use. $\endgroup$ – Ashwin Geet D'Sa Aug 14 '20 at 15:47
  • $\begingroup$ If you starting with your own data, then you may have to label the data that you collect. $\endgroup$ – Ashwin Geet D'Sa Aug 14 '20 at 15:47
  • $\begingroup$ So the main part would be preparing the data; then, the model chosen would do the rest, would it not? I have this doubt as when I think of email classification I have some words which can allow me to distinguish between spam/not spam. Of course, if I have a labelled dataset, I should get a vocabulary made by all words from spam set and another one made by words from not spam set. Then, based on their frequency in each email in the test set, I should be able to classify them. Am I missing something? $\endgroup$ – Math Aug 14 '20 at 15:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.