2
$\begingroup$

I am new on Machine Learning and building models but a lot of tutorials has given me the chance to learn more about this topic. I am trying to build a predictive model for detecting fake news. The percentage of data with labels 1 e 0 is the following:

       T
0    2015
1     798

It is not well balanced, unfortunately, as you can see. I split the dataset as follows:

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.30, stratify=y)

i.e. 70% train and 30% test. I hope it makes sense, though I have unbalanced classes. Then, after cleaning text by removing stopwords and punctuation (should I have done something else?), I ran different models, specifically MultiNaive Bayes, SVM and Logistic Regression, getting the following results:

MNB : 84%

  precision    recall  f1-score   support

           0       0.88      0.90      0.89       476
           1       0.45      0.40      0.42        95

    accuracy                           0.82       571
   macro avg       0.66      0.65      0.66       571
weighted avg       0.81      0.82      0.81       571

SVM: Accuracy: 0.8336252189141856

Precision: 0.5 Recall: 0.2736842105263158 (Terrible results!)

Logistic regression: 0.8546409807355516

All the tutorial show that the steps for building a good model when you have some text, are removing stopwords and punctuation and extra words. I have done all these things, but probably there will be something that I could do more to improve the results. I read that, in general, who gets results above 99% met problems like overfitting: however, I would really have liked to get a 92% (at least). What do you think? How could I improve further the models? Do you think that having unbalanced classes could have affected the results?

Any suggestions would be greatly appreciated it.

$\endgroup$
1
  • $\begingroup$ Do you have to limit yourself to Machine Learning models? I mean, are you limited to these or have you experimented with other models? $\endgroup$
    – Danny
    Oct 13 '20 at 14:40
1
+50
$\begingroup$

A few ideas:

  • As mentioned by @weareglenn in general there is no way to know if the performance obtained on some data is good or bad, unless we know the performance of other systems which have been applied to the same task and dataset. So yes, your results are "acceptable" (at least it does the minimum job of beating the random baseline). However given that that your approach is quite basic (no offense!), its reasonably likely that the performance could be improved. but that's just an educated guess, and there's no way to know by how much it can be improved.
  • To me the level of imbalance is not that bad. Given the low recall on the minority class (fake news) you could try to oversample it if you want to increase recall, but be aware that this is likely to decrease precision (i.e. increase False Positive errors = class 0 predicted as 1). In my opinion you don't have to, unless for your task you must minimize False Negative errors.
  • You could try a lot of things with the features, and I'm quite confident that there is room for improvement at this level:
    • First as mentioned by @weareglenn you should try without removing punctuation, maybe even without removing stop words.
    • Then you could play with the frequency: very often excluding the words with a low frequency in the global training vocabulary allows the model to generalize better (i.e. it avoids overfitting). Try with different minimum frequency threshold: 2,3,4,... (depends how large is your data).
    • More advanced: use feature selection, preferably with a method such as genetic learning, but it might take time because it will redo the training+test process many times. Individual feature selection (e.g. with information gain or conditional entropy) might work, but it's rarely very good.
    • If you want to go very advanced, you could even borrow methods from automatic stylometry, i.e. methods used to identify the style of a document/author (the PAN shared tasks is a good source of data/systems). Some use quite complex methods and features which could be relevant for identifying fake news. A simple thing I like to try is to use characters n-grams as features, it's sometimes surprisingly effective. You could also imagine using more advanced linguistic features: lemmas, Part-Of-Speech (POS) tags.
  • You didn't mention Decision Trees in your methods, I would definitely give it a try (random forests for the ensemble method version).
$\endgroup$
6
  • $\begingroup$ Thank you so much for all your very good advice, @Erwan. Very interesting the PAN website! May I ask you if you have some good paper to suggest for learning and see more complex approaches? I agree with you about my method, but from reading many papers, no one has written more complex approaches to follow or explained what they did to clean text and extract features. Without removing stopwords and punctuation I got 0.43 (precision, 1)) and 0.55 (recall,1); accuracy =0.80.Could you please tell me what you mean with 'playing with the frequency'?I'm using CountVectorizer for features extraction. $\endgroup$
    – LdM
    Oct 17 '20 at 17:02
  • $\begingroup$ do you mean using TF-IDF? $\endgroup$
    – LdM
    Oct 17 '20 at 17:07
  • $\begingroup$ @LucaDiMauro well I haven't followed the field recently so my references are a bit old: you could look at the unmasking method or the General Impostor method. Note that these methods are specifically meant for author identification/verification, they might be applicable to your problem but there's no guarantee. There are also a few surveys, here is a recent one but I don't know what it's worth. $\endgroup$
    – Erwan
    Oct 17 '20 at 22:18
  • $\begingroup$ Also I forgot to mention the SemEval Shared Tasks, some of them might be closer to your problem but I'm less familiar with these. In general with a shared task you should look at the "overview paper" written by the organizers (to understand how they define the problem and whether their definition fits yours), and then look at the individual papers by the participants which describe their system. Usually these are quite detailed and sometimes the software is made available on github. $\endgroup$
    – Erwan
    Oct 17 '20 at 22:23
  • $\begingroup$ About the frequency threshold: I don't know if there are any predefined functions in Python for that (probably not) but anyway if you want to do some advanced feature engineering you will have to implement things yourself, I'm afraid! For the frequency threshold it's not very difficult: you need (1) to calculate the frequency for every token in the training data (globally, not only for one instance), (2) to make a list of the tokens which have a frequency at least equal to N and (3) use only these token as features (i.e. ignore all the others). $\endgroup$
    – Erwan
    Oct 17 '20 at 22:28
2
$\begingroup$

If you have a lot of data - down-sample your negative class to achieve 50/50 split on your fake news/real news classification. If you don't have much data - you can use techniques like SMOTE to up-sample the lesser class.

You seem to have better accuracy than randomly choosing fake/real which is a good sign. Your probability of a negative class based on your data split is 71.6% - and you are able to achieve 85.4% accuracy with LogReg. Don't get too down on that (especially if you are new to ML).

I would recommend checking out Gradient Boosting or Bagging algos if this is an NLP problem - these usually yield the best results for me when I'm encountered with sparse text data in classification.

As for the punctuation and stop words this is a common first step - however it's not good general advice for any problem. Do you think the presence of exclamation points might weed out some fake news in your data? If so I would include punctuation. If not - you're probably already on the right track. Removing stop words and punc only makes sense if the context of your specific problem calls for it.

More generally - your desire to reach 92% accuracy might not be possible given the difficulty of your problem. This is not to say it's not possible but keep in mind that the tutorials you may follow online are pre-determined to show that you can get good results. Some projects are simply harder than others (and some are not even possible given the context).

Good luck!

$\endgroup$
5
  • $\begingroup$ Hi weareglenn, first of all thank you for your answer and advice. I totally agree with you about punctuation, actually, as well as to not convert all text to lower case. Could you please explain me better what you mean when you say: seem to have better accuracy than randomly choosing fake/real which is a good sign ? About 92%: I read a lot of research papers where researchers said to have reached 99.8% or values above 98%, removing punctuation and stopwords, i.e. running the basic code everyone can find in a tutorial. that is 'incredible' for me. Reading their papers, it does not seem they ran $\endgroup$
    – LdM
    Oct 12 '20 at 1:14
  • $\begingroup$ any other code for cleaning or preparing data in an according way. I do not know if an 82%-85% can be considered good as results. I have to check including punctuation in the model and see what happens. (but I would not expect better results... :() $\endgroup$
    – LdM
    Oct 12 '20 at 1:16
  • $\begingroup$ I tried with GradientBoostingRegressor in sklearn ( max_depth=2, n_estimators=best_n_estimators, learning_rate=1.0) , getting 0.22245. $\endgroup$
    – LdM
    Oct 12 '20 at 1:22
  • $\begingroup$ @LucaDiMauro - based on the split of negative to positive class you can set a benchmark for your problem by saying "I'm going to randomly pick if an article is fake news or real based on the proportion of my data that is real or fake". In your case: generate some random number and if that is over 71.6% (the % of your class imbalance) predict real news otherwise you predict fake. Tally those results & compare to your real labels & generate your scoring metrics and you have this baseline to compare your real models to. If you do this you will see that you are performing better than this baseline $\endgroup$ Oct 12 '20 at 15:44
  • $\begingroup$ Try BaggingClassifier from sklearn. Also make sure you tune hyperparameters via some sort of method (GridSearchCV for instance). You also seem to be tracking a variety of scoring metrics (accuracy, recall, precision). Try to figure out one metric you care about the most & iterate to maximize this (for example f1 score). Only use accuracy if you balance your classes via downsampling or smote $\endgroup$ Oct 12 '20 at 15:50
1
$\begingroup$

In an Imbalanced dataset, we don't look at the accuracy as a whole.
Either check the Precision/Recall ratio Or individual classes accuracy.

With that, I believe your 85% accuracy is not of much use.
Individual recall are -
Class_0 - 0.90
Class_1
- $\color{red}{0.40}$
It implies, 60 out of 100 fake news is missed

Also, support of 95 and 471 is equivalent to 20% of total data and that also not stratified on y. Not sure why is this when split is 30% and stratified.

It means, the model is not able to learn probably because of Class Imbalance. Though 798:2015 is not too Imbalanced.

Please follow the strategy to handle Imbalanced dataset e.g. Undersampling, Over-sampling, Using appropriate metrics etc. [Check internet/SE for that]

$\endgroup$
2
  • $\begingroup$ Thanks for your answer 10xAI. I will have a look at the topics you mentioned. So my results are not good at all, aren't they? Though accuracy is >80% $\endgroup$
    – LdM
    Oct 13 '20 at 21:56
  • $\begingroup$ Calculate accuracy for each class separately and decide yourself. Read these for understanding the problem. stats.stackexchange.com/a/312830/256691 ; machinelearningmastery.com/… $\endgroup$
    – 10xAI
    Oct 14 '20 at 8:21
0
$\begingroup$

Yes, having unbalanced classes will affect your results. Besides the data augmentation techniques suggested above, you could also consider using Optuna with a risk-based performance score that accounts for how undesirable false negatives are relative to false positives.

This was the motivation for my master's thesis and I would love to see it implemented somewhere. Even using ROC Area Under the Curve (AUC) is not as meaningful as risk; see the last link at the bottom of this answer for an illustrative figure.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.