1
$\begingroup$

I want to predict the start and end indices of text where a certain type of propaganda technique is used like smears, name-calling, loaded language etc. Some examples from the dataset are:

['THERE ARE ONLY TWO GENDERS\n\nFEMALE \n\nMALE\n', 'This is not an accident!', "SO BERNIE BROS HAVEN'T COMMITTED VIOLENCE EH?\n\nPOWER COMES FROM THE BARREL OF A GUN, COMRADES.\n\nWHAT ABOUT THE ONE WHO SHOT CONGRESSMAN SCALISE OR THE DAYTON OHIO MASS SHOOTER?\n"]

[[[0, 41]], [], [[47, 83], [3, 14], [33, 41], [163, 175], [85, 93], [0, 176]]]

So, 0 and 41 mean that the whole text from 1st example comes under a certain category i.e. from index 0 to 41.
The next one has nothing weird in it.
Then we have 'Slogan' from 47 to 83 i.e. 'POWER COMES FROM THE BARREL OF A GUN' , and for 3 to 14 there is 'BERNIE BROS' which is highlighted as 'name calling'.

I have tried using regression here with an LSTM model but the results are very poor which I expected. I am looking for the right approach to solve this problem. Any help will be highly appreciated. Thanks!

Improve this question
$\endgroup$
2
  • $\begingroup$ Do you have labels, so do you have the indices attached to the text to learn from it? $\endgroup$
    – Peter
    Feb 16, 2022 at 10:17
  • $\begingroup$ yes, I have them. $\endgroup$
    – Asim
    Feb 16, 2022 at 10:26

1 Answer 1

Reset to default
1
$\begingroup$

You could use a "masked language model" (MLM) which predicts if a (short) piece of text or sentence belongs to some class (labels can be derived from the indices, I guess). With LSTM you only go in one direction (start to end) while with bidirectional encoder (BERT like models), you go in both directions which is a great improvement.

Original BERT uses MLM as well as "next sentence prediction" (NSP) during learning. However, maybe MLM with classification at the end may be sufficient. MLM works in the way, that you first learn the nature of the text by "masking" random words and try to predict them. This is very helpful to make a final (downstream) classification regarding to what kind of category a text belongs.

You may also use a pretrained BERT model and fine tune it. Finding relevant parts of text is one of the downstream tasks BERT can do.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I understand what MLM is and have used BERT previously for text classification but I am confused that how MLM will predict the index of words? Are you suggesting that I mask the words present in the [start, end] interval and use MLM with BERT to predict them? How will that work while inference? Can you please elaborate on that and I would accept your answer? Thanks! $\endgroup$
    – Asim
    Feb 16, 2022 at 11:51
  • 1
    $\begingroup$ I would not try to predict the index but the kind of class a short piece of text belongs to. If this works, you can feed short text sequences in a windowed function to the model and observe the "boundaries" where one class occurs and ends. However, this is just an idea and I don't know if it is really efficient. However, find tuning a pretrained BERT may be even better, but I don't know about a implementation that fit to your problem 1:1 $\endgroup$
    – Peter
    Feb 16, 2022 at 11:55
  • $\begingroup$ The issue is that I don't have the short piece of text belongs to. It also has to be predicted. This is the task2 of the "Detection of Persuasive Techniques in Texts and Images" challenge. $\endgroup$
    – Asim
    Feb 16, 2022 at 12:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.