I have a small (2000 samples) dataset of newspaper headlines and their humorous conterparts where only one word is changed to sound silly, for example:
Original headline: Police <officer> arrested for abuse of authority
Humorous headline: Police <dog> arrested for abuse of authority
I want to train a model to predict changed sentence by the original. I am planning to implement two models for this task: one for binary tagging of input sequences (whether a word in a sentence needs to be changed) and one for predicting sentences with changed words.
Example of Model 1 input: Police officer arrested for abuse of authority
Example of Model 1 output: <no-change> <change> <no-change> <no-change> <no-change> <no-change>
Example of Model 2 input: Police <...> arrested for abuse of authority
Example of Model 2 output: Police dog arrested for abuse of authority
I am going to use a RNN/LSTM model for sequence tagging. As for the changed word prediction task, I am thinking of either using LSTM (concatenation of two parallel LSTM layers - one to run forwards on left context of word and the other to run backwards on right context) or fine-tuning BERTForMaskedLM from huggingface/transformers.
The question is whether it would be appropraite considering the small number of data, or should I switch to some other models?