0
$\begingroup$

Currently making a bot that condenses news articles. I'm tagging sentences as important or not important using a simple BERT classifier. The results were... not great. I'm really interested in how I can improve the results using LSTM.

I'm now batching 5 sentences together, calculating their BERT encodings, and then using 2 LSTM Layers, one backwards one forwards, to predict if the sentence is important.

Unfortunately I'm now calculating 5 times the number of embeddings, and if it doesn't work, I can't seem to figure out how to feed a variable number of things into BERT using Tensorflow, to see if I can tweak some results.

Are there other methods to add surrounding sentences to this context?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

You're trying to do Extractive Text Summarization.

You're first approach is the right one, using BERT + CLassifier.

From NLP Progress, the best method so far is Presumm.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Just to clarify, are you saying I shouldn't use the LSTM on sentence outputs of BERT? I'm also interested in maybe making a few versions of an output. I.e. very condensed, somewhat condensed, etc. Presumm seems to have some of this, but if I want to have more values, is there something similar for a multi-class classifier? $\endgroup$
    – bbbbbb
    Nov 7, 2019 at 22:05
  • $\begingroup$ What you can do (if you have the adequate data for such a training) is simply training several classifiers. BERT will be shared among all classifiers and later you can choose which classifier you want (very condensed, somewhat condensed, etc...) $\endgroup$
    – Astariul
    Nov 7, 2019 at 23:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.