I would like to return a simple “true” or “false” for a given string which determines whether it is an incomplete sentence, like “which is why they usually”, or a complete entity, such as a chapter title or element in a list, for example “The boardwalk”.

I am open to using any machine learning architecture for this task and it’s possible I would provide context for the strings by showing surrounding lines for the algorithm to see if the sentence fragment fits in to some continuing text or not, but for a first draft I’ll probably just focus on the individual lines in isolation.

What would be the most standard way to approach this?

For example, maybe there’s already some pre-trained model that knows if something is a sentence fragment or not, which I could use.

Otherwise, if not, does it matter what kind of neural network I use, or is there a most standard one from PyTorch?

Is there a clever way to avoid needing to find or create training data? Can an unsupervised learning method naturally group lines of text into those which are similar and those which are different?

Or if I need to train it, are there algorithms which learn quickly, perhaps from just 100 examples?

Otherwise, I guess I will have to find or create a large source of data to train it on?

What would be the best way to approach this?

  • $\begingroup$ My guess is that you probably can do this almost directly using a language model. $\endgroup$
    – Erwan
    Oct 4, 2021 at 19:21
  • $\begingroup$ Thanks. Could you provide a little more detail about for example what library, algorithm or method you would use and how you would train it? Thank you very much $\endgroup$ Oct 5, 2021 at 7:06
  • $\begingroup$ I'm not up to date with the state of the art, I know the old n-gram language models from 10 years ago but the best language models are certainly now made with DL. The general principle is this: a language model is trained on a large corpus of sentences, then it can be applied to predict a probability that the input text is a valid sentence. $\endgroup$
    – Erwan
    Oct 5, 2021 at 11:41


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.