# Is there any NLP library or package which can help in adding comma, punctuation, newlines appropriately to text?

I have a movie transcript without commas, punctuation, or newlines. Is there any NLP technique that can help to implement this?

• But does it at least have capitalization?
– smci
May 1, 2020 at 5:03

This can be solved with "text segmentation". NLP libraries have code for breaking given text into :

• Sentences
• Phrases
• Words

With this, you can break text into sentences and insert . or ? for each sentence. Similarly, dependency tree will help with inserting some punctuation marks (not all).

Example (breaking text into sentences):

import spacy
text = "I was expecting a surplus of cute close-ups but Burton does surprisingly little to win us over He's never been big on treacle but a bit more warmth in this chilly movie which barely follows the outline of the 1941 original would have gone a long way"
text_sentences = nlp(text)
for sentence in text_sentences.sents:
print(sentence.text)


Output is :

I was expecting a surplus of cute close-ups but Burton does surprisingly little to win us over

and

He's never been big on treacle but a bit more warmth in this chilly movie which barely follows the outline of the 1941 original would have gone a long way

More details : https://spacy.io/usage/linguistic-features

• Unfortunately, this seems to work only because you have "He" with a capital letter already. I tried this on all-lowercase and didn't quite get this as output.
– ETL
Nov 25, 2021 at 7:25

You can try this transformer model called Re-Punctuate.

Code:

from transformers import T5Tokenizer, TFT5ForConditionalGeneration

tokenizer = T5Tokenizer.from_pretrained('SJ-Ray/Re-Punctuate')
model = TFT5ForConditionalGeneration.from_pretrained('SJ-Ray/Re-Punctuate')

input_text = 'the story of this brave brilliant athlete whose very being was questioned so publicly is one that still captures the imagination'
inputs = tokenizer.encode("punctuate: " + input_text, return_tensors="tf")
result = model.generate(inputs)

decoded_output = tokenizer.decode(result[0], skip_special_tokens=True)
print(decoded_output)


Example:

Input: the story of this brave brilliant athlete whose very being was questioned so publicly is one that still captures the imagination

Output: The story of this brave, brilliant athlete, whose very being was questioned so publicly, is one that still captures the imagination.