3
$\begingroup$

I have a movie transcript without commas, punctuation, or newlines. Is there any NLP technique that can help to implement this?

$\endgroup$
1
  • $\begingroup$ But does it at least have capitalization? $\endgroup$
    – smci
    May 1, 2020 at 5:03

2 Answers 2

0
$\begingroup$

This can be solved with "text segmentation". NLP libraries have code for breaking given text into :

  • Sentences
  • Phrases
  • Words

With this, you can break text into sentences and insert . or ? for each sentence. Similarly, dependency tree will help with inserting some punctuation marks (not all).

Example (breaking text into sentences):

import spacy
nlp = spacy.load('en_core_web_sm')
text = "I was expecting a surplus of cute close-ups but Burton does surprisingly little to win us over He's never been big on treacle but a bit more warmth in this chilly movie which barely follows the outline of the 1941 original would have gone a long way"
text_sentences = nlp(text)
for sentence in text_sentences.sents:
    print(sentence.text)

Output is :

I was expecting a surplus of cute close-ups but Burton does surprisingly little to win us over

and

He's never been big on treacle but a bit more warmth in this chilly movie which barely follows the outline of the 1941 original would have gone a long way

More details : https://spacy.io/usage/linguistic-features

$\endgroup$
1
  • 1
    $\begingroup$ Unfortunately, this seems to work only because you have "He" with a capital letter already. I tried this on all-lowercase and didn't quite get this as output. $\endgroup$
    – ETL
    Nov 25, 2021 at 7:25
0
$\begingroup$

You can try this transformer model called Re-Punctuate.

Code:

from transformers import T5Tokenizer, TFT5ForConditionalGeneration

tokenizer = T5Tokenizer.from_pretrained('SJ-Ray/Re-Punctuate')
model = TFT5ForConditionalGeneration.from_pretrained('SJ-Ray/Re-Punctuate')

input_text = 'the story of this brave brilliant athlete whose very being was questioned so publicly is one that still captures the imagination'
inputs = tokenizer.encode("punctuate: " + input_text, return_tensors="tf") 
result = model.generate(inputs)

decoded_output = tokenizer.decode(result[0], skip_special_tokens=True)
print(decoded_output)

Example:

Input: the story of this brave brilliant athlete whose very being was questioned so publicly is one that still captures the imagination

Output: The story of this brave, brilliant athlete, whose very being was questioned so publicly, is one that still captures the imagination.

Installation: https://huggingface.co/docs/transformers/installation

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.