3
$\begingroup$

I wish to perform sentence tokenization for sentence without punctuation, below is the code:

import nltk

def segment_sentences(text):
    # Download the Punkt tokenizer if necessary
    nltk.download('punkt')
    
    # Tokenize the text into sentences
    sentences = nltk.sent_tokenize(text)
    
    return sentences

input_text = "hello how are you today i hope you're doing well have a great day"

sentences = segment_sentences(input_text)

# Print the segmented sentences
for sentence in sentences:
    print(sentence)

Desired output

hello how are you today
i hope you're doing well
have a great day

But current output

hello how are you today i hope you're doing well have a great day

How should I address it?

$\endgroup$
1
  • $\begingroup$ Can you explain how is an inference engine intended/expected to know how to segment the words into sentences? $\endgroup$ Commented May 19, 2023 at 6:41

1 Answer 1

4
$\begingroup$

You can apply a previous step to add punctuation and proper casing to the text, and then segment the sentences. For this, you may use Re-punctuate. When applied to your text, Re-punctuate gives the following output:

Hello, how are you today? I hope you're doing well. Have a great day.

Then, if we apply your NLTK code to it, we obtain this output:

Hello, how are you today?
I hope you're doing well.
Have a great day.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.