3
$\begingroup$

I wish to perform sentence tokenization for sentence without punctuation, below is the code:

import nltk

def segment_sentences(text):
    # Download the Punkt tokenizer if necessary
    nltk.download('punkt')
    
    # Tokenize the text into sentences
    sentences = nltk.sent_tokenize(text)
    
    return sentences

input_text = "hello how are you today i hope you're doing well have a great day"

sentences = segment_sentences(input_text)

# Print the segmented sentences
for sentence in sentences:
    print(sentence)

Desired output

hello how are you today
i hope you're doing well
have a great day

But current output

hello how are you today i hope you're doing well have a great day

How should I address it?

Improve this question
$\endgroup$
1
  • $\begingroup$ Can you explain how is an inference engine intended/expected to know how to segment the words into sentences? $\endgroup$
    – WestCoastProjects
    May 19 at 6:41

1 Answer 1

Reset to default
4
$\begingroup$

You can apply a previous step to add punctuation and proper casing to the text, and then segment the sentences. For this, you may use Re-punctuate. When applied to your text, Re-punctuate gives the following output:

Hello, how are you today? I hope you're doing well. Have a great day.

Then, if we apply your NLTK code to it, we obtain this output:

Hello, how are you today?
I hope you're doing well.
Have a great day.
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.