0
$\begingroup$

enter image description here

Input_String is Text_Corpus of Jane Austen Book

output Should be : ['to', 'be', 'or', 'not', 'to', 'be', 'that', 'is', 'the', 'question']

But getting this Output : ['to', 'be,', 'or', 'not', 'to', 'be:', 'that', 'is', 'the', 'question!']

$\endgroup$
1
  • $\begingroup$ I don't know how to remove the special characters thats why I asked the question here. u can execute & try it. $\endgroup$
    – Mano
    Oct 22, 2018 at 9:36

5 Answers 5

1
$\begingroup$

Regular expressions can be used to create a simple tokenizer and normalizer:

from __future__ import annotations
import re

def tokens(text: str) -> list(str):
    "List all the word tokens in a text."
    return re.findall('[\w]+', text.lower())

assert tokens("To be, or not to be, that is the question:") == ['to', 'be', 'or', 'not', 'to', 'be', 'that', 'is', 'the', 'question']

Otherwise, use an established library like spaCy to generate a list of tokens.

$\endgroup$
1
  • 1
    $\begingroup$ Man, u r very good programmer. Your program is 100% correct. Thanks $\endgroup$
    – Mano
    Oct 22, 2018 at 9:49
0
$\begingroup$

This can be achieved by Regular Expresiions

import re
modified_string = re.sub(r'\W+', '', input_string) # on individual tokens 
modified_string = re.sub(r'[^a-zA-Z0-9_\s]+', '', input_string) # on sentence itself.Here I have modified RegEx to include spaces as well

'\W == [^a-zA-Z0-9_], so everything except numbers,alphabets and _ would be replaced by space

$\endgroup$
5
  • $\begingroup$ Output according to ur code is : 'tobeornottobethatisthequestion' $\endgroup$
    – Mano
    Oct 20, 2018 at 11:35
  • $\begingroup$ Updated the answer to include spaces is RegEx $\endgroup$
    – jimmy
    Oct 20, 2018 at 12:07
  • $\begingroup$ Still not working.. u can test ur code using any Input_string $\endgroup$
    – Mano
    Oct 20, 2018 at 12:12
  • $\begingroup$ It removes unnecessary characters from sentence. You need to do split() after this to get tokens $\endgroup$
    – jimmy
    Oct 20, 2018 at 16:43
  • $\begingroup$ Thanks but Brian Spiering program is short & easy $\endgroup$
    – Mano
    Oct 22, 2018 at 9:54
0
$\begingroup$

Here is another option for you, but it should be a bit more slow than the rest of the answers.

import string
s = 'to be, or not to be: that is the question!'
punct_set= set(string.punctuation)#Saving punctuation into a set
s = ''.join(ch for ch in s if ch not in punct_set)#Get every character and remove punct

Another way is to use the translate method. In python 3, a dictionary should be passed to the method. None maps the character that will be removed.

import string
s = 'to be, or not to be: that is the question!'
translation = dict.fromkeys(map(ord, string.punctuation), None)#Dictionary with punctuation to be removed
no_punct_s = s.translate(translation)
$\endgroup$
1
  • $\begingroup$ Thanks but Brian Spiering program is short & easy $\endgroup$
    – Mano
    Oct 22, 2018 at 9:54
0
$\begingroup$

I would use regex. I can only say what would work for the input sentence I can deduce from your desired output:

s = 'to be, or not to be: that is the question!'

I simply remove all characters that are not letters (upper or lower case) or spaces.

import re

pattern = r'[^A-Za-z ]'
regex = re.compile(pattern)

result = regex.sub('', s).split(' ')

print(result)

['to', 'be', 'or', 'not', 'to', 'be', 'that', 'is', 'the', 'question']


Edit

Based on the update comment from OP - my answer can be adjusted to work on each of the words via simple interation of the sentences:

cleaned_sentenced = []    # will become a list of lists

for sentence in sentences:
    temp = [regex.sub('', word) for word in sentence]
    cleaned_sentences.append(temp)

This uses regex as defined up above.

$\endgroup$
5
  • $\begingroup$ Not working with this List :- ['alice', 'how', 'long', 'is', 'forever', 'white', 'rabbit', 'sometimes', 'just', 'one', 'second'] $\endgroup$
    – Mano
    Oct 20, 2018 at 12:57
  • $\begingroup$ You need to specify in your question exactly what the inputs could be. My answer works for what you provided. Are you passing actual sentences (single strings) or are you giving the list of words? Please edit your question to include this information, do not edit my answer. $\endgroup$
    – n1k31t4
    Oct 20, 2018 at 13:16
  • $\begingroup$ Input_String is Text_Corpus of Jane Austen Book then I convert this corpus into the List_of_Words then I execute $\endgroup$
    – Mano
    Oct 20, 2018 at 15:44
  • $\begingroup$ @Mano - see my edit. $\endgroup$
    – n1k31t4
    Oct 20, 2018 at 17:59
  • $\begingroup$ Thanks . Your code is also correct but Brian Spiering program is short & easy. $\endgroup$
    – Mano
    Oct 22, 2018 at 9:52
0
$\begingroup$

You can simply use the python regular expression library re. It will look something like this:

import re

def text2word(text):
    '''Convert string of words to a list removing all special characters'''
    result = re.finall('[\w]+', text.lower())
    return result

If you can log the result on the console to see the output that the function returns

For example:

string = " To be or not to be: that is the question!"
print(text2word(string))

Ouput: ['to', 'be', 'or', 'not', 'to', 'be', 'that', 'is', 'the', 'question']

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.