0
$\begingroup$

enter image description here

Input_String is Text_Corpus of Jane Austen Book

output Should be : ['to', 'be', 'or', 'not', 'to', 'be', 'that', 'is', 'the', 'question']

But getting this Output : ['to', 'be,', 'or', 'not', 'to', 'be:', 'that', 'is', 'the', 'question!']

$\endgroup$
  • $\begingroup$ I don't know how to remove the special characters thats why I asked the question here. u can execute & try it. $\endgroup$ – Mano Oct 22 '18 at 9:36
0
$\begingroup$

Regular expressions can be used to create a simple tokenizer and normalizer:

import re

def tokens(text):
    "List all the word tokens in a text."
    return re.findall('[\w]+', text.lower())

Otherwise, use an established library like spaCy to generate a list of tokens.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Man, u r very good programmer. Your program is 100% correct. Thanks $\endgroup$ – Mano Oct 22 '18 at 9:49
0
$\begingroup$

This can be achieved by Regular Expresiions

import re
modified_string = re.sub(r'\W+', '', input_string) # on individual tokens 
modified_string = re.sub(r'[^a-zA-Z0-9_\s]+', '', input_string) # on sentence itself.Here I have modified RegEx to include spaces as well

'\W == [^a-zA-Z0-9_], so everything except numbers,alphabets and _ would be replaced by space

| improve this answer | |
$\endgroup$
  • $\begingroup$ Output according to ur code is : 'tobeornottobethatisthequestion' $\endgroup$ – Mano Oct 20 '18 at 11:35
  • $\begingroup$ Updated the answer to include spaces is RegEx $\endgroup$ – jimmy Oct 20 '18 at 12:07
  • $\begingroup$ Still not working.. u can test ur code using any Input_string $\endgroup$ – Mano Oct 20 '18 at 12:12
  • $\begingroup$ It removes unnecessary characters from sentence. You need to do split() after this to get tokens $\endgroup$ – jimmy Oct 20 '18 at 16:43
  • $\begingroup$ Thanks but Brian Spiering program is short & easy $\endgroup$ – Mano Oct 22 '18 at 9:54
0
$\begingroup$

Here is another option for you, but it should be a bit more slow than the rest of the answers.

import string
s = 'to be, or not to be: that is the question!'
punct_set= set(string.punctuation)#Saving punctuation into a set
s = ''.join(ch for ch in s if ch not in punct_set)#Get every character and remove punct

Another way is to use the translate method. In python 3, a dictionary should be passed to the method. None maps the character that will be removed.

import string
s = 'to be, or not to be: that is the question!'
translation = dict.fromkeys(map(ord, string.punctuation), None)#Dictionary with punctuation to be removed
no_punct_s = s.translate(translation)
| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks but Brian Spiering program is short & easy $\endgroup$ – Mano Oct 22 '18 at 9:54
0
$\begingroup$

I would use regex. I can only say what would work for the input sentence I can deduce from your desired output:

s = 'to be, or not to be: that is the question!'

I simply remove all characters that are not letters (upper or lower case) or spaces.

import re

pattern = r'[^A-Za-z ]'
regex = re.compile(pattern)

result = regex.sub('', s).split(' ')

print(result)

['to', 'be', 'or', 'not', 'to', 'be', 'that', 'is', 'the', 'question']


Edit

Based on the update comment from OP - my answer can be adjusted to work on each of the words via simple interation of the sentences:

cleaned_sentenced = []    # will become a list of lists

for sentence in sentences:
    temp = [regex.sub('', word) for word in sentence]
    cleaned_sentences.append(temp)

This uses regex as defined up above.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Not working with this List :- ['alice', 'how', 'long', 'is', 'forever', 'white', 'rabbit', 'sometimes', 'just', 'one', 'second'] $\endgroup$ – Mano Oct 20 '18 at 12:57
  • $\begingroup$ You need to specify in your question exactly what the inputs could be. My answer works for what you provided. Are you passing actual sentences (single strings) or are you giving the list of words? Please edit your question to include this information, do not edit my answer. $\endgroup$ – n1k31t4 Oct 20 '18 at 13:16
  • $\begingroup$ Input_String is Text_Corpus of Jane Austen Book then I convert this corpus into the List_of_Words then I execute $\endgroup$ – Mano Oct 20 '18 at 15:44
  • $\begingroup$ @Mano - see my edit. $\endgroup$ – n1k31t4 Oct 20 '18 at 17:59
  • $\begingroup$ Thanks . Your code is also correct but Brian Spiering program is short & easy. $\endgroup$ – Mano Oct 22 '18 at 9:52
0
$\begingroup$

You can simply use the python regular expression library re. It will look something like this:

import re

def text2word(text):
    '''Convert string of words to a list removing all special characters'''
    result = re.finall('[\w]+', text.lower())
    return result

If you can log the result on the console to see the output that the function returns

For example:

string = " To be or not to be: that is the question!"
print(text2word(string))

Ouput: ['to', 'be', 'or', 'not', 'to', 'be', 'that', 'is', 'the', 'question']

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.