I am working on to build a bag of words model from my text file. I I want to use the re.sub function from the re library. I am getting the following error;

TypeError: expected string or bytes-like object

I have coded the following;

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

""" importing data set """
from nltk.corpus import PlaintextCorpusReader
corpus_root = './'
wordlists = PlaintextCorpusReader(corpus_root, 'evidence1.txt')
stringx = wordlists.words(wordlists.fileids()[0])
print (stringx) 

""" cleaning the texts """
from nltk.text import Text
text = Text(stringx)

import re
lowerx = re.sub('[^a-zA-Z0-9]','',text)

I think I need right kind of object to pass on to re.sub


1 Answer 1


You are passing the variable text (where you stored the nltk.Text object) to re.sub (which expects a string str object, not a nltk.Text object).

Contrary to what that class's name may suggest, Text is not a string-like object itself, it simply stores strings. That may be what misled you.

Those strings are probably what you need when calling the re-related functionality below. You can access them normally and in a pythonic via standard enumeration methods, e.g. for-looping over that instance, for word in theText.... See the code below for a full example.

import re
import random
from nltk.text import Text

# Initialize a dummy text with integers as its words.
# With random proability, we add a non-alphanumeric 
# character to the word so that we can test the regular
# expression in the original example.
text = Text([
    str(random.randrange(1, 100)) + (
        '' if random.random() >= 0.5 else 
        random.choice(['!', ',', '?', ':', ';'])
    for _ in range(20)

# The tokens in the `Text` object can be accessed
# via enumeration (the class implements a Python iterator
# underlyingly)
print([x for x in text][:10])

# To avoid the exception, we comment the original line
# We can't pass the variable `text` to `re.sub` because
# it expects a string and `text` stores an instance of 
# NLTK's `Text` class.
#lowerx = re.sub('[^a-zA-Z0-9]','',text)

# Test for the expected behavior:
for token in text:
    lowertoken = re.sub('[^a-zA-Z0-9]', '', token)
    print('<input="%s"  output="%s">' % (token, lowertoken))

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.