How do I use NLTK Text object with Re library?

Question

I am working on to build a bag of words model from my text file. I I want to use the re.sub function from the re library. I am getting the following error;

TypeError: expected string or bytes-like object

I have coded the following;

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

""" importing data set """
from nltk.corpus import PlaintextCorpusReader
corpus_root = './'
wordlists = PlaintextCorpusReader(corpus_root, 'evidence1.txt')
wordlists.fileids()
wordlists.words('evidence1.txt')
stringx = wordlists.words(wordlists.fileids()[0])
print (stringx) 

""" cleaning the texts """
from nltk.text import Text
text = Text(stringx)
print(text)

import re
lowerx = re.sub('[^a-zA-Z0-9]','',text)

I think I need right kind of object to pass on to re.sub

Answer 1

You are passing the variable text (where you stored the nltk.Text object) to re.sub (which expects a string str object, not a nltk.Text object).

Contrary to what that class's name may suggest, Text is not a string-like object itself, it simply stores strings. That may be what misled you.

Those strings are probably what you need when calling the re-related functionality below. You can access them normally and in a pythonic via standard enumeration methods, e.g. for-looping over that instance, for word in theText.... See the code below for a full example.

import re
import random
from nltk.text import Text

# Initialize a dummy text with integers as its words.
# With random proability, we add a non-alphanumeric 
# character to the word so that we can test the regular
# expression in the original example.
text = Text([
    str(random.randrange(1, 100)) + (
        '' if random.random() >= 0.5 else 
        random.choice(['!', ',', '?', ':', ';'])
    )
    for _ in range(20)
])

# The tokens in the `Text` object can be accessed
# via enumeration (the class implements a Python iterator
# underlyingly)
print(text)
print(text.tokens[:10])
print(list(text)[:10])
print([x for x in text][:10])

# To avoid the exception, we comment the original line
# We can't pass the variable `text` to `re.sub` because
# it expects a string and `text` stores an instance of 
# NLTK's `Text` class.
#lowerx = re.sub('[^a-zA-Z0-9]','',text)

# Test for the expected behavior:
for token in text:
    lowertoken = re.sub('[^a-zA-Z0-9]', '', token)
    print('<input="%s"  output="%s">' % (token, lowertoken))