I've got labeled data in a csv which looks like:

Women Jacket A,Clothes
Mens Running Shoes B,Shoes
Children backpack,Bags

and a script:

from textblob.classifiers import NaiveBayesClassifier

with open('train_small.csv', 'r') as fp:
    cl = NaiveBayesClassifier(fp, format="csv")

print ( cl.classify("Womens running shoe") )

If I use train_small.csv with 3 categories and 32 rows, I get the printed result, correct and immediate.

If I use my full training set (train.csv, 16k rows, 10 categories) I won't get a result. Today, I ran my laptop for 6 hours and then I had to shut it down by pressing the power button since it was at a load of 60 and irresponsive.

System is:

Operating System: Kubuntu 19.04
Kernel Version: 5.0.0-15-generic
OS Type: 64-bit
Processors: 4 × Intel® Core™ i5-7200U CPU @ 2.50GHz
Memory: 7.7 GiB of RAM

Not great, but completely insufficient ..? Do I need to use another library / algorithm or just different hardware?

  • $\begingroup$ How large is the full dataset? Maybe loading fails because of RAM limit. Can you monitor RAM usage during the loading process? If it is fully employed, loading data could be the problem. $\endgroup$
    – Peter
    Jun 2 '19 at 19:49

Oh Wow, textblob's default setting can't deal with even a reasonable amount of data.

TL;DR: You do need better control over what the library does.

The class NaiveBayesClassifier, or more exactly it's superclass NLTKClassifier uses a default for of feature_extractor. In this case it uses basic_extractor, which, apart from stemming the words in some way (not relevant for us here) it does the following:

features = dict(((u'contains({0})'.format(word), (word in tokens))
    for word in word_features))

Once per each document (sample).

Ouch! That will create a python dictionary for every single document. And store in these dictionaries the term frequecy for each document. On 16k rows (documents/samples) that is very likely to be way too much memory.

On Ubuntu you likely have swap space, which is extra disk space used as memory, your OS figuring out it is out of memory starts swapping pages of memory onto this swap space. Yet, swap space if super slow compared to actual memory, therefore the computation takes forever. And, in the end, it is this swap space and this slowness that keeps the algorithm running forever, instead of overheating your CPU or making the machine completely frozen and out of memory.

Better approach to the problem

ML techniques do not understand words or strings in general, they only understand numbers. textblob hides this fact from you because it performs the feature extraction behind the scenes. That works for toy problems but will likely impair the solution of any more complex problem (be it because of problem complexity or amount of data, or both).

You need to take control of the feature extraction yourself. A Naive Bayes classifier is a rather simple algorithm and will work well with a plethora of extraction techniques. Term Frequency (simply counting the number of each distinct word in every document), will work well enough as long as you do not use expensive python dictionaries to hold the frequencies in memory. But once you get control of your feature extraction there are many more techniques to use: TF-IDF, n-grams. I find the sklearn's explanation of text features very enlightening. But I digress.

Unfortunately nltk (the library under texblob) does not make things any easier. It requires its training sets to have a dictionary interface. In other words, without a good deal of hacks nltk cannot train on big amounts of data.

Option 1: Hook into nltk directly

First lets assume that your file is named data.csv and we will try to extract features reading the file line-by-line. First let me generate a data file:

import nltk
nltk.download('gutenberg')  # just in case
from nltk.corpus import gutenberg
corpus = gutenberg.words('melville-moby_dick.txt')
with open('data.csv', 'w') as f:
    for i in range(10**4):  # 10k
        f.write(' '.join([w for w in corpus[i*3:i*3+3] if w != ','])
                + ',' + str(i%2) + '\n')

This was only generation of some data, nothing too relevant to the problem at hand. I do not have your file so I just generate a random 10k rows CSV using the words from Moby Dick text at random.

Onto the actual code. We will still use texblob's Naive Bayes but will not allow it to construct its huge list of dictionaries. Instead, we will hook into it directly:

from textblob.classifiers import NaiveBayesClassifier
import csv

cls = NaiveBayesClassifier('')

class FeatureDict(object):
    def __iter__(self):
        with open('data.csv') as f:
            for row in csv.reader(f):
                yield {w: w for w in row[0].split()}, row[1]

cls.train_features = FeatureDict()  # this is passed to nltk
cls.classify('bang Boom!')

It is a hack but runs in constant memory.

Option 2: switch to sklearn

sklearn's vectorizers will perform the operation within NumPy (or within the NumPy arrays inside pandas' data frames). Which are memory efficient - especially if compared with thousands of python dictionaries.

With sklearn one will need to do this in two steps: one for data vectorization and another for the prediction. make_pipeline joins the steps. Assuming the same data prepared in data.csv we can do:

from sklearn.feature_extraction.text import tfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import make_pipeline
import pandas as pd

data = pd.read_csv('data.csv', names=['doc', 'label'])
# some words are badly formatted, sklearn have issues with them
data = data.dropna()
model = make_pipeline(TfidfVectorizer(), MultinomialNB())
model.fit(data['doc'], data['label'])
model.predict(['buggy ho!'])

The sklearn solution is slightly more elegant but may blow memory at some point (500k samples perhaps). In that case, one would need to hack sklearn itself to do things in constant memory. 

  • $\begingroup$ About y: Where would that one be used? And NaiveBayesClassifier will come from from nltk.classify import NaiveBayesClassifier or from textblob.classifiers import NaiveBayesClassifier? I first used textblob and received this error: ValueError: too many values to unpack then I changed to nltk and receive now: AttributeError: 'list' object has no attribute 'samples'. So I guess something about my training data must be wrong (or the feature extractor?) At the moment, I'm using the train_small.csv for testing. $\endgroup$
    – Chris
    Jun 5 '19 at 19:11
  • $\begingroup$ @Chris - You're correct: I was wrong on nltk behaviour. I tested and nltk did break for me on lots of data too. This requires some good hacking to do in constant memory, I have updated the answer considerably. $\endgroup$
    – grochmal
    Jun 6 '19 at 23:07
  • $\begingroup$ Option 1 was working out technically. The code would run. But it always predicted only one category. So I looked into sklearn and decided to switch. I've used your approach and extended it with this tutorial: kaggle.com/kinguistics/… Working smooth now, thank you very much :) $\endgroup$
    – Chris
    Jun 8 '19 at 4:35
  • $\begingroup$ @Chris - nltk seems to be dependent on data ordering (but that was from a quick look at the code), sklearn will shuffle data before training by default. So that is where my suspicion lies. You can try shuffling by hand (perhaps with a smaller set) and see if things go better. But that is a theoretical problem now. $\endgroup$
    – grochmal
    Jun 9 '19 at 1:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.