8
$\begingroup$

I have a document classification project where I am getting site content and then assigning one of numerous labels to the website according to content.

I found out that tf-idf could be very useful for this. However, I was unsure as to when exactly to use it.

Assumming a website that is concerned with a specific topic makes repeated mention of it, this was my current process:

  1. Retrieve site content, parse for plain text
  2. Normalize and stem content
  3. Tokenize into unigrams (maybe bigrams too)
  4. Retrieve a count of each unigram for the given document, filtering low length and low occurrence words
  5. Train a classifier such as NaiveBayes on the resulting set

My question is the following: Where would tf-idf fit in here? Before normalizing/stemming? After normalizing but before tokenizing? After tokenizing?

Any insight would be greatly appreciated.


Edit:

Upon closer inspection, I think I may have run into a misunderstanding at to how TF-IDF operates. At the above step 4 that I describe, would I have to feed the entirety of my data into TF-IDF at once? If, for example, my data is as follows:

[({tokenized_content_site1}, category_string_site1), 
 ({tokenized_content_site2}, category_string_site2), 
...
 ({tokenized_content_siten}, category_string_siten)}]

Here, the outermost structure is a list, containing tuples, containing a dictionary (or hashmap) and a string.

Would I have to feed the entirety of that data into the TF-IDF calculator at once to achieve the desired effect? Specifically, I have been looking at the scikit-learn TfidfVectorizer to do this, but I am a bit unsure as to its use as examples are pretty sparse.

$\endgroup$
8
$\begingroup$

As you've described it, Step 4 is where you want to use TF-IDF. Essentially, TD-IDF will count each term in each document, and assign a score given the relative frequency across the collection of documents.

There's one big step missing from your process, however: annotating a training set. Before you train your classifier, you'll need to manually annotate a sample of your data with the labels you want to be able to apply automatically using the classifier.

To make all of this easier, you might want to consider using the Stanford Classifier. It will perform the feature extraction and build the classifier model (supporting several different machine learning algorithms), but you'll still need to annotate the training data by hand.

$\endgroup$
  • $\begingroup$ Prior to training the classifier, I do format the data in tuples of ({tokenized content}, category). As I have the training set websites in a database and already categorized, it is not a problem. The labels that the classifier will be able to apply will only be those it has seen from the annotated training set, correct? $\endgroup$ – user991710 Dec 11 '14 at 0:18
  • $\begingroup$ Re: "The labels that the classifier will be able to apply will only be those it has seen from the annotated training set, correct?" Correct. In supervised learning, the classifier won't be able to create new/unseen categories. If you want to do that, you should look into something like clustering or topic modeling. $\endgroup$ – Charlie Greenbacker Dec 11 '14 at 11:36
  • $\begingroup$ Thank you very much for the information! As you answered my question as well, I'll be accepting this as the answer. $\endgroup$ – user991710 Dec 11 '14 at 12:10
  • $\begingroup$ My apologies for bringing this back after accepting the above answer, but I reckoned that I would have better odds of getting an answer to my update if I asked you directly. So as to avoid long comments, I would greatly appreciate it if you could take a look at my edit in the original post. $\endgroup$ – user991710 Dec 11 '14 at 16:28
  • 1
    $\begingroup$ RE: "would I have to feed the entirety of my data into TF-IDF at once?" Yes, that's how it works. RE: "I have been looking at the scikit-learn TfidfVectorizer to do this, but I am a bit unsure as to its use as examples are pretty sparse." Here's an example I wrote: github.com/charlieg/… -- it's probably best if you use a corpus of documents as input, rather than some dict+string tuple you created. $\endgroup$ – Charlie Greenbacker Dec 14 '14 at 14:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.