# Document classification: tf-idf prior to or after feature filtering?

I have a document classification project where I am getting site content and then assigning one of numerous labels to the website according to content.

I found out that tf-idf could be very useful for this. However, I was unsure as to when exactly to use it.

Assumming a website that is concerned with a specific topic makes repeated mention of it, this was my current process:

1. Retrieve site content, parse for plain text
2. Normalize and stem content
3. Tokenize into unigrams (maybe bigrams too)
4. Retrieve a count of each unigram for the given document, filtering low length and low occurrence words
5. Train a classifier such as NaiveBayes on the resulting set

My question is the following: Where would tf-idf fit in here? Before normalizing/stemming? After normalizing but before tokenizing? After tokenizing?

Any insight would be greatly appreciated.

Edit:

Upon closer inspection, I think I may have run into a misunderstanding at to how TF-IDF operates. At the above step 4 that I describe, would I have to feed the entirety of my data into TF-IDF at once? If, for example, my data is as follows:

[({tokenized_content_site1}, category_string_site1),
({tokenized_content_site2}, category_string_site2),
...
({tokenized_content_siten}, category_string_siten)}]


Here, the outermost structure is a list, containing tuples, containing a dictionary (or hashmap) and a string.

Would I have to feed the entirety of that data into the TF-IDF calculator at once to achieve the desired effect? Specifically, I have been looking at the scikit-learn TfidfVectorizer to do this, but I am a bit unsure as to its use as examples are pretty sparse.

• Prior to training the classifier, I do format the data in tuples of ({tokenized content}, category). As I have the training set websites in a database and already categorized, it is not a problem. The labels that the classifier will be able to apply will only be those it has seen from the annotated training set, correct? – user991710 Dec 11 '14 at 0:18