# Approach to Text Classification Problem

Much of the vocabulary here is new to me so forgive me if I misspeak.

I'm attempting to find an approach to a very simple classification problem. I have a set of description such as:

House : Building with several walls some windows and a roof

Car : Vehicle with four wheels, a steering wheel, and an engine

We have a third party giving a description of what they:

My House : Red walls, a balcony, a roof and a great view

Give these loose natural language descriptions what would be a good approach to classifying "My House" as a "House". Would a simple word counting approach be applicable, perhaps with some sort of emitting dictionary {"the", "a", etc}. Any alternative algorithms to consider?

• could you add some more examples? – Stereo Oct 19 '16 at 16:51

It looks like a classical email classification problem: Spam or Ham. In the following I will use some specific slang, do not worry, look it up in Google, if you will see specific terms you do not know.

Prerequisite: ideally your dataset will be "balanced", having, say, 50% cars and 50% houses, could be 40/60 as well. Problems will arise, when one class will be say < 10%.

So called "bag of words" can be seen as a start.

Steps:

1. Your dataset consist of two columns, say, "description" and "label"
2. Label has only two values car or house
3. Tokenize your decription column (single words)
4. Cleanse the desc column: remove stop words (like and, the , a - as they do not have any value), remove punctuation, possibly stem the words, remove numbers
5. calculate document-term-matrix, possibly use TF-IDF

By the end of this operation you will have a dataset with a label column (car or house) and long number of word (or even n-gram, e.g. "great_view" is a bi-gram) columns containing binary values:

Label; vehicle, balcony, wheel, ..., ...., great_view
Car,   1,        0,      1, ....., ...., ..., 0
House, 0,        1,      0, ..., ...,         1


Then use naive bayes or logistic regression as a start to train your model. Pre-process every new description as above and use your trained model to assign a probability to be a "car" or a "house", check the confusion matrix, maybe adjust the threshold .

Everything I described can be done e.g. in R or Python. In R use the text mining package "tm".

I think that word counting is a good start. I believe you can adapt spam filtering techniques to your task. i.e If a spam filter finds a group of words is spam or not spam, your filter would classify a group of words as a description of a house and not car or a car and not a house.

From Wikipedia:Naive Bayes spam filtering

"Naive Bayes classifiers are a popular statistical technique of e-mail filtering. They typically use bag of words features to identify spam e-mail, an approach commonly used in text classification. Naive Bayes classifiers work by correlating the use of tokens (typically words, or sometimes other things), with spam and non-spam e-mails and then using Bayes' theorem to calculate a probability that an email is or is not spam. Naive Bayes spam filtering is a baseline technique for dealing with spam that can tailor itself to the email needs of individual users and give low false positive spam detection rates that are generally acceptable to users. It is one of the oldest ways of doing spam filtering, with roots in the 1990s."