Which machine (or deep) learning methods could suit my text classification problem?

Question

I am long-time engineer with almost zero machine learning experience, who is trying to determine a good starting point to solve my problem (hopefully using machine learning).

The problem (I'll keep it simple):

• Ultimately, I wish to be able to automatically assign a category to a financial transaction description
• For example: "121217 POST XX123 TONYS COFFEE" with an amount of $5, should map to "Food & Drink"
• Transaction descriptions are unstructured and often consist of unnatural language; also sometimes words can be truncated, or concatenated to others words
• There may or may not be "features" such as country/region codes, dates, etc., in a description
• Overall, there is no guarantee to the order or structure of the tokens in a description
• A transaction will always have an amount

Assumptions:

• I maintain a list of potential categories (maybe, 20 or 30 in total)
• I could maintain a huge list of business names mapped to their distinctive category (may not be necessary to use, though?)
• There is an existing set of training labelled data (raw descriptions/amounts and the category they belong to; in the thousands, not millions)

A best effort to extract a business name from a description, could be considered, but it would be great if that was not necessary for the accuracy of a final system.

I was originally thinking about NLP somewhat, but as this is fairly non-natural language, with no semantics, I believe there is no use of NLP. NER (named entity recognition) is maybe not really helpful either, as it is generally required to understand a text somewhat in order to determine entities.

I am toying with the idea of downloading GloVes pre-trained word vectors (https://nlp.stanford.edu/projects/glove/), to help determine words related to categories, though I am unsure how as of now or how well that might work. The idea might be if I trained something to say "Jimmy's Diner" -> "Food & Drink", then, for example, "Bobby's Rest" might map to "Food & Drink" too, as that is the nearest category to it in terms of word relationships/distance. It's depends on the possibility of being able to query the word embedding in such a way, as well as train it.

I guess in other to train a system using labelled data, I'd need to extract features from the description. The problem is, what features? Some features might be useless (unique identifiers, concatenated words, etc.). I'd need the system to be somewhat forgiving in terms of polluting it with useless features (avoidable to a degree, but probably not inevitable).

Either way, it would be great to hear how some of you experts might begin to approach this: what ML techniques would you deem most suitable?

I have researched machine learning and deep learning & the associated frameworks quite a bit over the last few days, but there is so many areas, with so much potential, that it is hard to to know where to begin.

Answer 1

You state that some of the words may occasionally be truncated or concatenated. Thus I would extract n-grams from your Strings and then use that into a bag-of-words vector.

How does this work?

---

n-grams is a feature extraction technique for language based data. It segments the Strings such that roots of words can be found, ignoring verb endings, pluralities etc...

The segmentation works as follows:

The String: Hello World

2-gram: "He", "el", "ll", "lo", "o ", " W", "Wo", "or", "rl", "ld" 3-gram: "Hel", "ell", "llo", "lo ", "o W", " Wo", "Wor", "orl", "rld" 4-gram: "Hell", "ello", "llo ", "lo W", "o Wo", " Wor", "Worl", "orld"

Thus in your example, if we use 4-grams, truncations of the word Hello would appear to be the same. And this similarity would be captured by your features.

---

Bag-of-Words builds a dictionary of the words it has seen during the training phase. Then using the word the frequency of each word in the example a vector is created. This can then be used with any standard machine learning technique.

---

Due to the high number of grams that will result, you will want to do some feature dimensionality reduction. You can use techniques such as PCA and LDA to determine which features (grams) are most pertinent to your decision boundary.

Answer 2

Referencing @JahKnows answer, I think what he may be trying to refer to is stemming not n-grams. (I would've commented but not enough reputation). Using the python package nltk it should allow you to stem the words given that you want to only get the roots of each word.

From what I've worked on, n-grams are sequences of words with length n. They are still very helpful for the bag of words suggestion. The python library scikit-learn has a class CountVectorizer which lets you create this model. In addition, it also lets you set a max number of features which can act as your feature reduction.

Further, if you were going this route, I would continue and use Tf-idf which will find out 'how important' a word is to a document. You can read more on the wiki. Scikit-learn has a great tutorial on working with text data which utilizes both CountVectorizer and Tf-idf: Working With Text Data.

The last steps would be is to choose a multi-class classification model and feed it into a OneVsRestClassifier (all in scikit-learn) and choose which one works the best. (As a start you can try Naive Bayes, SVM, etc.) This should work well since you already have a lot (I'm assuming) of labeled data.

Good luck!

Answer 3

Facebook's fasttex is really great for loosely formatted text. Fasttext is a library for doing several tasks, the one you'll want to use is text classification. The best part of fasttext it uses character n-grams that instead of working with full words (which @JahKnows described well). The hardest part of using the tool is formatting the training data properly, but just look through the provided example and you should get it pretty quickly.