# Creating Bag of words

What is the best technology to be used to create my custom bag of words with N-grams to apply to. I want to know a functionality that can be achieved over GUI. I cannot use spot fire as it is not available in the organization. Though i can get SAP Hana or R-hadoop. But R-hadoop is bit challenging, any suggessions.

create my custom bag of words with N-grams to apply to

My initial recommendation would be to use the NLTK library for Python. NLTK offers methods for easily extracting bigrams from text or ngrams of arbitrary length, as well as methods for analyzing the frequency distribution of those items. However, all of this requires a bit of programming.

a functionality that can be achieved over GUI

That's tricky. Have you looked into GATE? I'm not exactly sure if/how GATE does what you want, but it does offer a GUI.

• I did look at GATE, i was not successful. I am not familiar with python. Any other tool.. I know spotfire but its way too expensive.. – Anup Ash Sep 10 '14 at 16:33
• Sorry, I don't know of any easy-to-use free tools that do exactly what you want to do. – Charlie Greenbacker Sep 11 '14 at 13:56
• This is as easy as it is going to get, if you want to DIY without spending money. – Emre Oct 10 '14 at 16:22

You can use SKlearn, It is a python library. It is simplest method which i like with minimal code. You can follow this link http://scikit-learn.org/stable/modules/feature_extraction.html

• Thanks Harjeet, SKlearn is simple but it does not solve my problem. SKlearn is more of analytical algorithm approach which R also provides but I am looking for something like we do in Google, put in n-words and it gives all suitable hits.. – Anup Ash Sep 10 '14 at 16:43

Check out LightSide for a GUI introduction to text mining in general, and more specifically for generating features quickly. It was developed (and I believe is still being developed) by researchers at CMU and it's how I got hooked on text mining. There is quite a bit of functionality right out of the box, and you can quickly import CSV data into the application, extract features, and start running experiments. It also makes use of suites of algorithms from several other prominent, well-regarded open source toolkits like Weka and LibLinear so you know you're using something credible under the hood. That being said, both of these last mentioned toolkits are definitely worth checking out for added functionality, even if they have a bit of a steeper learning curve. Hope that helps.

Another library that I use that has not been mentioned yet is gensim. Its Dictionary module allows you to convert a list of words into a list of (id,count) pairs. It also has an allow_update variable that updates the dictionary in the case that new words are encountered at runtime.

It also has built-in support for TF-IDF, LSI, and LDA, among other models.