I am working on a text-clustering problem. My goal is to create clusters with similar context, similar talk. I have around 40 million posts from social media. To start with I have written clustering using K-Means and Tf-Idf. The following code suggests what I am doing.

Here are main steps:

  • Do some pre-processing
  • Create tfidf_matrix while using tokenization and stemming
  • Run K-Means on the tf-idf matrix
  • Have the result

    csvRows = []
    title = []
    synopses = []
    filename = "cc.csv"
    num_clusters = 20
    pkl_file = "doc_cluster.pkl"
    generate_pkl = False
    if len(sys.argv) == 1:
        print("Will use "+pkl_file + " to cluster")
    elif sys.argv[1] == '--generate-pkl':
        print("Will generate a new pkl file")
        generate_pkl = True
    # pre-process data
    with open(filename, 'r') as csvfile:
        # creating a csv reader object
        csvreader = csv.reader(csvfile)
        # extracting field names through first row
        fields = csvreader.next()
        # extracting each data row one by one
        duplicates = 0
        for row in csvreader:
        # removes the characters specified
        if line not in synopses:
            duplicates += 1
    stopwords = nltk.corpus.stopwords.words('english')
    stemmer = SnowballStemmer("english")
    def tokenize_and_stem(text):
        # first tokenize by sentence, then by word to ensure that punctuation is caught as it's own token
        tokens = [word for sent in nltk.sent_tokenize(
        text) for word in nltk.word_tokenize(sent)]
        filtered_tokens = []
        # filter out any tokens not containing letters (e.g., numeric tokens, raw punctuation)
        for token in tokens:
        if re.search('[a-zA-Z]', token):
        stems = [stemmer.stem(t) for t in filtered_tokens]
        return stems
    def tokenize_only(text):
        # first tokenize by sentence, then by word to ensure that punctuation is caught as it's own token
        tokens = [word.lower() for sent in nltk.sent_tokenize(text)
              for word in nltk.word_tokenize(sent)]
        filtered_tokens = []
        # filter out any tokens not containing letters (e.g., numeric tokens, raw punctuation)
        for token in tokens:
        if re.search('[a-zA-Z]', token):
        return filtered_tokens
    totalvocab_stemmed = []
    totalvocab_tokenized = []
    for i in synopses:
        # for each item in 'synopses', tokenize/stem
        allwords_stemmed = tokenize_and_stem(i)
        # extend the 'totalvocab_stemmed' list
        allwords_tokenized = tokenize_only(i)
    vocab_frame = pd.DataFrame(
        {'words': totalvocab_tokenized}, index=totalvocab_stemmed)
    print 'there are ' + str(vocab_frame.shape[0]) + ' items in vocab_frame'
    # define vectorizer parameters
    tfidf_vectorizer = TfidfVectorizer(max_df=0.8, max_features=200000,
                               min_df=0.0, stop_words='english',
                               use_idf=True, tokenizer=tokenize_and_stem, ngram_range=(1, 3))
    tfidf_matrix = tfidf_vectorizer.fit_transform(synopses)
    terms = tfidf_vectorizer.get_feature_names()
    # dist = 1 - cosine_similarity(tfidf_matrix)
    km = KMeans(n_clusters=10, max_iter=1000,
    clusters = km.labels_.tolist()
    # uncomment the below to save your model
    # since I've already run my model I am loading from the pickle
        joblib.dump(km,  pkl_file)
        print("Generated pkl file " + pkl_file)
    km = joblib.load(pkl_file)
    clusters = km.labels_.tolist()
    films = {'title': title,  'synopsis': synopses, 'cluster': clusters, }
    total_count = len(films['synopsis'])
    csvRows = []
    for idx in range(total_count):
        'title': films['title'][idx],
        'cluster': films['cluster'][idx]
    print('Creating cluster.csv')
    with open('cluster.csv', 'w') as output:
        writer = csv.DictWriter(output, csvRows[0].keys())
        print("\ncreated cluster.csv")

The results are not very satisfactory. They are very average. What could be done to improve my clustering algorithm? I would still want to use K-Means but what another approach could be used in place of Tf-Idf?

Also, if you guys think that there is a better alternative to K-Means, please suggest and it even more helpful, if you could point me to sources/examples, where people have already done similar stuff.

I will always run the clustering on the volume close to 40 Million.

  • $\begingroup$ What about word embeddings? Is the dataset open source? $\endgroup$ – Aditya Jun 15 '18 at 17:00
  • $\begingroup$ @Aditya I have prepared the data set from twitter, blogs, forums etc. Could you give an example of how to use word-embeddings? $\endgroup$ – Suhail Gupta Jun 15 '18 at 17:54
  • $\begingroup$ Checkout fast.ai , Racheal Thomas workshop on this, Deep Learning.ai Andrew NG on YouTube there's a course on NLP Sequence Modelling $\endgroup$ – Aditya Jun 15 '18 at 18:02
  • $\begingroup$ How did you scrapped so many tweets? Can the dataset be shared? It will help me in my ongoing internship $\endgroup$ – Aditya Jun 15 '18 at 18:03
  • $\begingroup$ Text clustering is hard. Do not expect it to "just" work. In particular with algorithms such as k-means that make very different assumptions on your data... Word embeddings are all the rage, but I doubt they work actually much better. It's just that people want the results to be better. In the end, you still have 300 dimensional vectors, with plenty of anomalous documents, and k-means neither is good for high dimensions, nor for such noisy data. $\endgroup$ – Has QUIT--Anony-Mousse Jun 16 '18 at 18:16

You will likely see an improvement by using an algorithm like GloVe in place of Tf-Idf. Like Tf-Idf, GloVe represents a group of words as a vector. Unlike Tf-Idf, which is a Bag-of-Words approach, GloVe and similar techniques preserve the order of words in a tweet. Knowing what word comes before or after a word of interest is valuable information for assigning meaning. This Article runs through different techniques and gives a good description of each one. Also, This Script on Kaggle shows how to use pretrained word vectors to represent tweets.

For your clustering, I recommend checking out Density-Based clustering. K-means is a decent all-purpose algorithm, but it's a partitional method and depends on assumptions that might not be true, such as clusters being roughly equal in size. This is almost certainly not the case. This Blog has a great discussion on clustering for text. If you go with Density-Based and you use Python, I highly recommend HDBSCAN by Leland McInnes.

Good luck!

  • $\begingroup$ This all looks good. I have tried HDBSCAN (I am working with python) with a few sample examples on their documentation site. Could you point me to a good example that uses GloVe rather than Tf-Idf as an input? $\endgroup$ – Suhail Gupta Jun 16 '18 at 11:34
  • $\begingroup$ Also, is there a limit to the max posts that can be passed to HDBSCAN? When I pass tf_idf matrix as c.fit(tfidf_matrix) for a huge document set, it throws memory error $\endgroup$ – Suhail Gupta Jun 16 '18 at 12:31

You can try using n_grams.


n-grams is a feature extraction technique for language based data. It segments the Strings such that roots of words can be found, ignoring verb endings, pluralities etc...

The segmentation works as follows:

The String: Hello World

2-gram: "He", "el", "ll", "lo", "o ", " W", "Wo", "or", "rl", "ld" 3-gram: "Hel", "ell", "llo", "lo ", "o W", " Wo", "Wor", "orl", "rld" 4-gram: "Hell", "ello", "llo ", "lo W", "o Wo", " Wor", "Worl", "orld"

Thus in your example, if we use 4-grams, truncations of the word Hello would appear to be the same. And this similarity would be captured by your features.

  • $\begingroup$ I am ready using stemming in the code I shared $\endgroup$ – Suhail Gupta Jun 16 '18 at 14:58
  • $\begingroup$ Is there a way I could use GloVe / gensim here? $\endgroup$ – Suhail Gupta Jun 16 '18 at 14:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.