Unsupervised Text Classification with Python: Kmeans

Question

I am working on a project to build a text classifier of questions being asked. There are no labels provided in my data so I have chosen to go with an unsupervised approach. This solution needs to read a new question, and determine which category of classification it falls into

I have been working on a KMeans model, but despite everything I try I just cannot get any decent results. I have even hand selected a data set, cleaned it further by hand to make it as straightforward as possible, and my clusters are still overlapping, and I am getting a low silhouette score of 0.26015371238537227 (this is the highest I've been able to get). My raw data isn't very great to work with, but I still don't understand how it performs so poorly when I hand select/clean the best examples to use as a test. I have included my code for kmeans section below.

Does anyone have any suggestions? I'm a kind of at a loss here. Are there any other good ways to classify unlabeled text? I'm also relatively new to this type of work and this is my first ML based project too so I'm sure there is some knowledge gap to account for as well. I'm about ready to give up on the unsupervised approach and do some manual work to label the data myself and build a supervised model. Figured I'd check here first to see if all hope is lost or not. Any help is greatly appreciated!

    data = pd.DataFrame(text)
    data.columns = ['KM']
    tfidf = TfidfVectorizer(max_df=0.80, min_df=5, max_features=10000)
    text = tfidf.fit_transform(data['KM'].values.astype('U'))

    kmeans = KMeans(n_clusters=8, init='k-means++', max_iter=300, n_init=10, 
    random_state=20)
    kmeans.fit(text)

    clusters = kmeans.predict(text)

    pca = PCA(n_components=2)
    two_dim = pca.fit_transform(text.todense())

    scatter_x = two_dim[:, 0]
    scatter_y = two_dim[:, 1]

    plt.style.use('ggplot')

    fig, ax = plt.subplots()
    fig.set_size_inches(20,10)

    cmap = {0: 'green', 1: 'blue', 2: 'red', 3: 'orange', 4: 'black', 5: 'purple', 6: 
    'yellow', 7: 'pink'}

    for group in np.unique(clusters):
        ix = np.where(clusters == group)
        ax.scatter(scatter_x[ix], scatter_y[ix], c=cmap[group], label=group)

    ax.legend()
    plt.xlabel('PCA 0')
    plt.ylabel('PCA 1')
    plt.show()

    order_centroids = kmeans.cluster_centers_.argsort()[:, ::-1]

    terms = tfidf.get_feature_names()
    for i in range(8):
        print("Cluster %d:" % i, end='')
        for ind in order_centroids[i, :10]:
            print(' %s' % terms[ind], end='')
        print()


    score = silhouette_score(text, labels=kmeans.predict(text))
    print(score)  

Answer 1

If you're questions are relatively short and the number of labels are not excessive, I would strongly suggest you take a look at zero-shot classification which can be used to classify data to categories without training a model. Both multi-class and multi-label classification are possible.

Specifically I recommend using a pre-trained language model such as BART MNLI

You can try it out of the box with some of your questions and labels and see how it works: TRY!

From my experience I think that unsupervised approaches won't give the desired results. So if zero-shot classification is no option for you for whatever reason, I would suggest you try something like snorkel to programmatically build some training data.