1
$\begingroup$

I am working on a project to build a text classifier of questions being asked. There are no labels provided in my data so I have chosen to go with an unsupervised approach. This solution needs to read a new question, and determine which category of classification it falls into

I have been working on a KMeans model, but despite everything I try I just cannot get any decent results. I have even hand selected a data set, cleaned it further by hand to make it as straightforward as possible, and my clusters are still overlapping, and I am getting a low silhouette score of 0.26015371238537227 (this is the highest I've been able to get). My raw data isn't very great to work with, but I still don't understand how it performs so poorly when I hand select/clean the best examples to use as a test. I have included my code for kmeans section below.

Does anyone have any suggestions? I'm a kind of at a loss here. Are there any other good ways to classify unlabeled text? I'm also relatively new to this type of work and this is my first ML based project too so I'm sure there is some knowledge gap to account for as well. I'm about ready to give up on the unsupervised approach and do some manual work to label the data myself and build a supervised model. Figured I'd check here first to see if all hope is lost or not. Any help is greatly appreciated!

    data = pd.DataFrame(text)
    data.columns = ['KM']
    tfidf = TfidfVectorizer(max_df=0.80, min_df=5, max_features=10000)
    text = tfidf.fit_transform(data['KM'].values.astype('U'))

    kmeans = KMeans(n_clusters=8, init='k-means++', max_iter=300, n_init=10, 
    random_state=20)
    kmeans.fit(text)

    clusters = kmeans.predict(text)

    pca = PCA(n_components=2)
    two_dim = pca.fit_transform(text.todense())

    scatter_x = two_dim[:, 0]
    scatter_y = two_dim[:, 1]

    plt.style.use('ggplot')

    fig, ax = plt.subplots()
    fig.set_size_inches(20,10)

    cmap = {0: 'green', 1: 'blue', 2: 'red', 3: 'orange', 4: 'black', 5: 'purple', 6: 
    'yellow', 7: 'pink'}

    for group in np.unique(clusters):
        ix = np.where(clusters == group)
        ax.scatter(scatter_x[ix], scatter_y[ix], c=cmap[group], label=group)

    ax.legend()
    plt.xlabel('PCA 0')
    plt.ylabel('PCA 1')
    plt.show()

    order_centroids = kmeans.cluster_centers_.argsort()[:, ::-1]

    terms = tfidf.get_feature_names()
    for i in range(8):
        print("Cluster %d:" % i, end='')
        for ind in order_centroids[i, :10]:
            print(' %s' % terms[ind], end='')
        print()


    score = silhouette_score(text, labels=kmeans.predict(text))
    print(score)  
$\endgroup$
4
  • $\begingroup$ Welcome to DataScienceSE. Imho you're using the wrong tool: it's not because you don't have labels that you should use unsupervised. Clustering is useful for grouping instances by similarity when one doesn't have any expectations about the resulting groups. If you know the classes that you want to obtain, you need a classification method. It's like saying I don't have a boat so I'm going by car: if the destination is an island it's not gonna work ;) $\endgroup$ – Erwan Mar 16 at 22:40
  • $\begingroup$ Hey! So would you suggest doing some manual effort to classify some training data by hand for use in a different model? Or so you have any suggestions for some other approaches to try? I've been looking at some LDA topic modeling as well but not sure how well-suited that is for providing a classification for each record of text $\endgroup$ – Jim Mar 18 at 1:39
  • $\begingroup$ Hi Jim, LDA is certainly a better option for clustering text by topic, in general this would work better than k-means and it provides you with a list of top words for each cluster/topic. However it's still unsupervised so it's not sure that the topics will correspond exactly to what you want. Note that you can play with the number of clusters/topics parameter: with a quite high value you might be able to label some of the clusters/topics manually, so essentially label a large group of instances at once. Then you could use this as training set in a supervised classifier to label the ... $\endgroup$ – Erwan Mar 18 at 11:31
  • $\begingroup$ ... rest of the instances, as some of the clusters/topics probably won't correspond to any of your classes. How well this would work depends on your data, but I think it's a reasonable balance between reliability of the predictions and annotation effort. Of course the ideal case for supervised classification is to have a large sample of manually annotated instances, but this is not always practical. $\endgroup$ – Erwan Mar 18 at 11:32
1
$\begingroup$

If you're questions are relatively short and the number of labels are not excessive, I would strongly suggest you take a look at zero-shot classification which can be used to classify data to categories without training a model. Both multi-class and multi-label classification are possible.

Specifically I recommend using a pre-trained language model such as BART MNLI

You can try it out of the box with some of your questions and labels and see how it works: TRY!

From my experience I think that unsupervised approaches won't give the desired results. So if zero-shot classification is no option for you for whatever reason, I would suggest you try something like snorkel to programmatically build some training data.

$\endgroup$
1
  • $\begingroup$ Thank you! Yeah I am also seeing why a lot of people tend to not use unsupervised learning for text clustering lol. I'll look into the other options you posted here I appreciate the feedback $\endgroup$ – Jim Mar 16 at 18:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.