How to cluster some text using TensorFlow

Question

Apologies for any inaccuracies due to the infancy in this field.

I'm trying to learn on how to return a dataset with three classes to be clustered by TensorFlow.

At this stage, I've read a lot and experimented with several Colabs, but none of them points to the clustering technique when it comes to text (just classification).

The whole point is I don't have any clue of how the text could be classified, and I though the recursive process used by TensorFlow would have offered a better approach and results of what I normally get by using K-Means.

I drafted a Colab myself, but it's a real mess of examples now, so I better not to share at this stage.

In a nutshell, I tokenised the keywords, created a dictionary, converted my text into a dense matrix and then started using the models. However, when it comes to using the fit function, I end up getting a series of error with the most recent one being "ValueError: No gradients provided for any variable".

I would appreciate some support and hints also on materials to read if any. The release of TensorFlow 2 makes things even more complicated as there are even less examples and trying to adapt the ones for TF1 not always work.

Thanks

UPDATE:

I've cleaned up the project, inferring from another example taken from here. Clearly the original example was working, but not mine.

In my version I'm trying to train the model to learn three different classes. My questions at this stage are:

• How can I create a suitable sets of labels when the Dense output is greater than 1?

  Hot encoding is not an option apparently as the label shape is not inferred from the size of the array as I initially thought, but from the values itself.

• How can I predict a new unseen keyword/text?

  Here, do I need to train all the time the model to "see" the new keywords. I would expect so, but how?

• How can I print/export the output of my prediction?

Answer 1

I do not see many benefits of using TensorFlow here. TensorFlow (and PyTorch) are great tools if you need to compute gradients of a loss function and use it in an optimizer. Clustering does not give the kind of training that would allow you to train an RNN or a Transformer that would give you a reasonable representation.

In your case, I would try:

• represent the by average word embedding of content words

• represent the text using BERT

And then do use a clustering algorithm of your choice. You can indeed implement k-means or hierarchical clustering in TensorFlow.

Answer 2

You could either do it in TensorFlow or not, IMHO.

1. One way is to use pretrained embeddings, or some pretrained model such as BERT to generate a representation of y

2. You can also do it with a TensorFlow model. For example, you could feed each piece of text (processed as a sequence of tokens) into an Autoencoder, take the compressed representation of your data, and later run some clustering techniques such as k-Means on that. You could either use Conv or RNN layers for the Encoder and the Decoder. TensorFlow model can work either with pretrained and trained-on-scratch embeddings. You con create your own by putting an Embedding() layer at the input of your Neural Network.

3. A very fast and effective alternative is to train a doc2vec model. gensim library offers built-in functions for that.

PS: I think that TensorFlow 2.x makes things much easier compared to the 1.x, that was not at all simple and super verbose. Pretty much any Keras example you can find around can be ported to TF2 effortlessly.