What are popular deep learning models for tabular data of texts?

Question

I have a tabular dataset where every column is of type "text" (i.e. not categorical variable and essentially it can be anything).

Let's suppose that the task is classification

What are some popular methods to classify such data? I have this idea to transform each row to a document where values are separated with a special character (much like CSV). Then I could train a language model like BERT. I guess the special character can be a word by itself so it could signal the model the notion of columns.

Is that approach popular and worth the shot? What are other approaches are known to be successful for my task?

Answer 1

One option could be to use a BERT encoder to tokenize and encode the words and then use a Convolutional Neural Network for the classification task. If you need a tutorial on how to do it, check this article.

Also you can fine-tune a Transformer model, like BERT or Google's T5, to do the classification. But they can take long to train, so try CNN first and if you are not happy with the results think about transformers.