How do you handle the free-text fields in tabular data in ML/DL?

Question

While we see a number of cases where the input data is only a single text fields (for the X variable) in NLP tasks, e.g. a tweet with a sentiment label being the only numerical field.

But how do you handle the free-text fields in tabular data in ML/DL? The text field(s) is/are among all the numeric fields in a table! I think this is tricky to handle. It can be comment fields or some log data in some fields along with many other numeric fields. List as many approaches as possible. Any idea?

For easy discussion, the 'free-text' defined here refers to a bunch of text where each row of data in the dataset can has variable length in the text.

And the goal of this question is to find ways to transform such text field(s) such that they can be included into ML/DL models.

Answer 1

There are various text representation techniques, ranging from bag-of-words methods like TFIDF to embeddings. These techniques are used to build a fixed-length representation of any input text.

• If the text is the only input in an instance, as in text classification for example, then this representation is directly usable as the vector of features values.
• If there are other features or several distinct texts which must be represented separately, then the vector and the other features can be concatenated.

In general one should be careful to prevent the resulting number of features to become too high. For example, it's very common to ignore the least frequent words in the case of a bag-of-words representation.