0
$\begingroup$

I'm trying to build a binary classifier where the features are mostly numerical (about 20) and there are a couple of unstructured short text fields as well. What is currently considered the state of the art for combining these types of features?

I've tried building a separate classifier (logistic regression, TFIDF) using the text data alone, and then including that output score of that classifier as an additional when training using the rest of the numerical features (random forest, taking care to train each classifier on different folds of the data to prevent signal leakage). It works alright, but I think a better performance is possible.

A variant is to simply train two separate classifiers, one using text and the other using numerican features and then stacking those two. And finally, another idea is to use neural networks, have two input networks, one CNN/LSTM for the text, and another dense for the numerical features, and then combining them and having a single output.

Are there other approaches I haven't thought of that are worth a try?

$\endgroup$
1
$\begingroup$

Regarding the first question, to the best of my knowledge, there is no SOTA approach. It depends on the task.

For the second one, did you try to combine the numerical values with the TFIDF vectors, and then feed the final vector(s) to the classifier? Probably not. I usually try this way. But, if your vocabulary is too large, try to reduce it by considering top N-words, because combining small numerical vector with a huge TFIDF vector may not allow the numerical values to affect the results.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks. It feels weird to concatenate sparse features sets with dense ones, plus as you noted, there is the possibility of the tfidf features overshadowing the numerical ones. Even with limiting the vocabulary, they would still be an order of magnitude more. $\endgroup$ – Ansari Apr 22 at 17:00
  • $\begingroup$ I don't want to answer you yes or no, but I tried this way and it works well. I tested it in my approach for a research shared task and I was ranked first. Despite that, what you've mentioned is still a scientific explanation. BTW, I do this with classical classifiers (SVM, NB, etc.), in the case of Neural Networks, I concatenate two different branches (one for text and another for the numerical values) at the very last steps of the network. $\endgroup$ – Ghanem Apr 22 at 17:14
  • 1
    $\begingroup$ Thanks! I'll give it a shot. $\endgroup$ – Ansari Apr 22 at 17:19
0
$\begingroup$

To improve your results you should try other embeddings instead of tf-idf. Some examples are word2vec, FastText, Elmo, Flair or transformer-based embeddings like BERT. In the past I got great results with a single neural network which uses embeddings and numerical features as input features. But, as mentioned before, it depends on your concrete problem.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Did you concatenate the embeddings with the numerical features or have separate branches in your network and then combine their outputs at a more reasonable dimension? $\endgroup$ – Ansari Apr 22 at 20:25
  • $\begingroup$ Yes, I concatenated numerical features and embeddings. I got best results with flair embeddings. $\endgroup$ – NiklasF Apr 23 at 19:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.