1
$\begingroup$

I'm trying to build a binary classifier where the features are mostly numerical (about 20) and there are a couple of unstructured short text fields as well. What is currently considered the state of the art for combining these types of features?

I've tried building a separate classifier (logistic regression, TFIDF) using the text data alone, and then including that output score of that classifier as an additional when training using the rest of the numerical features (random forest, taking care to train each classifier on different folds of the data to prevent signal leakage). It works alright, but I think a better performance is possible.

A variant is to simply train two separate classifiers, one using text and the other using numerican features and then stacking those two. And finally, another idea is to use neural networks, have two input networks, one CNN/LSTM for the text, and another dense for the numerical features, and then combining them and having a single output.

Are there other approaches I haven't thought of that are worth a try?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

Regarding the first question, to the best of my knowledge, there is no SOTA approach. It depends on the task.

For the second one, did you try to combine the numerical values with the TFIDF vectors, and then feed the final vector(s) to the classifier? Probably not. I usually try this way. But, if your vocabulary is too large, try to reduce it by considering top N-words, because combining small numerical vector with a huge TFIDF vector may not allow the numerical values to affect the results.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks. It feels weird to concatenate sparse features sets with dense ones, plus as you noted, there is the possibility of the tfidf features overshadowing the numerical ones. Even with limiting the vocabulary, they would still be an order of magnitude more. $\endgroup$
    – Ansari
    Apr 22, 2020 at 17:00
  • $\begingroup$ I don't want to answer you yes or no, but I tried this way and it works well. I tested it in my approach for a research shared task and I was ranked first. Despite that, what you've mentioned is still a scientific explanation. BTW, I do this with classical classifiers (SVM, NB, etc.), in the case of Neural Networks, I concatenate two different branches (one for text and another for the numerical values) at the very last steps of the network. $\endgroup$
    – Minions
    Apr 22, 2020 at 17:14
  • 1
    $\begingroup$ Thanks! I'll give it a shot. $\endgroup$
    – Ansari
    Apr 22, 2020 at 17:19
0
$\begingroup$

To improve your results you should try other embeddings instead of tf-idf. Some examples are word2vec, FastText, Elmo, Flair or transformer-based embeddings like BERT. In the past I got great results with a single neural network which uses embeddings and numerical features as input features. But, as mentioned before, it depends on your concrete problem.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Did you concatenate the embeddings with the numerical features or have separate branches in your network and then combine their outputs at a more reasonable dimension? $\endgroup$
    – Ansari
    Apr 22, 2020 at 20:25
  • 1
    $\begingroup$ Yes, I concatenated numerical features and embeddings. I got best results with flair embeddings. $\endgroup$
    – NiklasF
    Apr 23, 2020 at 19:08
  • $\begingroup$ @NiklasF Could you share a code snippet of combining text features with numerical ones, and how did you feed those into a nn? Have you used a pre-trained model, or did you create one from scratch (so you decide exactly the input of your nn). $\endgroup$
    – George Petropoulos
    Apr 23, 2021 at 9:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.