We want to solve a regression problem of the form "given two objects $x$ and $y$, predict their score (think about it as a similarity) $w(x,y)$". We have 2 types of features:

  • For each object, we have about 1000 numerical features, mainly of the following types: 1) "Historical score info", e.g. historical means $w(x,\cdot)$ up to the point we use the feature; 2) 0/1 features meaning whether object $x$ has a particular attribute, etc.
  • For each object, we have a text which describes the object (description is not reliable, but still useful).

Clearly, when predicting a score for a pair $(x,y)$, we can use features for both $x$ and $y$.

We are currently using the following setup (I omit validation/testing):

  • For texts, we compute their BERT embeddings and then produce a feature based on the similarity between the embedding vectors (e.g. cosine similarity between them).
  • We split the dataset into fine-tuning and training datasets. The fine-tuning dataset may be empty meaning no fine-tuning.
  • Using the fine-tuning dataset, we fine-tune BERT embeddings.
  • Using the training dataset, we train decision trees to predict the scores.

We compare the following approaches:

  • Without BERT features.
  • Using BERT features, but without fine-tuning. There is some reasonable improvement in prediction accuracy.
  • Using BERT features, with fine-tuning. The improvement is very small (but the prediction using only BERT features improved, of course).

Question: Is there something simple I'm missing in this approach? E.g. maybe there are better ways to use texts? Other ways to use embeddings? Better approaches compared with decision trees?

I tried to do multiple things, without any success. The approaches which I expected to provide improvements are the following:

  • Fine-tune embeddings to predict difference between $w(x,y)$ and mean $w(x, \cdot)$. The motivation is that we already have a feature "mean $w(x,\cdot)$", which is a baseline for an object $x$, and we are interested in the deviation from this mean.

  • Use NN instead of decision trees. Namely, I use few dense layers to turn embedding vectors into features, like this:

          nn.Linear(768 * 2, 1000),
          nn.Linear(1000, 500),
          nn.Linear(500, 100),
          nn.Linear(100, 10),

    After that, I combine these new $10$ features with $2000$ features I already have, and use similar architecture on top of them:

          nn.Linear(10 + n_features, 1000),
          nn.Linear(1000, 500),
          nn.Linear(500, 100),
          nn.Linear(100, 1),

But as a result, my prediction is much worse compared with decision trees. Are there better architectures suited for my case?


I found two articles that typically deals with real world data which consists of both numerical data and text data. Hope these approaches help you guys out

First Article

This article shows different solutions to combine natural language processing and traditional features in one single model in keras (end to end learning)

Solution: To utilize end-to-end learning neural networks, instead of manually stacking models, by combining these different feature spaces inside the neural network.

  1. Special Tokens/ embeddings : Here the metadata is typically special embeddings. Transforming data into categorial features, because these embeddings can exist or not exist. Made possible by increasing the vocabulary size by the number of additional features and treating them as additional words. This approach has some drawbacks as the embedding space mixes up text and meta data.

enter image description here 2. Multiple input models: Here a bidirectional LSTM model’s output is combined with metadata. Two input layers treated in separate “data paths”(nlp_input and meta_input). Typically NLP data goes through the embedding transformation and the LSTM layer. The meta data is just normalised and concatenated with the LSTM output directly (nlp_out). This combined vector is the full representation of input and can be finally classified in a fully-connected layer.

enter image description here

Below implementation for 100 words and 10 additional features

nlp_input = Input(shape=(seq_length,)) 
meta_input = Input(shape=(10,))
emb = Embedding(output_dim=embedding_size, input_dim=100, input_length=seq_length)(nlp_input) 
nlp_out = Bidirectional(LSTM(128))(emb) 
concat = concatenate([nlp_out, meta_input]) 
classifier = Dense(32, activation='relu')(concat) 
output = Dense(1, activation='sigmoid')(classifier) 
model = Model(inputs=[nlp_input , meta_input], outputs=[output])

Second Article

This article focuses on practical NLP model from simple classifiers to deep neural networks

  1. scikit-learn (e.g. for Tfidf) By using a function transformer to separate numeric and text data, Tfidf Vectorizer on texts and combine using sklearn.pipeline.FeatureUnion in Pipeline.

enter image description here

import numpy as np

from sklearn.feature_extraction.text import TfidfVectorizer

from sklearn.pipeline import Pipeline, FeatureUnion

from sklearn.ensemble import RandomForestClassifier

from sklearn.preprocessing import FunctionTransformer

from sklearn.model_selection import GridSearchCV, StratifiedKFold

# Create Function Transformer to use Feature Union

def get_numeric_data(x):

    return [record[:-2].astype(float) for record in x]

def get_text_data(x):

    return [record[-1] for record in x]

transfomer_numeric = FunctionTransformer(get_numeric_data)

transformer_text = FunctionTransformer(get_text_data)

# Create a pipeline to concatenate Tfidf Vector and Numeric data

# Use RandomForestClassifier as an example

pipeline = Pipeline([

    ('features', FeatureUnion([

            ('numeric_features', Pipeline([

                ('selector', transfomer_numeric)


             ('text_features', Pipeline([

                ('selector', transformer_text),

                ('vec', TfidfVectorizer(analyzer='word'))



    ('clf', RandomForestClassifier())


# Grid Search Parameters for RandomForest

param_grid = {'clf__n_estimators': np.linspace(1, 100, 10, dtype=int),

              'clf__min_samples_split': [3, 10],

              'clf__min_samples_leaf': [3],

              'clf__max_features': [7],

              'clf__max_depth': [None],

              'clf__criterion': ['gini'],

              'clf__bootstrap': [False]}

# Training config

kfold = StratifiedKFold(n_splits=7)

scoring = {'Accuracy': 'accuracy', 'F1': 'f1_macro'}

refit = 'F1'

# Perform GridSearch

rf_model = GridSearchCV(pipeline, param_grid=param_grid, cv=kfold, scoring=scoring, 

                         refit=refit, n_jobs=-1, return_train_score=True, verbose=1)

rf_model.fit(X_train, Y_train)

rf_best = rf_model.best_estimator_
  1. Pytorch (e.g. for LSTM, BERT)

enter image description here

class LSTMTextClassifier(nn.Module):

  def __init__(self, vocab_size, embed_size, lstm_size, dense_size, numeric_feature_size, output_size, lstm_layers=1, dropout=0.1):


        self.vocab_size = vocab_size

        self.embed_size = embed_size

        self.lstm_size = lstm_size

        self.output_size = output_size

        self.lstm_layers = lstm_layers

        self.dropout = dropout

        self.embedding = nn.Embedding(vocab_size, embed_size)

        self.lstm = nn.LSTM(embed_size, lstm_size, lstm_layers, dropout=dropout, batch_first=False)

        self.dropout = nn.Dropout(0.2)

        self.fc1 = nn.Linear(lstm_size, dense_size)

        self.fc2 = nn.Linear(dense_size + numeric_feature_size, output_size)

        self.softmax = nn.LogSoftmax(dim=1)

    def init_hidden(self, batch_size):

        weight = next(self.parameters()).data

        hidden = (weight.new(self.lstm_layers, batch_size, self.lstm_size).zero_(),

                  weight.new(self.lstm_layers, batch_size, self.lstm_size).zero_())


        return hidden

    def forward(self, nn_input_text, nn_input_meta, hidden_state):

        batch_size = nn_input_text.size(0)

        nn_input_text = nn_input_text.long()

        embeds = self.embedding(nn_input_text)

        lstm_out, hidden_state = self.lstm(embeds, hidden_state)

        lstm_out = lstm_out[-1,:,:]

        lstm_out = self.dropout(lstm_out)

        dense_out = self.fc1(lstm_out)

        concat_layer = torch.cat((dense_out, nn_input_meta.float()), 1)

        out = self.fc2(concat_layer)

        logps = self.softmax(out)

        return logps, hidden_state


class BertTextClassifier(nn.Module):

    def __init__(self, hidden_size, dense_size, numeric_feature_size, output_size, dropout=0.1):


        self.output_size = output_size

        self.dropout = dropout


        # Use pre-trained BERT model

        self.bert = BertModel.from_pretrained('bert-base-uncased', output_hidden_states=True, output_attentions=True)

        for param in self.bert.parameters():

            param.requires_grad = True

        self.weights = nn.Parameter(torch.rand(13, 1))

        self.dropout = nn.Dropout(dropout)

        self.fc1 = nn.Linear(hidden_size, dense_size)

        self.fc2 = nn.Linear(dense_size + numeric_feature_size, output_size)

        self.softmax = nn.LogSoftmax(dim=1)

    def forward(self, input_ids, nn_input_meta):        

        all_hidden_states, all_attentions = self.bert(input_ids)[-2:]

        batch_size = input_ids.shape[0]

        ht_cls = torch.cat(all_hidden_states)[:, :1, :].view(13, batch_size, 1, 768)

        atten = torch.sum(ht_cls * self.weights.view(13, 1, 1, 1), dim=[1, 3])

        atten = F.softmax(atten.view(-1), dim=0)

        feature = torch.sum(ht_cls * atten.view(13, 1, 1, 1), dim=[0, 2])

        dense_out = self.fc1(self.dropout(feature))

        concat_layer = torch.cat((dense_out, nn_input_meta.float()), 1)

        out = self.fc2(concat_layer)

        logps = self.softmax(out)

        return logps
  • $\begingroup$ Thanks for the reply! While the second approach looks reasonable (although I'm not sure about using tf-idf features in decision trees), the first one is somewhat weird: it looks pretty strange to me to use LSTM on top of BERT embeddings (as I know, BERT actually replaced LSTM). Building a NN layer on top of pooled embeddings looks more meaningful. $\endgroup$
    – Dmitry
    Jul 27 at 15:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.