practical improvements worth trying over plain LSTM in text classification?

Question

I have a dataset of about 1 million tweets corresponding to about 30,000 user accounts, labelled with binary data (classifying the tweet as written by a bot).

With that amount of data, I could use a LSTM-based NLP approach to take advantage of the sequence data in the text.

I would like to get the best accuracy possible, but don't know what models are available that could be combined with or build on LSTM-based approaches.

I also wonder, as one part of the data is sequential and the other stationary (account metadata), whether some kind of hybrid classifier could do better than using just an LSTM.

Are there any hybrid models or any other proven models that would be worth investigation further, to implement for my project?

Answer 1

I think the main thing you would need to worry about is how to extract meaningful account metadata. But this is your dataset and you probably know the details about it well enough now. There are some details about bot hunting account features / metadata that may be useful from this blackhat paper, looking for things like "Number of tweets relative to account age" and "Average hours tweeted per day" which would presumably features you would have to engineer yourself out of your tweet/account metadata.

When it comes to integrating them into your model, you want to do something like concatenating features into single layers in the model. You didn't specify a language here, and the above link is in R, but the basic code snippet is pretty readable about how to concatenate embedded categorical vectors with other features.

embedding_size_weekday=3
embedding_size_bridge=2

## 1
inp1 <- layer_input(shape = c(1), name = 'inp_weekday')
inp2 <- layer_input(shape = c(1), name = 'inp_bridge')
inp3 <- layer_input(shape = c(2), name = 'inp_otherVars')

## 2
embedding_out1 <- inp1 %>% layer_embedding(input_dim = 7+1, output_dim = embedding_size_weekday, input_length = 1, name="embedding_weekday") %>%  layer_flatten()

embedding_out2 <- inp2 %>% layer_embedding(input_dim = 4+1, output_dim = embedding_size_bridge, input_length = 1, name="embedding_bridge") %>%  layer_flatten()

## 3
combined_model <- layer_concatenate(c(embedding_out1, embedding_out2, inp3)) %>%
  layer_dense(units=32, activation = "relu") %>%
  layer_dropout(0.3) %>%
  layer_dense(units=10, activation = "relu") %>%
  layer_dropout(0.15) %>%
  layer_dense(units=1)

## 4
model <- keras::keras_model(inputs = c(inp1, inp2, inp3), outputs = combined_model)

model %>% compile(loss = "mean_squared_error", optimizer = "sgd", metric="accuracy") 

summary(model)

In terms of what the actual keras code for layering multiple inputs should look like, I think the docs have a nice example of your different options: (Probably best to experiment here to find out what works for you)

import keras

input1 = keras.layers.Input(shape=(16,))
x1 = keras.layers.Dense(8, activation='relu')(input1)
input2 = keras.layers.Input(shape=(32,))
x2 = keras.layers.Dense(8, activation='relu')(input2)
# equivalent to added = keras.layers.add([x1, x2])
added = keras.layers.Add()([x1, x2])

out = keras.layers.Dense(4)(added)
model = keras.models.Model(inputs=[input1, input2], outputs=out) # two inputs

Answer 2

A properly written LSTM network is quite powerful for NLP - why do you think that "combining" this with something else is the answer???

The project you're describing should be pretty easy to implement. The only barrier you have here is how much labeled data you can collect. Personally, I would focus on getting hundreds of thousands of examples, in a balanced dataset, and then feed that (with NLP approaches) into an LSTM network. You should be able to generate very good accuracy rates in a reasonable amount of time.