# practical improvements worth trying over plain LSTM in text classification?

I have a dataset of about 1 million tweets corresponding to about 30,000 user accounts, labelled with binary data (classifying the tweet as written by a bot).

With that amount of data, I could use a LSTM-based NLP approach to take advantage of the sequence data in the text.

I would like to get the best accuracy possible, but don't know what models are available that could be combined with or build on LSTM-based approaches.

I also wonder, as one part of the data is sequential and the other stationary (account metadata), whether some kind of hybrid classifier could do better than using just an LSTM.

Are there any hybrid models or any other proven models that would be worth investigation further, to implement for my project?

## 2 Answers

I think the main thing you would need to worry about is how to extract meaningful account metadata. But this is your dataset and you probably know the details about it well enough now. There are some details about bot hunting account features / metadata that may be useful from this blackhat paper, looking for things like "Number of tweets relative to account age" and "Average hours tweeted per day" which would presumably features you would have to engineer yourself out of your tweet/account metadata.

When it comes to integrating them into your model, you want to do something like concatenating features into single layers in the model. You didn't specify a language here, and the above link is in R, but the basic code snippet is pretty readable about how to concatenate embedded categorical vectors with other features.

embedding_size_weekday=3
embedding_size_bridge=2

## 1
inp1 <- layer_input(shape = c(1), name = 'inp_weekday')
inp2 <- layer_input(shape = c(1), name = 'inp_bridge')
inp3 <- layer_input(shape = c(2), name = 'inp_otherVars')

## 2
embedding_out1 <- inp1 %>% layer_embedding(input_dim = 7+1, output_dim = embedding_size_weekday, input_length = 1, name="embedding_weekday") %>%  layer_flatten()

embedding_out2 <- inp2 %>% layer_embedding(input_dim = 4+1, output_dim = embedding_size_bridge, input_length = 1, name="embedding_bridge") %>%  layer_flatten()

## 3
combined_model <- layer_concatenate(c(embedding_out1, embedding_out2, inp3)) %>%
layer_dense(units=32, activation = "relu") %>%
layer_dropout(0.3) %>%
layer_dense(units=10, activation = "relu") %>%
layer_dropout(0.15) %>%
layer_dense(units=1)

## 4
model <- keras::keras_model(inputs = c(inp1, inp2, inp3), outputs = combined_model)

model %>% compile(loss = "mean_squared_error", optimizer = "sgd", metric="accuracy")

summary(model)


In terms of what the actual keras code for layering multiple inputs should look like, I think the docs have a nice example of your different options: (Probably best to experiment here to find out what works for you)

import keras

input1 = keras.layers.Input(shape=(16,))
x1 = keras.layers.Dense(8, activation='relu')(input1)
input2 = keras.layers.Input(shape=(32,))
x2 = keras.layers.Dense(8, activation='relu')(input2)
# equivalent to added = keras.layers.add([x1, x2])
added = keras.layers.Add()([x1, x2])

out = keras.layers.Dense(4)(added)
model = keras.models.Model(inputs=[input1, input2], outputs=out) # two inputs


A properly written LSTM network is quite powerful for NLP - why do you think that "combining" this with something else is the answer???

The project you're describing should be pretty easy to implement. The only barrier you have here is how much labeled data you can collect. Personally, I would focus on getting hundreds of thousands of examples, in a balanced dataset, and then feed that (with NLP approaches) into an LSTM network. You should be able to generate very good accuracy rates in a reasonable amount of time.

• I am looking to improve the accuracy rate achieved using LSTM.I believe, I have sufficient labeled data to build the network – aastha Oct 21 '18 at 21:11
• @aastha well what kind of rates are you getting with LSTM now? – I_Play_With_Data Oct 21 '18 at 21:13
• I am getting about ~96% accuracy [this is for a project, I am looking to implement a hybrid model, or any model that has proven to perform better than LSTM in a certain scenario, on the problem at hand] – aastha Oct 22 '18 at 14:01
• @aastha 96% for NLP against social media is pretty good. LSTM is a widely-accepted approach for this because of these types of accuracy rates. Are you sure that the final 4% (if even possible) is worth the effort you're seeking? – I_Play_With_Data Oct 22 '18 at 14:05
• I would like to try. – aastha Oct 22 '18 at 14:15