2
$\begingroup$

I'm working on a reviews classification model with only two categories 0 (negative) and 1 (positive). I'm using pre-trained word2vec from google with LSTM. The problem is I get an accuracy of around 50% where it should be around 83% according to this paper. I tried many different hyperparameters combination and still gets a horrible accuracy. I also tried to change the data preprocessing techniques and tried stemming but it hasn't resolved the problem

here's my code

X, y = read_data()
X = np.array(clean_text(X)) #apply data preprocessing  
tokenizer = Tokenizer()
tokenizer.fit_on_texts(X)

#converts text to sequence and add padding zeros
sequence = tokenizer.texts_to_sequences(X)
X_data = pad_sequences(sequence, maxlen = length, padding = 'post')

X_train, X_val, y_train, y_val = train_test_split(X_data, y, test_size = 0.2)

#Load the word2vec model
word2vec = KeyedVectors.load_word2vec_format(EMBEDDING_FILE, binary=True)

word_index = tokenizer.word_index
nb_words = min(MAX_NB_WORDS, len(word_index))+1

embedding_matrix = np.zeros((nb_words, EMBEDDING_DIM))
null_words = []
for word, i in word_index.items():
    if word in word2vec.wv.vocab:
        embedding_matrix[i] = word2vec.word_vec(word)
    else:
        null_words.append(word)

embedding_layer = Embedding(embedding_matrix.shape[0], # or len(word_index) + 1
                            embedding_matrix.shape[1], # or EMBEDDING_DIM,
                            weights=[embedding_matrix],
                            input_length=701,
                            trainable=False)

model = Sequential()
model.add(embedding_layer)
model.add(LSTM(100))
model.add(Dropout(0.4))
model.add(Dense(2, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

I also tried other optimizers like AdaMax and MSLE loss function. I'm just confused if the problem isn't with the model and preprocessing where could it be? Thanks

$\endgroup$
1
$\begingroup$

Its huge discrepancy, I suspect a lie.

While +33% can be achieved, you said that you tried very different architectures and you did not get even close. Dont expect that one tweak, one layer, one xyz can give you all of a suddenly such a huge increase. If you did not get closer using suggestions from the paper there is also a possibility (not saying they did, but it has been done before in ML papers- lying about accuracy achieved without providing code) that they lied.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.