0
$\begingroup$

I created a model in Keras that predicts 4 sentiments/emotions based on text input. Size of my data:

label_1 : 100.000
label_2 : 100.000
label_3 : 100.000
label_4 : 50.000

Validation data: 45.000

I have set the class weights to: class_weight = {'label_1':1, 'label_2':1, 'label_3':1, 'label_1':2}

I have used scikit-learn for vectorising with CountVectorizer. For preprocessing, I have converted all to lowercase, removed emails, urls/links and removed stopwords.

This is my model:

model = Sequential()
model.add(Dense(50, input_dim = features.shape[1], activation = 'relu')) # input layer requires input_dim param
model.add(Dense(100, activation = 'relu'))
model.add(Dense(50, activation = 'relu'))
model.add(Dropout(0.5))
model.add(Dense(4, activation='softmax'))


opt = SGD(lr = 0.001, momentum = 0.01)
model.compile(loss="categorical_crossentropy", optimizer = opt, metrics=['accuracy'])

es = EarlyStopping(monitor='val_loss', min_delta = 0.03, patience = 120, verbose=1, mode='auto')
history = model.fit(features, results, validation_split = 0.25, shuffle = True, class_weight = class_weight, epochs = 600, batch_size=512, verbose=2, callbacks=[es])

score = model.evaluate(x_test, y_test, batch_size=512)
print()
print(history.history.keys())
print()
print(score)
print('Test loss:', score[0],  'Test accuracy:', score[1])

Test loss: 0.215 Test accuracy: 0.900

These are the accuracy and loss graphs:

enter image description here

enter image description here

This is my prediction:

validation_features = transformerVectoriser.transform(validation_features)
prediction = model.predict_classes(validation_features , batch_size=512) # making prediction

These are confusion_matrix and accuracy_report:

[[11678   256  1181    23]
 [  477 12023   432    13]
 [ 1538   322 10947    18]
 [   16     9    17  6050]]


precision    recall  f1-score   support

0       0.85      0.89      0.87     13138
1       0.95      0.93      0.94     12945
2       0.87      0.85      0.86     12825
3       0.99      0.99      0.99      6092

accuracy                    0.90     45000
macro avg. 0.92   0.92      0.92     45000
weighted avg  0.91  0.90    0.90     45000

My question is, how can I reduce loss? I have tried to change number of layers, nodes and dropout layers, to change optimisers, learning rate and momentum, to change number of epochs and batch size, and to change max number of words (I have tried with 4.000, 5.000, 6.000, 8.000, 10.000, 12.000)

It does not matter what I change, my accuracy is always around 87%-90% (and I think thats good), but my loss is always around 0.21-0.24 (and I do not like that). On some data science related blogs I have read that loss should be less than 0.1. Is that true? What is acceptable loss value for multiclass classification?

Do you have any advices?

$\endgroup$
0
$\begingroup$

There is no universal loss value result that you can or should achieve. Beside your model it will heavily depend on the data that you have.

What is good or bad result is subjective and depends on your use-case.

It's a very broad question how to improve the result of a neural network and you should be more specific but some ideas that can work generally:

  1. Preprocess your data differently (i.e. use TF-IDF or use text as a sequence)
  2. Collect more data (neural networks usually prefer that, also your test set seems small)
  3. Use different optimizer (i.e. Adam)
  4. Use different architecture (i.e. some RNN)
  5. Use different algorithm (i.e. XGBoost)
  6. Use BatchNormalization
  7. Use some callback that modifies your learning rate
  8. Tune your parameters
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.