0
$\begingroup$

I am going to train machine learning models that assign certain tags to a paragraph describing an activity. In my database, for a give paragraph of description (X), there are several corresponding tags related to it (Y). I hope to improve the classification accuracy.

I built several machine learning models through Scikit-learn-learn (such as SVC, DecisionTreeClassifier, KNeighborsClassifier , RadiusNeighborsClassifier, ExtraTreesClassifier, RandomForestClassifier, MLPClassifier, RidgeClassifierCV) and neural network models through Keras. The best accuracy (harsh metric) that I can get is 47% using OneVsRestClassifier(SGDClassifier).

print(X)
0        Contribution to METU HS Ankara Lab Protocols ...
1        Attend the MakerFaire in Hannover to demonstr...
2        Organize a "Biotech Day" and present the proj...
3        Contact and connect with Community Labs in Eu...
4        Invite "Technik Garage," a German Community L...
5        Present the project to the biotechnology comp...
6        Visit one of Europe's largest detergent plant...
...

print(y2)
0                                       [Community Event]
1                 [Project Presentation, Community Event]
2               [Project Presentation, Teaching Activity]
3          [Conference/Panel Discussion, Consult Experts]
4          [Conference/Panel Discussion, Consult Experts]
5       [Conference/Panel Discussion, Project Presenta...
6                                       [Consult Experts]
...

...

from sklearn.preprocessing import MultiLabelBinarizer
mlb = MultiLabelBinarizer()
mlb_y2 = mlb.fit_transform(y2)
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, mlb_y2, test_size=0.2, random_state=52)

Scikit-learn:
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.pipeline import Pipeline
from sklearn.multiclass import OneVsRestClassifier
from sklearn.linear_model import SGDClassifier

pipe = Pipeline(steps=[('vect', CountVectorizer()), ('tfidf', TfidfTransformer()),('classifier', OneVsRestClassifier(SGDClassifier(loss = 'hinge', alpha=0.00026, penalty='elasticnet', max_iter=2000,tol=0.0008, learning_rate = 'adaptive', eta0 = 0.12)))])
pipe.fit(X_train, y_train) 
print("test model score: %.3f" % pipe.score(X_test, y_test))
print("train model score: %.3f" % pipe.score(X_train, y_train))
test model score: 0.478
train model score: 0.801 (overfitting exist! I adjusted the penalty & alpha term, but it doesn't improve much. I don't know whether there is any other way to do the regulation.)

Keras:
from keras.preprocessing.text import Tokenizer
from keras.preprocessing.sequence import pad_sequences

tokenizer = Tokenizer(num_words=300, lower=True)
tokenizer.fit_on_texts(X)
sequences = tokenizer.texts_to_sequences(X)
vocab_size = len(tokenizer.word_index) + 1
x = pad_sequences(sequences, padding='post', maxlen=80)

from keras.models import Sequential
from keras.layers import Dense, Activation, Embedding, Flatten, GlobalMaxPool1D, Dropout, Conv1D, LSTM, SpatialDropout1D
from keras.callbacks import ReduceLROnPlateau, EarlyStopping, ModelCheckpoint
from keras.losses import binary_crossentropy
from keras.optimizers import Adam
import sklearn

filter_length = 1000

model = Sequential()
model.add(Embedding(input_dim=vocab_size, output_dim= 70, input_length=80))
model.add(Dropout(0.1))
model.add(Conv1D(filter_length, 3, padding='valid', activation='relu', strides=1))
model.add(GlobalMaxPool1D())
#model.add(SpatialDropout1D(0.1))
#model.add(LSTM(100, dropout=0.1, recurrent_dropout=0.1))
model.add(Dense(len(mlb.classes_)))
model.add(Activation('sigmoid'))

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['categorical_accuracy'])

callbacks = [ReduceLROnPlateau(),EarlyStopping(patience=4),
    ModelCheckpoint(filepath='model-conv1d.h5', save_best_only=True)]

history = model.fit(X_train, y_train,epochs=80,batch_size=500,
validation_split=0.1,verbose=2,callbacks=callbacks)

from keras import models
cnn_model = models.load_model('model-conv1d.h5')
from sklearn.metrics import accuracy_score
y_pred = cnn_model.predict(X_test)
accuracy_score(y_test,y_pred.round())

Out: 0.4405555555555556 (I think the neural network model has more room for improvement. But I'm not sure how to achieve that.)

I hope the accuracy reach at least 60%. Could you guys give me some advice on improving my code for Scikit-learn and Keras model?

More specifically, 1. Is there a way to improve the OneVsRestClassifier(SGDClassifier)? 2. Is there a way to improve my convolutional neural network? Or use some form of recurrent neural network? (I tried simple RNN, but it doesn't work well)

PS: In my way of calculating accuracy, for a description(X) if the model outputs [0, 0, 0, 1, 0, 1](y_pred) and the correct output is [0, 0, 0, 1, 0, 0](y_test), my accuracy would be 0 instead of 5/6.

This question is quite long. Thank you guys so much!

$\endgroup$
  • $\begingroup$ Hi. Interesting problem. Could you be more specific in what area you think you need improvements? Right now it sounds a bit like "take a look and make it better" which is too broad for the types of questions in this forum. Thank you. $\endgroup$ – oW_ Jul 8 '19 at 19:56
  • $\begingroup$ @oW_ Yeah! I just edit my question. Thanks for your advice! $\endgroup$ – Sebastian Jul 8 '19 at 22:29
1
$\begingroup$

Keep in mind that Accuracy is not the perfect evaluation metric in Multi-Label Learning. The reason is simple, as you also mentioned in your question. Predicting 5 from 6 correctly is far better than predicting 0 from 6. There are different metrics to use in MLL.

  1. MicroPrecision - MicroRecall

You can try Micro-Precision and/ord Micro-Recall. What you do in that case, is to calculate the Precision (or Recall) for each class and then get the average of those values.

  1. Hamming Loss

Another metric is the Hamming Loss which is the fraction of labels that are incorrectly predicted

If you still need to improve your model, then you can try another approach on how you design it.

For example, you can use a Classifier Chain

For each label you have and you need to predict, you create one Binary Classification Model. For example, a Random Forest. For the first label, you use all the features and you try to predict just the first label. For the second one, you use your features + the prediction of the first label. For the third, the features + the predictions of both first and second label.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Got it, thank you! Also, do you think there is any way to improve my OneVsRestClassifier() model? $\endgroup$ – Sebastian Jul 10 '19 at 20:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.