I am going to train machine learning models that assign certain tags to a paragraph describing an activity. In my database, for a give paragraph of description (X), there are several corresponding tags related to it (Y). I hope to improve the classification accuracy.
I built several machine learning models through Scikit-learn-learn (such as SVC, DecisionTreeClassifier, KNeighborsClassifier , RadiusNeighborsClassifier, ExtraTreesClassifier, RandomForestClassifier, MLPClassifier, RidgeClassifierCV) and neural network models through Keras. The best accuracy (harsh metric) that I can get is 47% using OneVsRestClassifier(SGDClassifier).
print(X) 0 Contribution to METU HS Ankara Lab Protocols ... 1 Attend the MakerFaire in Hannover to demonstr... 2 Organize a "Biotech Day" and present the proj... 3 Contact and connect with Community Labs in Eu... 4 Invite "Technik Garage," a German Community L... 5 Present the project to the biotechnology comp... 6 Visit one of Europe's largest detergent plant... ... print(y2) 0 [Community Event] 1 [Project Presentation, Community Event] 2 [Project Presentation, Teaching Activity] 3 [Conference/Panel Discussion, Consult Experts] 4 [Conference/Panel Discussion, Consult Experts] 5 [Conference/Panel Discussion, Project Presenta... 6 [Consult Experts] ...
from sklearn.preprocessing import MultiLabelBinarizer mlb = MultiLabelBinarizer() mlb_y2 = mlb.fit_transform(y2) from sklearn.model_selection import train_test_split X_train, X_test, y_train, y_test = train_test_split(X, mlb_y2, test_size=0.2, random_state=52) Scikit-learn: from sklearn.feature_extraction.text import CountVectorizer from sklearn.feature_extraction.text import TfidfTransformer from sklearn.pipeline import Pipeline from sklearn.multiclass import OneVsRestClassifier from sklearn.linear_model import SGDClassifier pipe = Pipeline(steps=[('vect', CountVectorizer()), ('tfidf', TfidfTransformer()),('classifier', OneVsRestClassifier(SGDClassifier(loss = 'hinge', alpha=0.00026, penalty='elasticnet', max_iter=2000,tol=0.0008, learning_rate = 'adaptive', eta0 = 0.12)))]) pipe.fit(X_train, y_train) print("test model score: %.3f" % pipe.score(X_test, y_test)) print("train model score: %.3f" % pipe.score(X_train, y_train)) test model score: 0.478 train model score: 0.801 (overfitting exist! I adjusted the penalty & alpha term, but it doesn't improve much. I don't know whether there is any other way to do the regulation.) Keras: from keras.preprocessing.text import Tokenizer from keras.preprocessing.sequence import pad_sequences tokenizer = Tokenizer(num_words=300, lower=True) tokenizer.fit_on_texts(X) sequences = tokenizer.texts_to_sequences(X) vocab_size = len(tokenizer.word_index) + 1 x = pad_sequences(sequences, padding='post', maxlen=80) from keras.models import Sequential from keras.layers import Dense, Activation, Embedding, Flatten, GlobalMaxPool1D, Dropout, Conv1D, LSTM, SpatialDropout1D from keras.callbacks import ReduceLROnPlateau, EarlyStopping, ModelCheckpoint from keras.losses import binary_crossentropy from keras.optimizers import Adam import sklearn filter_length = 1000 model = Sequential() model.add(Embedding(input_dim=vocab_size, output_dim= 70, input_length=80)) model.add(Dropout(0.1)) model.add(Conv1D(filter_length, 3, padding='valid', activation='relu', strides=1)) model.add(GlobalMaxPool1D()) #model.add(SpatialDropout1D(0.1)) #model.add(LSTM(100, dropout=0.1, recurrent_dropout=0.1)) model.add(Dense(len(mlb.classes_))) model.add(Activation('sigmoid')) model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['categorical_accuracy']) callbacks = [ReduceLROnPlateau(),EarlyStopping(patience=4), ModelCheckpoint(filepath='model-conv1d.h5', save_best_only=True)] history = model.fit(X_train, y_train,epochs=80,batch_size=500, validation_split=0.1,verbose=2,callbacks=callbacks) from keras import models cnn_model = models.load_model('model-conv1d.h5') from sklearn.metrics import accuracy_score y_pred = cnn_model.predict(X_test) accuracy_score(y_test,y_pred.round()) Out: 0.4405555555555556 (I think the neural network model has more room for improvement. But I'm not sure how to achieve that.）
I hope the accuracy reach at least 60%. Could you guys give me some advice on improving my code for Scikit-learn and Keras model?
More specifically, 1. Is there a way to improve the OneVsRestClassifier(SGDClassifier)? 2. Is there a way to improve my convolutional neural network? Or use some form of recurrent neural network? (I tried simple RNN, but it doesn't work well)
PS: In my way of calculating accuracy, for a description(X) if the model outputs [0, 0, 0, 1, 0, 1](y_pred) and the correct output is [0, 0, 0, 1, 0, 0](y_test), my accuracy would be 0 instead of 5/6.
This question is quite long. Thank you guys so much!