So, I have found that there are many ways to classify words with sklearn's SVM algorithm. But I want to classify questions by taxonomy, as shown in the following dataset:
The goal of this task is to predict the taxonomy given the pdf file / string (question). The questions are the following:
- How can I modify the below code to train a question-based classification model?
- How can I train a question classification model with SVM?
For this task, I have used the following Python libraries
import pandas as pd import numpy as np from nltk.tokenize import word_tokenize, sent_tokenize from sklearn.feature_extraction.text import CountVectorizer from sklearn.preprocessing import LabelEncoder from collections import defaultdict from nltk.corpus import wordnet as wn from sklearn.feature_extraction.text import TfidfVectorizer from sklearn import model_selection, naive_bayes, svm from sklearn.metrics import accuracy_score
Here, I have split the dataset into train and test sets.
Train_X, Test_X, Train_Y, Test_Y = model_selection.train_test_split(Corpus['question'],Corpus['taxonomy'],test_size=0.3) Encoder = LabelEncoder() Train_Y = Encoder.fit_transform(Train_Y) Test_Y = Encoder.fit_transform(Test_Y)
I have used TF IDF transformer (TfidfTransformer) in sklearn library like this
Tfidf_vect = TfidfVectorizer(max_features=5000) Tfidf_vect.fit(Corpus['question']) Train_X_Tfidf = Tfidf_vect.transform(Train_X) Test_X_Tfidf = Tfidf_vect.transform(Test_X)
But this will break down questions into words for each question. The following code was used to make a word classifier to predict the taxonomy from words
SVM = svm.SVC(C=1.0, kernel='linear', degree=3, gamma='auto') SVM.fit(Train_X_Tfidf,Train_Y) # predict the labels on validation dataset predictions_SVM = SVM.predict(Test_X_Tfidf)
Any help would be highly appreciated!