# Sklearn SVM question classification

So, I have found that there are many ways to classify words with sklearn's SVM algorithm. But I want to classify questions by taxonomy, as shown in the following dataset:

The goal of this task is to predict the taxonomy given the pdf file / string (question). The questions are the following:

• How can I modify the below code to train a question-based classification model?
• How can I train a question classification model with SVM?

For this task, I have used the following Python libraries

import pandas as pd
import numpy as np
from nltk.tokenize import word_tokenize, sent_tokenize
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.preprocessing import LabelEncoder
from collections import defaultdict
from nltk.corpus import wordnet as wn
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn import model_selection, naive_bayes, svm
from sklearn.metrics import accuracy_score


Here, I have split the dataset into train and test sets.

Train_X, Test_X, Train_Y, Test_Y =
model_selection.train_test_split(Corpus['question'],Corpus['taxonomy'],test_size=0.3)

Encoder = LabelEncoder()
Train_Y = Encoder.fit_transform(Train_Y)
Test_Y = Encoder.fit_transform(Test_Y)


I have used TF IDF transformer (TfidfTransformer) in sklearn library like this

Tfidf_vect = TfidfVectorizer(max_features=5000)
Tfidf_vect.fit(Corpus['question'])
Train_X_Tfidf = Tfidf_vect.transform(Train_X)
Test_X_Tfidf = Tfidf_vect.transform(Test_X)


But this will break down questions into words for each question. The following code was used to make a word classifier to predict the taxonomy from words

SVM = svm.SVC(C=1.0, kernel='linear', degree=3, gamma='auto')
SVM.fit(Train_X_Tfidf,Train_Y)
# predict the labels on validation dataset
predictions_SVM = SVM.predict(Test_X_Tfidf)


Any help would be highly appreciated!

• what do you mean by questions classifier, predict which sentences are questions. Maybe if it has ? at the end, it's a question – Dirk Nachbar Jul 11 '20 at 20:29
• @DirkNachbar - No.. These all are questions. So I just want to predict the taxonomy (remember, understand, apply) to a given question (question string). – Kasun Jul 12 '20 at 4:26