So, I have found that there are many ways to classify words with sklearn's SVM algorithm. But I want to classify questions by taxonomy, as shown in the following dataset:

enter image description here

The goal of this task is to predict the taxonomy given the pdf file / string (question). The questions are the following:

  • How can I modify the below code to train a question-based classification model?
  • How can I train a question classification model with SVM?

For this task, I have used the following Python libraries

import pandas as pd
import numpy as np
from nltk.tokenize import word_tokenize, sent_tokenize
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.preprocessing import LabelEncoder
from collections import defaultdict
from nltk.corpus import wordnet as wn
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn import model_selection, naive_bayes, svm
from sklearn.metrics import accuracy_score

Here, I have split the dataset into train and test sets.

Train_X, Test_X, Train_Y, Test_Y = 

Encoder = LabelEncoder()
Train_Y = Encoder.fit_transform(Train_Y)
Test_Y = Encoder.fit_transform(Test_Y)

I have used TF IDF transformer (TfidfTransformer) in sklearn library like this

Tfidf_vect = TfidfVectorizer(max_features=5000)
Train_X_Tfidf = Tfidf_vect.transform(Train_X)
Test_X_Tfidf = Tfidf_vect.transform(Test_X)

But this will break down questions into words for each question. The following code was used to make a word classifier to predict the taxonomy from words

SVM = svm.SVC(C=1.0, kernel='linear', degree=3, gamma='auto')
# predict the labels on validation dataset
predictions_SVM = SVM.predict(Test_X_Tfidf)

Any help would be highly appreciated!

  • $\begingroup$ what do you mean by questions classifier, predict which sentences are questions. Maybe if it has ? at the end, it's a question $\endgroup$ – Dirk Nachbar Jul 11 '20 at 20:29
  • $\begingroup$ @DirkNachbar - No.. These all are questions. So I just want to predict the taxonomy (remember, understand, apply) to a given question (question string). $\endgroup$ – Kasun Jul 12 '20 at 4:26

So, the question is asking how to model the following problem: predicting question taxonomy, given the question.

The first thing that leaps into my mind is to use an encoder-decoder architecture. For this, you would use an encoder to encode the question as a sequence of words/tokens. Here we could use a sequential model, such as an RNN or LSTM. This encoder then encodes the question into "hidden representation".
Then the hidden representation is decoded, using a normal feedforward NN with a final 3-node softmax layer, which creates a probability distribution over question taxonomies.

For the input, you would convert the tokens into word embeddings and then feed them in one-by-one for each question. For the output, you would just obtain the index with the highest probability as the class label.

An SVM is a good idea, however this treats the question as a bag of words (i.e. word order does not matter). So any old random question such as "how what learn learn this that" (even though hugely incoherent") can still be classified.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.