How can I train a NLP model with discrete labels that is based on multiple text input features?


I'm trying to predict the difficulty of a 4-option multiple choice exam question (probability of a test-taker selecting the correct response) based on the text of the question along with its possible responses. I'm hoping to be able to take into account how some incorrect yet convincing responses, the exact subject of which is relative to the content of the question, can skew the difficulty of the exam question, as well as how the wording of the question can make the question misleading.

Intuition, Attempts and Options:

My intuition is that the content of the question and its responses are both significant in the prediction of its difficulty. However, when using a library like Spacy, NLTK or Textacy, training seems to be done on only one text column at a time. I'm looking at potentially five text columns at a time, or two if I concatenate the question responses together.

I haven't been able to find much on the topic, but here is an attempt I found. I thought this attempt was flawed because they were just doing a single-column train multiple times, and for example training the City against a salary value and concatenating that to your train of Job Description against salary value is not going to give a meaningful improvement to your first model, since the Job Description did not depend on the City when training.

My options that I've found are:

  • Following the above attempt after all (which I think is flawed)
  • Concatenating my text features (which I can't understand why that would make sense in this case but seems to be the norm)
  • Eliminating some of my features entirely, such as narrowing the question down by subject matter and disposing of the question content and just training on the response options concatenated together (which also removes some very important information in the question content that can lead to prediction values)

Thoughts and advice? Is there a library that can make this easier? Thanks!

  • $\begingroup$ An update on my thoughts: I've noticed that there exist some NLP models that can predict the similarities between trained words. I've yet to look into this further, but maybe a possible solution to the difficulty of multiple choice questions lies in the similarity between the correct option and the incorrect ones. However, to train such a model, I would need unreasonable amounts of data on each of my different question topics. Maybe this problem is too expansive to apply to a machine learning algorithm? $\endgroup$ – Carl Molnar Mar 1 '19 at 13:55

Concatenating the whole question and its answers in a RNN could be an option to try, but then always use a reserved special token (or various) to mark where the questions start. E.g. you could concatenate like:

Question text <1> answer 1 <2> answer 2 <3> answer 3 <4> answer 4

where <1>, <2>... are the special tokens so that the model, with enough examples, may be able to understand its meaning.

The "enough examples" is worth stressing, specially as this type of model may require a large complexity (in terms of number of parameters) to make it work, and hence you will need a large enough data set, possibly of the order of the 10k or larger. You can also test some data augmentation by mixing the order of the answers in the input, and of course changing the correct label.

|improve this answer|||||
  • $\begingroup$ Thanks for your answer! This seems really interesting, I realize you're right that there must be some way to denote each question if I'm gonna concatenate everything. I have about 32k questions but the subject matter varies. If I was to look at just one subject, I'd have at most 3.2k questions per subject, which is worrying. I'll consider this idea my best course of action for now! $\endgroup$ – Carl Molnar Mar 1 '19 at 5:45
  • $\begingroup$ Do you have any examples of this kind of solution? That would help me so much. $\endgroup$ – Carl Molnar Mar 1 '19 at 5:51
  • $\begingroup$ Hi Carl, no, I don't have any example. Another idea is to write your model as a combination of two rnn, which, for example, concatenate their last activation. One rnn receives the question and the other rnn one answer (of the four given). If the answer is the correct one, the model should return a 1, otherwise a zero. $\endgroup$ – Escachator Mar 3 '19 at 23:32

NLP is rather quite hyper-dimensional. I'd go data-driven way and use some pretrained embedder. Nowadays there're a few to choose from, like LASER from Facebook. There's unofficial pypi lib, though it works just fine. If you want to reach seminal-like scores, there's no point in doing NLP by hand. Embedders usually cover dozens of languages, so you can feed training data in any language you want. Your models will also work for those languages out-of-the-box, even if you trained them on other languages. If you need some custom stuff, you could pick BERT from Google, though you'll have to push it yourself further. It isn't really pretrained that much.

You can try to encode the question and each of the answers separately and go ensemble with it. You could also try to scram it all into encoder. It should do just fine.

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.