How can I train a NLP model with discrete labels that is based on multiple text input features?
I'm trying to predict the difficulty of a 4-option multiple choice exam question (probability of a test-taker selecting the correct response) based on the text of the question along with its possible responses. I'm hoping to be able to take into account how some incorrect yet convincing responses, the exact subject of which is relative to the content of the question, can skew the difficulty of the exam question, as well as how the wording of the question can make the question misleading.
Intuition, Attempts and Options:
My intuition is that the content of the question and its responses are both significant in the prediction of its difficulty. However, when using a library like Spacy, NLTK or Textacy, training seems to be done on only one text column at a time. I'm looking at potentially five text columns at a time, or two if I concatenate the question responses together.
I haven't been able to find much on the topic, but here is an attempt I found. I thought this attempt was flawed because they were just doing a single-column train multiple times, and for example training the City against a salary value and concatenating that to your train of Job Description against salary value is not going to give a meaningful improvement to your first model, since the Job Description did not depend on the City when training.
My options that I've found are:
- Following the above attempt after all (which I think is flawed)
- Concatenating my text features (which I can't understand why that would make sense in this case but seems to be the norm)
- Eliminating some of my features entirely, such as narrowing the question down by subject matter and disposing of the question content and just training on the response options concatenated together (which also removes some very important information in the question content that can lead to prediction values)
Thoughts and advice? Is there a library that can make this easier? Thanks!