0
$\begingroup$

My task is to create a QA-model. I give it a context and a question that it should answer. The answer is usually one word, so a very simplified input would be e.g.

Context: "Max eats a banana. Now Max and Tim are going to the gym. While Max eats a banana on the way, Tim eats some chocolate."

Question: "What does Max eat?"

Answer: "(a) banana(.)"

I use a pretained model that predicts start and end tokens. To be more precise, it assigns a value to each of the tokens. Assuming each word would be tokenized into 1 token, I would have 2 vectors of length ~30, representing how likely it is that the answer to the question starts or ends at token x.

This is how the predictions for the start_tokens could look like: X = [10, 3, -4, 1, ..., 9, ...] where 10 and 9 are the index of the 2 relevant occurrences of "Max"

The training data is also labeled with those start and end indices (on char level, but I can easily transform it to token level). I thought of a comparison matrix like Y = [10, -10, -10, -10, ..., 10, ...], basically assigning all valid start indices the value 10 and all the rest -10.

This does not perform well, tho. I tried many different loss functions on this setup but everything fails. Any ideas on how to get this running? Am I maybe on the completely wrong track here with what I'm doing?

$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.