I'm building models for SQUAD (Stanford Question Answering) dataset (https://rajpurkar.github.io/SQuAD-explorer). Stanford doesn't release its test set. It only provides us with training and dev dataset.

Here is my question:

When I tune hyper-parameters of different models, I tuned them on "dev" data given by Stanford. However, we treated "dev" set as "test" set since we didn't have an access to the real "testing" data. We pick only 1 model to submit among different models based on the performance on "dev" dataset.

so I was wondering if I should have separated the training set further into train and dev, and tuned the hyperparams on a newly separated dev set, not the dev set given by Stanford, since we are using "dev" set as "test" set. ​ Did I somehow cheat? Do I need to create another "dev" data from existing train data, splitting it further, and tune the hyperparams there, and check the perf number on given "dev" set?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.