0
$\begingroup$

I'm building models for SQUAD (Stanford Question Answering) dataset (https://rajpurkar.github.io/SQuAD-explorer). Stanford doesn't release its test set. It only provides us with training and dev dataset.

Here is my question:

When I tune hyper-parameters of different models, I tuned them on "dev" data given by Stanford. However, we treated "dev" set as "test" set since we didn't have an access to the real "testing" data. We pick only 1 model to submit among different models based on the performance on "dev" dataset.

so I was wondering if I should have separated the training set further into train and dev, and tuned the hyperparams on a newly separated dev set, not the dev set given by Stanford, since we are using "dev" set as "test" set. ​ Did I somehow cheat? Do I need to create another "dev" data from existing train data, splitting it further, and tune the hyperparams there, and check the perf number on given "dev" set?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.