I'm building models for SQUAD (Stanford Question Answering) dataset (https://rajpurkar.github.io/SQuAD-explorer). Stanford doesn't release its test set. It only provides us with training and dev dataset.
Here is my question:
When I tune hyper-parameters of different models, I tuned them on "dev" data given by Stanford. However, we treated "dev" set as "test" set since we didn't have an access to the real "testing" data. We pick only 1 model to submit among different models based on the performance on "dev" dataset.
so I was wondering if I should have separated the training set further into train and dev, and tuned the hyperparams on a newly separated dev set, not the dev set given by Stanford, since we are using "dev" set as "test" set. Did I somehow cheat? Do I need to create another "dev" data from existing train data, splitting it further, and tune the hyperparams there, and check the perf number on given "dev" set?