"cross-validation on the training set" while development and test set are distinct from the training: does it make sense? semantic mistake?

Question

I got stuck on this paragraph from the academic article "Measuring news sentiment": https://www.sciencedirect.com/science/article/pii/S0304407620303535#tbl3

"As is best practice, we split the labeled dataset into a training set, a development set, and a hold-out test set. The development and test sets have 100 observations each, leaving 600 observations for the training set. (..) hyper-parameter optimization is done through grid search, using cross-validation on the training set to evaluate model performance for each possible set of hyper-parameters. The optimal model is then evaluated against the development set. Finally, after all models have been developed, we test them all against our hold-out test set for final results."

To me, what they describe (the bold parts) is incoherent. I would like to know if I'm wrong or not. I understand "cross-validation on the training set" as doing the validation on a subsample of the training set, e.g. doing k-fold cross validation. But if you split the dataset into training, development, and test set, why would you do validation within the training set, as I understand the authors assert (second bold part)? The validation should be done on the development set, shouldn't it? Is that a semantic mistake or are they doing something I don't understand?

Answer 1

The validation set (or development or dev set) is used as the intermediate performance indicator, e.g. after optimizing the hyperparameters in this case. That means we cannot use it while we're still training the model, if we do so, the model quickly learns the validation set and the metric wouldn't be a true estimator of model performance.

The model is built on the training dataset, intermediate performance measured on the validation dataset, and the final performance measured on the test dataset.

Generally, when we are

• testing parameters,
• tuning hyper-parameters,
• or anytime we are frequently evaluating model performance

we need to create a second holdout sample, called the validation dataset.