3
$\begingroup$

I got stuck on this paragraph from the academic article "Measuring news sentiment": https://www.sciencedirect.com/science/article/pii/S0304407620303535#tbl3

"As is best practice, we split the labeled dataset into a training set, a development set, and a hold-out test set. The development and test sets have 100 observations each, leaving 600 observations for the training set. (..) hyper-parameter optimization is done through grid search, using cross-validation on the training set to evaluate model performance for each possible set of hyper-parameters. The optimal model is then evaluated against the development set. Finally, after all models have been developed, we test them all against our hold-out test set for final results."

To me, what they describe (the bold parts) is incoherent. I would like to know if I'm wrong or not. I understand "cross-validation on the training set" as doing the validation on a subsample of the training set, e.g. doing k-fold cross validation. But if you split the dataset into training, development, and test set, why would you do validation within the training set, as I understand the authors assert (second bold part)? The validation should be done on the development set, shouldn't it? Is that a semantic mistake or are they doing something I don't understand?

$\endgroup$
1

1 Answer 1

2
$\begingroup$

The validation set (or development or dev set) is used as the intermediate performance indicator, e.g. after optimizing the hyperparameters in this case. That means we cannot use it while we're still training the model, if we do so, the model quickly learns the validation set and the metric wouldn't be a true estimator of model performance.

The model is built on the training dataset, intermediate performance measured on the validation dataset, and the final performance measured on the test dataset.

Generally, when we are

  • testing parameters,
  • tuning hyper-parameters,
  • or anytime we are frequently evaluating model performance

we need to create a second holdout sample, called the validation dataset.

$\endgroup$
2
  • $\begingroup$ Thanks. But then do you agree that the formulation "using cross-validation on the training set" is incorrect and/or confusing, given that the authors stated that the validation set was already set aside? I understand the "on the training set" as "within the training set". $\endgroup$
    – Julien
    Oct 10, 2022 at 15:29
  • $\begingroup$ Doing the cross validation on the training set is part of training the model. You cannot check the the performance of validation set without having the model. That's the point here. $\endgroup$
    – Miss.Alpha
    Oct 10, 2022 at 20:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.