I am doing a few experiments on medical data. I am about to transfer learn the pretrained networks for my problem.

Firstly, I have to pick a network architecture. Secondly, I would like to optimize it's parameters/parameters of optimizer, to get better performance.

I would like to pick the network architecture based on 10-fold cross validation of several architectures. I will perform cross validation in a way that I have data split to train:test in a 80:20 manner, then train is split into 10 splits. Test set shouldn't ever change. Based on the cross validation, I would like to pick a model I would optimize further on validation set.

Is it okay to test the best architecture on test data too, to see whatever I am moving in a right direction, before optimizing the parameters or am I cheating?

The dataset is imbalanced with relatively high class variance, so I am not even sure whatever the 2 folds for testing will trully represent the dataset.



You shouldn't choose the best model based on the performance on test set. You should run cross-validation and pick the best model from there and then assess performance on a test set. Otherwise you would just getting the best model and overfitting your test set.

Regarding imbalanced data sets, you should be aware of metrics used to assess your model in this case, such as recall, precision, etc., and not only accuracy. In python, you can do StratifiedKFold, where you will split your data getting the same proportion for train and test set regarding target variable.

| improve this answer | |
  • $\begingroup$ I probably wrote it wrong: 1. I will pick best model from cross validation 2. Can I evaluate it on test data to see whatever I am moving towards the right solution? 3. I will optimize hyperparameters based on val. 4. I will evaluate final solution on test and maybe compare with performance before optimizing hyperparameters. Would that make sense? $\endgroup$ – sob3kx Mar 9 '19 at 15:00
  • $\begingroup$ 2. Yes, after choosing the best model from cross-validation, you must assess performance on test set. What I understood was that after it cross-validation iteration you would check performance on test set and pick the best, this is not right. The way you are doing it is the best option, picking the best and calculate results. 4. Yes, of course. In this way you can see if the model behaves in a different way when you change hyper parameters, for example, changing the number of kernels filters, etc. $\endgroup$ – Victor Oliveira Mar 9 '19 at 15:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.