# How to properly apply CrossValidation and/or split the dataset?

I have a particular problem and do not really now how to properly validate my experiments in this scenario.

• There is one big data set with 100.000 samples, 99.000 y=0, 1.000 y=1
• Each sample has 1.000 Features
• There are 10 different subsets of feature combinations which have to be evaluated to get information about more or less expressive feature groups
• Because of the different number and type of features each subset, an adequate models architecture has to be figured out manually
• The performance of a few different kinds of models have to be evaluated
• The performance of several different sampling techniques have to be evaluated
• 10-fold Cross Validation has to be applied
• Random Search has to be applied for each experimental configuration (Combination of subset + model + sampling technique)

These facts lead to the following experimental routine:

train, val = splitDataset()
iterate each featureSubset:
iterate each modelType:
model = manuallySearchGoodArchitecture(featureSubset, modelType, train, val)

iterate each samplingTechnique:
train_temporary = applySampling(samplingTechnique)
HPs = HPOptimization(model, train_temporary, val)
result =  k-FOldCV(model, HPs, samplingTechnique, train)
end
end
end


So, now I'm not sure about whether the CrossValidation is applied correctly. Because the model which has to be validated has already seen the train data (during manuallySearchGoodArchitecture and HPOptimization) and those are also used for testing purpose during CV.

1. So, is this routine faulty?
2. What would be the correct way of performing those experiments while applying proper validation technique?
3. How must the data set be splitted?

I'm not sure that I understand every part of the process but there is one clear issue with it: because the CV is applied in the inner loop, there is a serious risk of overfitting the model with respect to the other parameters (feature subset, model type, sampling technique). Depending on the goal, this is not necessarily wrong but it's important to interpret the results accordingly. For instance in this case the results will show how different subsets of features perform on the data, but this difference in performance across subsets shouldn't be considered reliable: it's possible that a particular subset happens to be better than another by chance.

It's quite a complex setting and I assume that there are efficiency constraints to take into account. If possible the most reliable results would be obtained by doing several stages of CV (or other techniques, e.g. bagging) using different subsets of data. For instance you could run the whole process a few times, each time using a different random subset of instances: in this way you can average performance and see whether a particular subset of features is constantly better than another (for example, same idea for other parameters).

[edited] Disclaimer: I don't know if there is any standard way to proceed with a complex multi-level setting like this, my advice is based only on the experience I had with a few broadly similar cases.

Generally the idea is that every choice to make can be considered as an hyper-parameter, including the subset of features, the type of model, the architecture, the sampling technique. Therefore I think that ideally one would cross-validate at every level, i.e. put every loop level in a function and call this function k times with a different subset of data. It would look like something like this:

train1, val1 = splitDataset()
iterate each featureSubset:
train2, val2 = splitData(train1)
resultTrain2 = kfoldCV(processLevel2, train2)
resultLevel2 = apply(resultTrain2, val2)
resultLevel1 = apply(resultLevel2, val1)

processLevel2:
iterate each modelType:
train3, val3 = splitData(train2)
resultTrain3 = kfoldCV(processLevel3, train3)
...


remark: I'm not 100% sure about the algorithm, maybe I over-complicated it. I think it gives the general idea though.

But of course following this logic the computational complexity becomes way too high, so you will probably have to take a few shortcuts. One thing I've tried successfully in the past is to use genetic learning in order to optimize different parameters at the same time: that would mean having different "genes" which represent the different parameters (feature subset, model type, etc.), each with its set of values, and then run the genetic process which is supposed to converge to a set of optimal values for all the parameters (I was using CV every time a particular combination of parameters is evaluated). But again I don't know if it's a very orthodox method :)

[edit2]

After more thought I think I would try to do something like this:

innerData, valOuter = splitDataset() // keep large amount for validation set, say between 20-40%
train, valInner = splitDataset(innerData)
iterate each featureSubset:
iterate each modelType:
model = manuallySearchGoodArchitecture(featureSubset, modelType, train, val)

iterate each samplingTechnique:
train_temporary = applySampling(samplingTechnique)
HPs = HPOptimization(model, train_temporary, valInner)
result =  k-FOldCV(model, HPs, samplingTechnique, train)
end
end
end

bestHPCombinations = select top N HPCombinations from first stage
for each HPCombination in bestHPCombinations
model = train(innerData)
result = apply(model, valOuter)
end


It's simpler: the idea is just to re-evaluate the results from the first stage on some fresh data in order to avoid overfiting (here HPCombination includes featuresSubset, modelType, etc.). The second stage could also include CV or bagging for more reliable results. But in general I think this option would be reasonably reliable, since it's unlikely that the best models from the first stage would also be the best models in the second stage by chance only.

• Hey Erwan, many thanks for your comprehensive response! You're right in your assumptions about efficiency constraints. So, the idea was it to briefly search for an acceptable architecture each subset (and model type of course) manually which will be used for any sampling technique. But perform a brief HPOptimization (without CV due to computational cost) for each sampling (because if size of data vary, its very likely that optimal batchsize / lr also differ). Finally apply the CV for the previously found best HPs each sampling technique to get a kind of reliable performance for this config. – SamSampleman Jan 19 '20 at 8:28
• But I'm aware that there is a serious risk of not using the really best HP config within the final CV due to lacking a previous CV for HPOptimization. One think I which is not totally clear to me, what do you mean by "there is a serious risk of overfitting" ? Could you elaborate this point? Could you provide brief pseudocode of the proper evaluation process (supposed there aren't ANY time/efficiency constraints) ? Many thanks for your support. – SamSampleman Jan 19 '20 at 8:31
• @SamSampleman I tried to give more detail in the answer. the risk of overfitting is due to the fact that at the highest level every choice (features, model...) behaves like an hyper-parameter: it might be good just by chance with a particular CV split. – Erwan Jan 19 '20 at 14:47
• Thanks for your great response! One further question regarding your idea. So I understood K-FoldCV especially to achieve a more robust insight about a models performance. Meaning to avoid picking a very easy test set for testing purpose by chance. So we simply validate the model 10 times on different train/test set configurations to exclude the possibility of just having luck. Your algorithm uses K-FoldCV as kind of Hyper parameter optimization am I right? Isn't there the risk in your algorithm that the model could perform pretty damn good, just because 'val1' was pretty easy by chance. – SamSampleman Jan 21 '20 at 15:13
• If I'm not mistaken this can be avoided by looking at the detail of resultLevel2 and compare with resultLevel1: normally the results for the same model should be roughly similar. But to be honest I'm a bit confused by my own method here: I'm thinking it might be more relevant to compare the results of the training and validation set (i.e. resultsTrain2 and resultsLevel2), because that's where overfitting could be detected... I'm fairly sure there's a way to minimize the chance factor but I don't really see it clearly right now! – Erwan Jan 21 '20 at 19:23