How to properly apply CrossValidation and/or split the dataset?

Question

I have a particular problem and do not really now how to properly validate my experiments in this scenario.

• There is one big data set with 100.000 samples, 99.000 y=0, 1.000 y=1
• Each sample has 1.000 Features
• There are 10 different subsets of feature combinations which have to be evaluated to get information about more or less expressive feature groups
• Because of the different number and type of features each subset, an adequate models architecture has to be figured out manually
• The performance of a few different kinds of models have to be evaluated
• The performance of several different sampling techniques have to be evaluated
• 10-fold Cross Validation has to be applied
• Random Search has to be applied for each experimental configuration (Combination of subset + model + sampling technique)

These facts lead to the following experimental routine:

train, val = splitDataset()
iterate each featureSubset:
    iterate each modelType:
        model = manuallySearchGoodArchitecture(featureSubset, modelType, train, val)

        iterate each samplingTechnique:
            train_temporary = applySampling(samplingTechnique)
            HPs = HPOptimization(model, train_temporary, val)
            result =  k-FOldCV(model, HPs, samplingTechnique, train)
        end
    end
end

So, now I'm not sure about whether the CrossValidation is applied correctly. Because the model which has to be validated has already seen the train data (during manuallySearchGoodArchitecture and HPOptimization) and those are also used for testing purpose during CV.

1. So, is this routine faulty?
2. What would be the correct way of performing those experiments while applying proper validation technique?
3. How must the data set be splitted?

Answer 1

I'm not sure that I understand every part of the process but there is one clear issue with it: because the CV is applied in the inner loop, there is a serious risk of overfitting the model with respect to the other parameters (feature subset, model type, sampling technique). Depending on the goal, this is not necessarily wrong but it's important to interpret the results accordingly. For instance in this case the results will show how different subsets of features perform on the data, but this difference in performance across subsets shouldn't be considered reliable: it's possible that a particular subset happens to be better than another by chance.

It's quite a complex setting and I assume that there are efficiency constraints to take into account. If possible the most reliable results would be obtained by doing several stages of CV (or other techniques, e.g. bagging) using different subsets of data. For instance you could run the whole process a few times, each time using a different random subset of instances: in this way you can average performance and see whether a particular subset of features is constantly better than another (for example, same idea for other parameters).

---

[edited] Disclaimer: I don't know if there is any standard way to proceed with a complex multi-level setting like this, my advice is based only on the experience I had with a few broadly similar cases.

Generally the idea is that every choice to make can be considered as an hyper-parameter, including the subset of features, the type of model, the architecture, the sampling technique. Therefore I think that ideally one would cross-validate at every level, i.e. put every loop level in a function and call this function k times with a different subset of data. It would look like something like this:

train1, val1 = splitDataset()
iterate each featureSubset:
    train2, val2 = splitData(train1)
    resultTrain2 = kfoldCV(processLevel2, train2)
    resultLevel2 = apply(resultTrain2, val2)
resultLevel1 = apply(resultLevel2, val1)

processLevel2:
    iterate each modelType:
        train3, val3 = splitData(train2)
        resultTrain3 = kfoldCV(processLevel3, train3)
        ...

remark: I'm not 100% sure about the algorithm, maybe I over-complicated it. I think it gives the general idea though.

But of course following this logic the computational complexity becomes way too high, so you will probably have to take a few shortcuts. One thing I've tried successfully in the past is to use genetic learning in order to optimize different parameters at the same time: that would mean having different "genes" which represent the different parameters (feature subset, model type, etc.), each with its set of values, and then run the genetic process which is supposed to converge to a set of optimal values for all the parameters (I was using CV every time a particular combination of parameters is evaluated). But again I don't know if it's a very orthodox method :)

---

[edit2]

After more thought I think I would try to do something like this:

innerData, valOuter = splitDataset() // keep large amount for validation set, say between 20-40%
train, valInner = splitDataset(innerData)
iterate each featureSubset:
    iterate each modelType:
        model = manuallySearchGoodArchitecture(featureSubset, modelType, train, val)

        iterate each samplingTechnique:
            train_temporary = applySampling(samplingTechnique)
            HPs = HPOptimization(model, train_temporary, valInner)
            result =  k-FOldCV(model, HPs, samplingTechnique, train)
        end
    end
end

bestHPCombinations = select top N HPCombinations from first stage
for each HPCombination in bestHPCombinations
     model = train(innerData)
     result = apply(model, valOuter)
end

It's simpler: the idea is just to re-evaluate the results from the first stage on some fresh data in order to avoid overfiting (here HPCombination includes featuresSubset, modelType, etc.). The second stage could also include CV or bagging for more reliable results. But in general I think this option would be reasonably reliable, since it's unlikely that the best models from the first stage would also be the best models in the second stage by chance only.