I'm not sure that I understand every part of the process but there is one clear issue with it: because the CV is applied in the inner loop, there is a serious risk of overfitting the model with respect to the other parameters (feature subset, model type, sampling technique). Depending on the goal, this is not necessarily wrong but it's important to interpret the results accordingly. For instance in this case the results will show how different subsets of features perform on the data, but this difference in performance across subsets shouldn't be considered reliable: it's possible that a particular subset happens to be better than another by chance.
It's quite a complex setting and I assume that there are efficiency constraints to take into account. If possible the most reliable results would be obtained by doing several stages of CV (or other techniques, e.g. bagging) using different subsets of data. For instance you could run the whole process a few times, each time using a different random subset of instances: in this way you can average performance and see whether a particular subset of features is constantly better than another (for example, same idea for other parameters).
[edited] Disclaimer: I don't know if there is any standard way to proceed with a complex multi-level setting like this, my advice is based only on the experience I had with a few broadly similar cases.
Generally the idea is that every choice to make can be considered as an hyper-parameter, including the subset of features, the type of model, the architecture, the sampling technique. Therefore I think that ideally one would cross-validate at every level, i.e. put every loop level in a function and call this function k times with a different subset of data. It would look like something like this:
train1, val1 = splitDataset()
iterate each featureSubset:
train2, val2 = splitData(train1)
resultTrain2 = kfoldCV(processLevel2, train2)
resultLevel2 = apply(resultTrain2, val2)
resultLevel1 = apply(resultLevel2, val1)
processLevel2:
iterate each modelType:
train3, val3 = splitData(train2)
resultTrain3 = kfoldCV(processLevel3, train3)
...
remark: I'm not 100% sure about the algorithm, maybe I over-complicated it. I think it gives the general idea though.
But of course following this logic the computational complexity becomes way too high, so you will probably have to take a few shortcuts. One thing I've tried successfully in the past is to use genetic learning in order to optimize different parameters at the same time: that would mean having different "genes" which represent the different parameters (feature subset, model type, etc.), each with its set of values, and then run the genetic process which is supposed to converge to a set of optimal values for all the parameters (I was using CV every time a particular combination of parameters is evaluated). But again I don't know if it's a very orthodox method :)
[edit2]
After more thought I think I would try to do something like this:
innerData, valOuter = splitDataset() // keep large amount for validation set, say between 20-40%
train, valInner = splitDataset(innerData)
iterate each featureSubset:
iterate each modelType:
model = manuallySearchGoodArchitecture(featureSubset, modelType, train, val)
iterate each samplingTechnique:
train_temporary = applySampling(samplingTechnique)
HPs = HPOptimization(model, train_temporary, valInner)
result = k-FOldCV(model, HPs, samplingTechnique, train)
end
end
end
bestHPCombinations = select top N HPCombinations from first stage
for each HPCombination in bestHPCombinations
model = train(innerData)
result = apply(model, valOuter)
end
It's simpler: the idea is just to re-evaluate the results from the first stage on some fresh data in order to avoid overfiting (here HPCombination includes featuresSubset, modelType, etc.). The second stage could also include CV or bagging for more reliable results. But in general I think this option would be reasonably reliable, since it's unlikely that the best models from the first stage would also be the best models in the second stage by chance only.