# Statistical comparison of model performance when training and validation data is always the same

I have a labelled dataset of text in 4 languages (1000 samples per languages makes a total of 4000 samples). In one experiment I would like to assess the performance of a classification algorithm (neural network) on an unseen language. That means using the 3000 tweets from three languages as training data and 1000 samples from the fourth language as validation data:

training data:
- 1000 English samples
- 1000 German samples
- 1000 Italian samples

validation data:
- 1000 Dutch samples


Now, I am trying to find a statistic to find if there is a significant difference between 2 versions of the classification algorithm. I have been reading and find that 2x5 cross-validation or 10x10 cross validation using a modified student t-test is (one of) the best options. While I can apply this to most other experiments, I don't think I can apply this directly to this particular experiment since I am not using folds.

My question is, if I can still use the corrected student t-test training the network a number of times on the same training and validation data? If so, what should be the number of training and validation folds? If not, what would be a better approach?

For reference, below is my Python implementation of the Nadeau and Bengio correction using the equation stated here.

def corrected_dependent_ttest(data1, data2, n_training_folds, n_test_folds, alpha):
n = len(data1)
differences = [(data1[i]-data2[i]) for i in range(n)]
sd = stdev(differences)
divisor = 1 / n * sum(differences)
test_training_ratio = n_test_folds / n_training_folds
denominator = sqrt(1 / n + test_training_ratio) * sd
t_stat = divisor / denominator
# degrees of freedom
df = n - 1
# calculate the critical value
cv = t.ppf(1.0 - alpha, df)
# calculate the p-value
p = (1.0 - t.cdf(abs(t_stat), df)) * 2.0
# return everything
return t_stat, df, cv, p