Statistical comparison of model performance when training and validation data is always the same

Question

I have a labelled dataset of text in 4 languages (1000 samples per languages makes a total of 4000 samples). In one experiment I would like to assess the performance of a classification algorithm (neural network) on an unseen language. That means using the 3000 tweets from three languages as training data and 1000 samples from the fourth language as validation data:

training data:
- 1000 English samples
- 1000 German samples
- 1000 Italian samples

validation data:
- 1000 Dutch samples

Now, I am trying to find a statistic to find if there is a significant difference between 2 versions of the classification algorithm. I have been reading and find that 2x5 cross-validation or 10x10 cross validation using a modified student t-test is (one of) the best options. While I can apply this to most other experiments, I don't think I can apply this directly to this particular experiment since I am not using folds.

My question is, if I can still use the corrected student t-test training the network a number of times on the same training and validation data? If so, what should be the number of training and validation folds? If not, what would be a better approach?

For reference, below is my Python implementation of the Nadeau and Bengio correction using the equation stated here.

def corrected_dependent_ttest(data1, data2, n_training_folds, n_test_folds, alpha):
    n = len(data1)
    differences = [(data1[i]-data2[i]) for i in range(n)]
    sd = stdev(differences)
    divisor = 1 / n * sum(differences)
    test_training_ratio = n_test_folds / n_training_folds  
    denominator = sqrt(1 / n + test_training_ratio) * sd
    t_stat = divisor / denominator
    # degrees of freedom
    df = n - 1
    # calculate the critical value
    cv = t.ppf(1.0 - alpha, df)
    # calculate the p-value
    p = (1.0 - t.cdf(abs(t_stat), df)) * 2.0
    # return everything
    return t_stat, df, cv, p

Answer 1

If your model is deterministic (no randomness), then repeating the training/testing on the exact same set of data is pointless - you will get the exact same answer every time. The benefit of cross-validation is that it provides an unbiased estimate of your model performance, and does so by using different perturbations of the train/test data. You can still do something similar, selecting 80% of your training data and testing on some subset of the test data, and repeatedly doing a resampling. There's a slight difference from traditional CV, where your train/test set are mutually exclusive and essentially define one another, whereas in this case, your training and test datasets can be defined totally independently (but that shouldn't be a problem).

Incidentally, what model are you using that you expect can accurately predict data that's completely unlike what it's been trained on? Usually the point of the training data is to provide the model with examples of the data you expect to see, along with the correct output. It's not clear to me why you'd want to train a model to classify tweets in English, but then test how it performs at classifying Dutch tweets - the model has never seen Dutch, so I don't expect it would perform well. It seems this evaluation would test how similar English and Dutch are, rather than testing how good your classification model is.