I have a labelled dataset of text in 4 languages (1000 samples per languages makes a total of 4000 samples). In one experiment I would like to assess the performance of a classification algorithm (neural network) on an unseen language. That means using the 3000 tweets from three languages as training data and 1000 samples from the fourth language as validation data:
training data: - 1000 English samples - 1000 German samples - 1000 Italian samples validation data: - 1000 Dutch samples
Now, I am trying to find a statistic to find if there is a significant difference between 2 versions of the classification algorithm. I have been reading and find that 2x5 cross-validation or 10x10 cross validation using a modified student t-test is (one of) the best options. While I can apply this to most other experiments, I don't think I can apply this directly to this particular experiment since I am not using folds.
My question is, if I can still use the corrected student t-test training the network a number of times on the same training and validation data? If so, what should be the number of training and validation folds? If not, what would be a better approach?
def corrected_dependent_ttest(data1, data2, n_training_folds, n_test_folds, alpha): n = len(data1) differences = [(data1[i]-data2[i]) for i in range(n)] sd = stdev(differences) divisor = 1 / n * sum(differences) test_training_ratio = n_test_folds / n_training_folds denominator = sqrt(1 / n + test_training_ratio) * sd t_stat = divisor / denominator # degrees of freedom df = n - 1 # calculate the critical value cv = t.ppf(1.0 - alpha, df) # calculate the p-value p = (1.0 - t.cdf(abs(t_stat), df)) * 2.0 # return everything return t_stat, df, cv, p