I ran two model-building procedures with different parameters on the same sample and obtained the selection of my optimized hyperparameter for each outer fold (each of the analyses had 100 outer folds in my nested-cross validation). This gives me a contingency table, with the two models as rows and the absolute frequencies for each of the possible hyperparameter categories as columns. For the sake of simplicity, let's call the rows day 1, and day 2, let the columns be different ice cream sorts and let the 100 folds be the same sample of people buying ice cream:

enter image description here

As you can 'see', on day 1 the distribution of my categorical variable is way more spread out than on day 2. On day two, the hundred people only bought vanilla and choco ice cream.

I would like to measure this difference in variance. My first idea was to use a Chi2-Goodness of Fit test, where I test the second row against the first row and treat this row as 'expected frequencies'. But I am not sure, if this is the right approach, and if the Chi2-Test does the right thing here. Can anyone recommend a test that handles this problem?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy