I have a dataset split into training, validation, and testing sets. I trained this model on the training data and evaluated on the validation and testing sets. Now I have an additional set of data that was collected from a separate source on which I have also assessed the classifier. I'm interested in testing whether the gain in performance compared to a random model is similar between the two datasets which have different numbers of positive samples. Is there a procedure that is standard to employ in such a case? Anything more robust than simply comparing {classifier test set #1 performance - random performance} to {classifier test set #2 performance - random performance}?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.