Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I trained a model using a small mri dataset(57 patients). The model's performance was so low(Train set 0.7, Val set 0.7, Test set 0.45).
I found the model segment tumor in upper part of brain well, couldn't segment tumor which is in middle part of brain.
So i stratified sampled the whole dataset with a position stratum(upper, not upper), the model performance was improved.
In this case, is it cheating to do that?
By having seen the results on the held-out test set, learning something and applying that to the model, you have essentially added some bias to your model. So if we are being strict, yes... You cheated a little bit ;)
Without having a solid (meaning unbiased) validation metric, you cannot objectively compare the two results. You used prior information to make an improved modelling decision, which is great :) However, that information flowed directly from your test data, which introduced the bias.
It can be difficult when you have limited data, I know. You could maybe look into data augmentation to synthetically increase your dataset. In your case, adding random rotations and mirrored images could help a great deal!
