0
$\begingroup$

I trained a model using a small mri dataset(57 patients). The model's performance was so low(Train set 0.7, Val set 0.7, Test set 0.45).

I found the model segment tumor in upper part of brain well, couldn't segment tumor which is in middle part of brain.

So i stratified sampled the whole dataset with a position stratum(upper, not upper), the model performance was improved.

In this case, is it cheating to do that?

$\endgroup$
0
$\begingroup$

By having seen the results on the held-out test set, learning something and applying that to the model, you have essentially added some bias to your model. So if we are being strict, yes... You cheated a little bit ;)

Without having a solid (meaning unbiased) validation metric, you cannot objectively compare the two results. You used prior information to make an improved modelling decision, which is great :) However, that information flowed directly from your test data, which introduced the bias.

It can be difficult when you have limited data, I know. You could maybe look into data augmentation to synthetically increase your dataset. In your case, adding random rotations and mirrored images could help a great deal!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.