So i'm doing cross validation and then i'm predicting using all the data on a test set ( a hold-out set ). My hold-out set has the same ratio on a column than the train ( seems thats how the test set was generated, a function that sampled it and tried to preserve the ratio for the target classes, and a particular column ) . My local CV is a bit lower than my score on the test set, and i think the problem is stemming from the fact that i'm using stratification only for 'y'.
Can lack of stratification of that feature be the reason of Cv & test scores aren't really close?
And if so how can i perform stratification for the target and a feature! Thanks
Edit : i'm already doing stratification on the target since my data is imbalanced.