# Cross validation while preserving a column (not the target ) distribution

So i'm doing cross validation and then i'm predicting using all the data on a test set ( a hold-out set ). My hold-out set has the same ratio on a column than the train ( seems thats how the test set was generated, a function that sampled it and tried to preserve the ratio for the target classes, and a particular column ) . My local CV is a bit lower than my score on the test set, and i think the problem is stemming from the fact that i'm using stratification only for 'y'.

Can lack of stratification of that feature be the reason of Cv & test scores aren't really close?

And if so how can i perform stratification for the target and a feature! Thanks

Edit : i'm already doing stratification on the target since my data is imbalanced.

Example: say for some observations, the target and the column take the following values: target = [0,1,0] and column = [A,A,B]. Then the combined column could look something like [A0,A1,B0] and could be used for stratification.
• My thinking is that you want to preserve the combination of values of target and column in your train and test set... if 25% of data is (target == 0)&(column=='A') then 25% of the combined column will be A0 and so the stratification will preserve the combination. Also, this conversation may be helpful: stackoverflow.com/questions/45516424/… – bradS Sep 3 '19 at 11:23