First i want to tell you that i'm trying to create a model in the right steps, which needs data exploration, data preprocessing (AFTER splitting the data into train/test), etc. For the part of data exploration i want to plot the data for visualizing the correlation,distribution, etc. I don't know when i supposed to do this so i decided to plot all the things i need BEFORE splitting the data into train/test. One of the plot is the correlation with seaborn heatmap, my data contains numerical and categorical values (which needs to be encoded to binary number), since i preprocess the data AFTER the splitting, the heatmap cannot include the categorical values since its not binaried yet. Now i'm really confused of how to tackle this issue, i still need to know the correlation between the categorical feature with the target but if i encode it BEFORE the split train/test, it will defies the proper manner that i want to follow. Anyone has solution or maybe opinion?

  • $\begingroup$ Why is it different after you do the split? just perform the same analysis, but on the train split. The data types are identical. $\endgroup$
    – Sean Owen
    Commented Mar 6, 2020 at 17:33
  • $\begingroup$ Plotting the train dataset? is it possible? i mean it's already numpy array and only have index instead of the header. Besides i need to know the correlation between the categorical feature which later will be train dataset and the target which later will be test dataset $\endgroup$ Commented Mar 7, 2020 at 1:00
  • $\begingroup$ You have the col names from the source DataFrame, presumably. I don't see how you can plot a thing, but not plot 80% of it? You can one-hot encode it, or not, as you need, for plotting. $\endgroup$
    – Sean Owen
    Commented Mar 7, 2020 at 2:17


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.