I have a data contains many categorical columns. When I sampled this data randomly a few times and applied one-hot encoding to categorical columns I noticed that it ended up with datasets with different column counts. Because not all categories in columns preserved in samples and different samples includes different subset of categories for each column. Is there a way to ensure all categorical columns in all samples contains all possible categories?

  • $\begingroup$ It´s hard to give you a good answer to your question, when you have no code example or result picture. So which library did you use --> keras or scikit? $\endgroup$
    – martin
    Nov 28 '20 at 0:23
  • $\begingroup$ You should encode only and exactly the values contained in the training data. Then for the test data you should apply the same encoding (not redo another encoding). If the test data contains a value not seen in the training data, replace this value. $\endgroup$
    – Erwan
    Nov 28 '20 at 1:21
  • $\begingroup$ @Martin, its about methods. I sampled data by sql in database server. It was too big for local computer or other servers we have for processing. $\endgroup$
    – tkarahan
    Nov 28 '20 at 5:59
  • $\begingroup$ @Erwan, the problem is first I sampled data in database server by using sql then I bring data to JupyterHub server and encoded in there. Some categorical columns contains too many categories about 50 or 60 and not all categories are preserved for this columns in sampling. So, after one hot encoding different training samples contains different number of columns. $\endgroup$
    – tkarahan
    Nov 28 '20 at 6:05
  • $\begingroup$ Forming stratas for each category of the columns that have many categories and then applying stratified sampling may be a solution. Any idea? $\endgroup$
    – tkarahan
    Nov 28 '20 at 10:36

The first thing we must accept that the sampling is probably doing the right job.
What I mean is that if only 10% is being sampled then some unique value which is less than 5 can be easily missed.
Ideally, you should club these values into some generic value i.e. OTHER_COL_1

But, if you want to get away with this natural result, you should apply some tweaking.

We may do the following -

  • Get the sample as you are doing now
  • Match the unique element of each column to the unique from the main data
  • Iterate on each col and missed unique value
  • Let's assume UNIQUE_4 is missed for COL_2
  • Sample all the records for UNIQUE_4 from COL_2 of main data and
  • Pick one random data out of it
  • $\begingroup$ I understand. Thank you. $\endgroup$
    – tkarahan
    Dec 1 '20 at 7:44
  • $\begingroup$ You can also use the "categories" parameter of Scikit-Learn OHE using which you can define all the possible values $\endgroup$
    – 10xAI
    Dec 1 '20 at 14:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.