I have a data contains many categorical columns. When I sampled this data randomly a few times and applied one-hot encoding to categorical columns I noticed that it ended up with datasets with different column counts. Because not all categories in columns preserved in samples and different samples includes different subset of categories for each column. Is there a way to ensure all categorical columns in all samples contains all possible categories?
$\begingroup$
$\endgroup$
6
-
$\begingroup$ It´s hard to give you a good answer to your question, when you have no code example or result picture. So which library did you use --> keras or scikit? $\endgroup$– martinCommented Nov 28, 2020 at 0:23
-
$\begingroup$ You should encode only and exactly the values contained in the training data. Then for the test data you should apply the same encoding (not redo another encoding). If the test data contains a value not seen in the training data, replace this value. $\endgroup$– ErwanCommented Nov 28, 2020 at 1:21
-
$\begingroup$ @Martin, its about methods. I sampled data by sql in database server. It was too big for local computer or other servers we have for processing. $\endgroup$– tkarahanCommented Nov 28, 2020 at 5:59
-
$\begingroup$ @Erwan, the problem is first I sampled data in database server by using sql then I bring data to JupyterHub server and encoded in there. Some categorical columns contains too many categories about 50 or 60 and not all categories are preserved for this columns in sampling. So, after one hot encoding different training samples contains different number of columns. $\endgroup$– tkarahanCommented Nov 28, 2020 at 6:05
-
$\begingroup$ Forming stratas for each category of the columns that have many categories and then applying stratified sampling may be a solution. Any idea? $\endgroup$– tkarahanCommented Nov 28, 2020 at 10:36
|
Show 1 more comment
1 Answer
$\begingroup$
$\endgroup$
2
The first thing we must accept that the sampling is probably doing the right job.
What I mean is that if only 10% is being sampled then some unique value which is less than 5 can be easily missed.
Ideally, you should club these values into some generic value i.e. OTHER_COL_1
But, if you want to get away with this natural result, you should apply some tweaking.
We may do the following -
- Get the sample as you are doing now
- Match the unique element of each column to the unique from the main data
- Iterate on each col and missed unique value
- Let's assume UNIQUE_4 is missed for COL_2
- Sample all the records for UNIQUE_4 from COL_2 of main data and
- Pick one random data out of it
-
-
$\begingroup$ You can also use the "categories" parameter of Scikit-Learn OHE using which you can define all the possible values $\endgroup$– 10xAICommented Dec 1, 2020 at 14:10