2
$\begingroup$

If we have a column like:

    Name
0  Alice
1    Bob
2   Dave

then, after numeric encoding, it becomes:

    Name
0  0
1  1
2  2

What if, however, we have a column like this:

              Names
0        Alice, Bob
1  Alice, Bob, Dave
2              Dave

One way to encode it would be like this:

   Alice  Bob  Dave
0      1    1     0
1      1    1     1
2      0    0     1

However, this creates a lot of extra columns. Is there a way to encode such a column in a way that doesn't end up with loads of extra columns, just like using numeric encoding instead of one-hot-encoding prevents too many columns from appearing in a column like the first one I showed?

If you're using Python, here's some code to reproduce my DataFrame:

import pandas as pd
df = pd.DataFrame({'Names': ['Alice, Bob', 'Alice, Bob, Dave', 'Dave']})

Edit: the ultimate purpose of this is to then pass this through a tree-based classifier.

$\endgroup$
1
$\begingroup$

The binary encoding seems to be the most natural, capturing exactly the present relationships and not introducing nonexistent relationships. The combinations of names then are thought of as vertices of a hypercube; naturally, if you collapse this down to one dimension, you will be introducing nonexistent relationships. Indeed, this is what happens with label encoding: Bob isn't really any closer to Alice than Dave is, but your label encoding makes that appear to be the case.

I've usually been happy enough with just the binary encoding, so I don't have much to back up the following, but some ideas:

For your case, you could still project the hypercube to one dimension; there are any number of ways to do it, and every one will add some additional information that shouldn't be there. You could experiment and try to find one that seems most suitable for your purposes.

That should sound reminiscent of word embedding, and suggests another method: use a neural network to generate "entity embeddings", a projection from your hypercube to some smaller-dimensional space. In this way, you uncover combinations of persons that are useful for your problem (maybe you end up with a feature that's "at least one of Alice and Dave", etc.). You could also do something more transparent, and do some feature engineering by hand to pick out interesting combinations and binning to keep the number of features manageable. For this you could consider target encoding each combination. (Indeed, that's a very easy solution by itself, though target encoding can easily lead to overfitting if you have very many people or some rare combinations of people.)

Finally, you might be able to split a row containing (Alice, Dave) into two, one with Alice and the other with Dave. You can now label-encode (or whatever else), and will need either a model that understands multiple rows per sample point, or a postprocessing method to deal with combining rows.

$\endgroup$
0
$\begingroup$

I think, it depends on how many unique values do you have and what do you want to do with this data next. Intermediate table seems good solution for now, but you're expecting overhead. So maybe it would be suitable for you to create bit masks in one-hot manner (Binary encoding).

$\endgroup$
7
  • $\begingroup$ Thanks for your answer. How would binary encoding work on this problem? $\endgroup$ Oct 3 '18 at 13:39
  • $\begingroup$ @marco_gorelli since you will use tree-based classifier I suggest you to visit this link $\endgroup$
    – GrozaiL
    Oct 3 '18 at 14:48
  • $\begingroup$ Thanks - I'd already seen that link (from another answer on this site), but it only explains how binary encoding works when each row has only one value. In my example, rows may take on multiple different values (e.g. 'Alice, Bob' as opposed to just 'Alice' or 'Bob') $\endgroup$ Oct 3 '18 at 14:55
  • $\begingroup$ Assume you have Alice, Bob and Dave. Then you binary encode them so that for example: Alice = 001, Bob = 010, Dave = 100. Then for Alice and Bob you have bit vector 011, Alice and Dave = 101, for all of them: 111. And that works as some kind of hash. So you can have some kind of number instead of table since it's bits. $\endgroup$
    – GrozaiL
    Oct 3 '18 at 15:01
  • $\begingroup$ Ok, but if I then end up assigning a different number to possible row value, how is this different from applying numeric encoding straight away (so Alice and Bob is 0, Alice and Dave 1, all of them 2)? $\endgroup$ Oct 3 '18 at 15:06
0
$\begingroup$

To reduce the number of columns produced by onehot encoding one solution might be target encoding. That is, if you have the following table :

   Alice  Bob  Dave Other_features Target
0      1    1     0             X1      0
1      1    1     0             X2      1
2      1    1     0             X3      0
3      1    1     1             X4      1
4      1    1     1             X5      1
5      1    1     1             X6      1
6      0    0     1             X7      0
7      0    0     1             X8      0

You can create a table :

Group               Group_Avg_target
Alice,Bob                      0.333
Alice, Bob, Dave                   1
Dave                               0

Then replace the values in your dataset :

   Group_Avg_target Other_features Target
0             0.333             X1      0
1             0.333             X2      1
2             0.333             X3      0
3                 1             X4      1
4                 1             X5      1
5                 1             X6      1
6                 0             X7      0
7                 0             X8      0

That way you'll end up with one column. That mainly works if you have enough instances to have meaningfull averages by groups of people. I would not recommand using that in general as you would lose a lot of interactions betweeen people, which might be bad to explain what happened.

Note : to use this solution you need to make sure that there is no 'target leak', ie use the average calculated on train to encode your test dataset.

$\endgroup$
-1
$\begingroup$

First, do the OneHotEncoder thing with your Name column which will give you three columns as you have stated above, Namely Alice, Bob, and Dave.

We have this Table after doing OneHotEncoding :

    Alice  Bob  Dave
0   1      1    0     
1   1      1    1   
2   0      0    1     

Now you can call apply function on these columns; pass axis=1 to use the apply function row-wise, then convert the dtype to str and then join them as follows (astype function converts the input into the give data-type):

df['AliceBobDave']= df[df.columns[0:]].apply(lambda x: ','.join(x.dropna().astype(int).astype(str)),axis=1)

The Above will give you output :

     Alice  Bob  Dave  AliceBobDave
0    1      1    0     1,1,0
1    1      1    1     1,1,1
2    0      0    1     0,0,1

Now run this command below to remove the columns not needed in the final dataframe. For ex. in this case we remove Alice, Bob, and Dave using the drop function in pandas:

df = df.drop(['Alice','Bob','Dave'], axis = 1)

After executing this command if we print df using print(df), we get:

    AliceBobDave
0   1,1,0
1   1,1,1
2   0,0,1

This is your desired output.

$\endgroup$
4
  • $\begingroup$ Thanks for your answer, Pravan. However, this isn't what I'm looking for, as the row values we get using this method are strings, and are no more useful to how the rows appeared in my original DataFrame. $\endgroup$ Oct 3 '18 at 13:38
  • $\begingroup$ You're welcome. $\endgroup$ Oct 3 '18 at 14:38
  • $\begingroup$ Perhaps what you can do instead is In the line of code where we join the columns you can replace the joining argument , with empty string, and this will give you values like 110 etc. In each row and then you can change the data type of the whole column to integer type by using astype function. $\endgroup$ Oct 3 '18 at 14:42
  • $\begingroup$ Ok, but then it would be no different to directly applying a label encoder. What I'm looking for is a way of capturing common values between rows, whilst not creating too many new columns. $\endgroup$ Oct 3 '18 at 14:45
-1
$\begingroup$

See my comments on @GrozaiL's post for my real thoughts on this (use some form of binary encoding, ideally one-hot columns), but if you're really dying for a method to assign values to subsets, why not a Godel Numbering? Assign each person a prime number and multiply them together, that'll give you one number for each subset and if you ever want to figure out which people were in the subset after the fact, you can just use the prime factorization of the assigned number to recover its membership! (Don't do this)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.