2
$\begingroup$

Let's assume we have data instances like this:

[
    [15, 20, ("banana","apple","cucumber"), ...],
    [91, 12, ("orange","banana"), ...],
    ...
]

I am wondering how I can encode the third element of these datapoints. For multiple features values we could use sklearn's OneHotEncoder, but as far as I could find out, it cannot handle inputs of different length.

Here is what I've tried out:

X = [[15, 20, ("banana","apple","cucumber")], [91, 12, ("orange","banana")]]

ct = ColumnTransformer(
    [
        ("genre_encoder", OneHotEncoder(), [2])
    ],
    remainder='passthrough'
)
print(ct.fit_transform(X))

This will only output

[[1.0 0.0 15 20]
 [0.0 1.0 91 12]]

as expected, because the tuples are handled as the possible values this feature can be represented with.

We can't embed our features directly (like [15, 12, "banana", "apple", "cucumber"]), because

  1. we don't know how many instances of this feature we will have (two? three?)
  2. each position would be interpreted as an own feature and thus if we had banana in the first nominal slot in one datapoint and in the second one in our second nominal slot, they would not count to the same "pool of values" a feature can embody

Example:

X = [["banana","apple","cucumber"], ["orange","banana", "cucumber"]]
enc = OneHotEncoder()
print(enc.fit_transform(X).toarray())

[[1. 0. 1. 0. 1.]
 [0. 1. 0. 1. 1.]]

This representation contains 5 slots instead of 4, because the first slot is interpreted as using banana or orange, the second one as apple or banana and the last one only has the option cucumber.

(This would also not solve the problem of having different amounts of feature values per datapoint. And replacing empty ones with None does not solve the problem either, because then None faces this positional ambiguity.)

Any idea how to encode those "Multi-Muliti-"features, that can take multiple values and consist of a varying amount of elements? Thank you in advance!

$\endgroup$
2
  • $\begingroup$ you can create a superset of all unique fruits and augment all sets with the missing fruits from each set (from the superset) and then one-hot encode the completed lists $\endgroup$
    – Nikos M.
    Commented Jan 29, 2021 at 18:12
  • $\begingroup$ @NikosM. thank you, this would work fine as well (Although I was trying to find a solution that would not need a complete new implementation - but I did not state that clear enough, sry) $\endgroup$ Commented Jan 30, 2021 at 13:37

1 Answer 1

3
$\begingroup$

I think you can transform this into a text preprocessing problem and then use CountVectorizer. You basically build "documents" by putting together all the words in your raw data and then use CountVectorizer on those documents.

from sklearn.feature_extraction.text import CountVectorizer

X = [["banana","apple","cucumber"], ["orange","banana", "cucumber"]]
# Create documents
X_ = [' '.join(x) for x in X]
enc = CountVectorizer()
print(enc.fit_transform(X_).toarray())

Returns

[[1 1 1 0]
 [0 1 1 1]]

which has 4 different values as you expected.

$\endgroup$
1
  • $\begingroup$ This is indeed working as expected, thank you! I implemented it into an own Encoder so that I could use it in the ColumnTransformer. It is not as perfect as I would have hoped (for example, I will have names consisting of two strings instead of one word long tokens), but maybe there is no better solution. $\endgroup$ Commented Jan 30, 2021 at 13:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.