0
$\begingroup$

Consider a target vector like ["x", "x", "y", "z", "z", "z", "x", "y", "y", ...] containing the names ("x", "y", "z", ...) of some writers. For examples, assume that we have some quotes and we want to predict who has quoted that.

I want to encode these target names to numerical values. If I use frequency encoding, then, if two writers appear the same number of times in the list, they will get the same value. This is not what I want because in this case, a machine learning algorithm considers those two different persons (labels) as one person (label). If I use ordering then I will have a huge difference in the labels. Because I actually have about 80,000 different authors. "Get dummies" also doesn't work because generated data frame doesn't fit into memory. So what is the best way to encode this target column in a data frame?

My second question is, do you use multi-class classification for this task? As I said there are 80,000 different writers so there will be 80,000 labels and each label appears, in average, about 50 times in the list. Some labels appear only one time and some about 400 times!!. In general, what supervised learning algorithm will you use for a task like this?

$\endgroup$
0
$\begingroup$

To answer your first question, I think OneHotEncoder from category_encoders should be up to the job, just set return_df=False to get a numpy array which you could wrap around a sparse matrix. I don't see how that would lead to a memory error unless you were operating in a very limited space.

If that is the case maybe you could simply try LabelEncoder, though this may be a less optimal solution.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy