1
$\begingroup$

I have a dataset

Inp1    Inp2        Inp3               Output
A,B,C   AI,UI,JI    Apple,Bat,Dog      Animals
L,M,N   LI,DO,LI    Lawn, Moon, Noon   Noun
X,Y     AI,UI       Yemen,Zombie       Extras

For these values, I need to apply a ML algorithm. Hence need an encoding technique. Tried a Label encoding technique, it encodes the entire cell to an int for eg.

Inp1    Inp2    Inp3    Output
5       4       8       0

But I need a separate encoding for each value in a cell. How should I go about it.

Inp1     Inp2    Inp3      Output
7,44,87  4,65,2  47,36,20  45

Integers are random here.

Improve this question
$\endgroup$
5
  • $\begingroup$ Welcome to DataScienceSE. I don't use pandas so I can't help with the details, but apparently your features are lists of categorical variables. The standard way would be to one-hot encode all these lists, i.e. replace each column with as many boolean features as there are possible values. You should probably discard the least frequent values, especially of the number is high. $\endgroup$
    – Erwan
    May 1, 2022 at 14:24
  • $\begingroup$ One-hot encode increases the number of columns, which is not desirable as there would be an issue of Dimensionality. $\endgroup$
    – spd
    May 1, 2022 at 14:30
  • $\begingroup$ Indeed, but if you keep encoding these lists as a single variable the ML model will certainly be unable to learn anything at all. The design of the features should be made to properly represent the semantic of the data and help the model find the relevant patterns. $\endgroup$
    – Erwan
    May 1, 2022 at 14:38
  • $\begingroup$ Yes, true. Thanks a lot. I am fairly new to this domain so trying to learn more. It would be great if you suggest any reference to go about this. $\endgroup$
    – spd
    May 1, 2022 at 16:20
  • $\begingroup$ This question would get better answers in Stackoverflow which is for programming questions. $\endgroup$
    – lpounng
    May 4, 2022 at 4:35

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.