Let's say I have a dataframe where some of the columns have lists of strings as values. I would like to use ML Algorithms on this dataframe.
In this case, I can:
- I could add many columns of 1's and 0's for each string that may appear in some list, which seems to me terribly inefficient as some of the items in the lists only appear very few times. If there were maximum of 10 different items that could be listed in all of the lists , I guess it's quite all right to use/
- I know some use SVD in these cases (even though I don't completely understand it yet.)
- could put inside the dataframe pandas series as values like some kind of vectors.
So I ask:
- what is the best way to feature engineer it assuming there are few items that could appear in the lists?
- what is the best way to feature engineer it assuming there are lots of items that could appear in the lists?
- Is there a certain way to do it that runs most efficiently with the popular ML\DL python packages?