I have one dataset of "Books" which contains 8 columns initially and out of which 3 of them contains text values which can be categorized. The 3 columns contains "Language-code", "Author Name" and "title" of the book. As sklearn LinearRegression don't take text as input so i decided to categorize these 3 columns by using "pandas_getdummies(...)" but after categorizing it the columns number exceeded to 20072 from 8 which is way too high.

The dataset url is: https://www.kaggle.com/jealousleopard/goodreadsbooks/downloads/goodreadsbooks.zip/6

So my queries are:

  1. What to do with the title name? Categorizing it doesn't seems right.
  2. What to with the rest 2 columns? If i leave the title name then the number of columns exceeds to 7646. Is there any other algorithm where i can directly feed the dataset without categorization?
  3. How to handle these large number of features after categorizing?

Algorithm like Decision Tree, can also work well on ordinal values, i.e. without OneHotEncoding. You can try this one.

Also Im not sure the importance of Title Name, so you have to take decision based upon the requirements. But I have avoided in my use case.

Also, generally when you have large number of categories, you can give a try by clubbing uncommon categories into one.

  • $\begingroup$ That's a Regression problem so i don't think i can use Decision Tree in that and please if you explain a bit more about "Clubbing Uncommon categories" or can provide any example link for that? $\endgroup$ – sak Jul 15 '19 at 6:05
  • 1
    $\begingroup$ Decision Tree works for both regression and classification. Other part, consider a feature Animal, which have values like dogs, cat, rabbit, cow, ox, tiger, lion and so on. If you observe the occurrences of all categories except tiger and lion are high, then you can merge tiger and lion two-categories and make it single one and give some suitable name "wild-animal"(or something like that). Perform this with other categories as well. You have to identify threshold for minimum value. $\endgroup$ – vipin bansal Jul 15 '19 at 7:44
  • $\begingroup$ oh great, I will remember that for future. thanks. $\endgroup$ – sak Jul 15 '19 at 8:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.