Collapsing categorical data into more than 3 categories

Question

I have a bunch of categorical, part of speech data that I want to collapse into fewer categories. np.where() won't do because I want to have 6 categories at the end: noun, verb, adjective, adverb, preposition, and other.

I've found out that I can use pandas.replace() in combination with a dictionary to do this.

So, I've made the following dictionary:

mappings = {"NN" : "noun", "NNS" : "noun", "NNP" : "noun",
            "VB" : "verb", "VBD" : "verb", "VBG" : "verb", "VBN" : "verb", "VBP" : "verb", "VBZ" : "verb",
            "JJ" : "adj", "JJR" : "adj", "JJS" : "adj",
            "RB" : "adv", "RBR" : "adv", "RBS" : "adv",
            "IN" : "prep"}

The problem is, there are a LOT more parts of speech present in the data. Is there a way for me to shove all of those other parts of speech into an "other" category, or will I have to manually type in all of the other possible parts of speech?

Answer 1

You could use numpy select function

You would need to adapt but it would be something like this:

nouns = ["NN","NNS","NNP"]
verbs = ["VB","VBD","VBG","VBN","VBP","VBZ"]
adjs = ["JJ","JJR","JJS"]
advs = ["RB","RBR","RBS"]
preps = ["IN"]

condlist = [
           df.my_colum.isin(nouns),
           df.my_colum.isin(verbs),
           df.my_colum.isin(adjs),
           df.my_colum.isin(advs),
           df.my_colum.isin(preps),
           ]


choicelist = ["noun","verb","adj","adv","prep"]


df["gruop"] = np.select(condlist= condlist, choicelist= choicelist, default = "other")