What kind of classification should I use?

Question

I'm very new to machine learning (and statistics) and I'm struggling with some basics (I'm using R as my primary environment).

I have a lexicon with words and all of their forms (in different genders, persons - for a Slavic language). So each word with its word forms is one training sample. My training attributes are last character, last two characters, last three characters of each word form. So each training sample contains from around 100 categorical attributes. I've got 10000 samples. One attribute has around 1700 levels, one around 1200, 5 around 500, others have less than 100 levels.

I have used DBSCAN (and gower distance) to cluster similars words to the same cluster. This step was pretty successful. I get (depending whether is a noun, verb or adjective) from 40-100 clusters.

Now here comes the part where I don't know how to continue. If I get a new (unknown) word I want to predict to which cluster it belongs. Sometimes for this new word I only know its base form, sometimes I also part of its word forms and sometimes I know all of its word forms.

So this should be some kind of classification with multiple classes. Question: which algorithm should I use? I've tried to use multinom (in R) but I get an error of too many weights. After I increased MaxNWts I get an error:

weights:  641676 (633954 variable)
Error: cannot allocate vector of size 396 Kb

Where should I continue? Clean the data? Use a different algorithm?

Also, around 15 attributes have only 1 level or NA. How does this impact to clustering (and classification)? Is it safe to remove those attributes?

Many thanks in advance!

Answer 1

Using your input data as-is (many categorical attributes), you should start with random forests. The levels of your features probably aren't ordinal in any sense, which would explain why multinom (a linear model) didn't work. A decision tree (component of a random forest) won't have trouble with categorical features that aren't ordinal.

If your features are correlated (I'm guessing they are highly correlated), you might also try first reducing the dimensionality of your input dataset. A nice way to do this is to view your categorical features as "words" and map each input sample (a word in all its forms) to a sparse vector (training a TF-IDF model). Then, reduce the dimensionality of your sparse vectors using LSI or LDA (dense vectors). Train a classifier (again, random forest is a good starting point) using these dense vectors as input data.