I'm trying to make text multilabel classification ~40 labels from products description. Labels are unbalanced. There are ~3 labels per sample. And I have ~250k samples.
I digged Kaggle's text classification competition from 2014 and it seems I'm using correct algorithm stack is: scikit-learn/one-vs-rest/tfidf (bigrams)/SGDClassifier (SVM)/f1 score (global - average='micro'). I was able to achieve 0.54 f1 score which for me is bearable as a first step. Now I run out of ideas so I want to ask you for some clues.
I tried so far:
- RandomForests - couldn't force them to achieve more than 0.30 f1 score and training time was so long...
- KNNs - I'm afraid that they won't be production ready, because of high prediction time
- stacking tfidf - PCA/SVD with 100-200 components didn't get significant boost when used with SVM (I must check RFs and KNNs)
- WordNet lemmatizer form - it gave some improvements, but it is slow overall
- GridSearch over parameters took a long time and didn't give a big improvement
In some moment I discovered that I can add class_weight=balanced, because when using one vs rest... and "rest" can become enormous.
I have problem:
- I can use at most of 30k data, because:
- LinearSVM is too slow to train with bigrams, but with more data it gives better results
- SGD-SVM is perfect in speed, but with more data it has higher much much recall (it simple returns more labels)
Do you know what's wrong with my approach to SGD-SVM?
Do you have some clues what is worth trying? I'm planning to use CNNs and RNNs with theano, but it will take some time. Maybe model ensemble?
What about infrastructure? My laptop is pretty efficient but I would like to set up local grid infrastructure (I was you using LFS job scheduler). Is there anything similar to for scikit-learn? Or maybe cloud: AWS/Azure/Google Cloud, what is the best (I think the cheapest).
How about environment? Jupyter + pycharm works pretty well, but I'm missing some features from RStudio (like showing all variables).