# Optimizing Weka for large data sets

First of all, I hope I'm in the right StackExchange here. If not, apologies!

I'm currently working with huge amounts of feature-value vectors. There are millions of these vectors (up to 20 million possibly). They contain some linguistic/syntactic features and their values are all strings.

Because most classifiers do not handle string data as values, I convert them to binary frequency values, so an attribute looks like this:

@attribute 'feature#value' numeric


And per row, the value is either 1 or it is absent (so note it's a sparse ARFF file).

The thing is, with 250K rows, there are over 500K attributes and so, most algorithms have a hard time with this.

There are a lot of algorithms. I'm really curious as to what you would consider a suitable one (preferably unsupervised, but anything works), and if you even have some ideas how I could improve performance. I can train on small subsets of data, but the results only get better when using large amounts of data (at least 7 million events).

For now, I've been using NaiveBayes variations (Multinomial and also DMNBText) and those are really the only ones that are able to chew up data with acceptable speed.