Optimizing Weka for large data sets

Question

First of all, I hope I'm in the right StackExchange here. If not, apologies!

I'm currently working with huge amounts of feature-value vectors. There are millions of these vectors (up to 20 million possibly). They contain some linguistic/syntactic features and their values are all strings.

Because most classifiers do not handle string data as values, I convert them to binary frequency values, so an attribute looks like this:

@attribute 'feature#value' numeric

And per row, the value is either 1 or it is absent (so note it's a sparse ARFF file).

The thing is, with 250K rows, there are over 500K attributes and so, most algorithms have a hard time with this.

There are a lot of algorithms. I'm really curious as to what you would consider a suitable one (preferably unsupervised, but anything works), and if you even have some ideas how I could improve performance. I can train on small subsets of data, but the results only get better when using large amounts of data (at least 7 million events).

For now, I've been using NaiveBayes variations (Multinomial and also DMNBText) and those are really the only ones that are able to chew up data with acceptable speed.

Thanks a lot. If you need more information, please let me know.

Cheers.

Answer 1

I would go for dimensionality reduction. You can start with SVD (should be available in Weka). If SVD is too slow / too memory consuming, then there are still some options:

• CUR-decomposition: a variant of singular-value decomposition that keeps the matrices of the decomposition sparse if the original matrix is sparse (see: this chapter of Mining Massive Datasets book)

• Random projections: projecting the data onto a random lower-dimensional subspace (see: the Random projection in dimensionality reduction: Applications to image and text data paper)

• Coresets: Given a matrix A, a coreset C is defined as a weighted subset of rows of A such that the sum of squared distances from any given k-dimensional subspace to the rows of A is approximately the same as the sum of squared weighted distances to the rows in C see the Dimensionality Reduction of Massive Sparse Datasets Using Coresets paper)

That's the tip of an iceberg. More approaches are there in the wild. The problem is that I doubt any of these solutions come with Weka (please, correct me if I am wrong on this). I would search for a usable Java implementation of any of these algorithms and try to port it to work with Weka's arff files.