2
$\begingroup$

First of all, I hope I'm in the right StackExchange here. If not, apologies!

I'm currently working with huge amounts of feature-value vectors. There are millions of these vectors (up to 20 million possibly). They contain some linguistic/syntactic features and their values are all strings.

Because most classifiers do not handle string data as values, I convert them to binary frequency values, so an attribute looks like this:

@attribute 'feature#value' numeric

And per row, the value is either 1 or it is absent (so note it's a sparse ARFF file).

The thing is, with 250K rows, there are over 500K attributes and so, most algorithms have a hard time with this.

There are a lot of algorithms. I'm really curious as to what you would consider a suitable one (preferably unsupervised, but anything works), and if you even have some ideas how I could improve performance. I can train on small subsets of data, but the results only get better when using large amounts of data (at least 7 million events).

For now, I've been using NaiveBayes variations (Multinomial and also DMNBText) and those are really the only ones that are able to chew up data with acceptable speed.

Thanks a lot. If you need more information, please let me know.

Cheers.

$\endgroup$
2
$\begingroup$

I would go for dimensionality reduction. You can start with SVD (should be available in Weka). If SVD is too slow / too memory consuming, then there are still some options:

That's the tip of an iceberg. More approaches are there in the wild. The problem is that I doubt any of these solutions come with Weka (please, correct me if I am wrong on this). I would search for a usable Java implementation of any of these algorithms and try to port it to work with Weka's arff files.

$\endgroup$
  • 1
    $\begingroup$ Thank you. I think I have no time to execute these kinds of steps. My solution is to request access to a more powerful machine on my University. I'll mark your answer as the solution anyway. $\endgroup$ – lennyklb May 6 '15 at 12:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.