Questions tagged [performance]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
4
votes
3answers
299 views

What technologies are fastest at performing joins on large datasets?

By "large", I mean in the range of 100m to 10b rows. I'm currently using both Hadoop MapReduce and Amazon RedShift. MapReduce has been a little disappointing here. Redshift works very well if the ...
4
votes
1answer
2k views

Scikit Learn Logistic Regression Memory Leak

I'm curious if anyone else has run into this. I have a data set with about 350k samples, each with 4k sparse features. The sparse fill rate is about 0.5%. The data is stored in a ...
12
votes
3answers
273 views

Measuring performance of different classifiers with different sample sizes

I'm currently using several different classifiers on various entities extracted from text, and using precision/recall as a summary of how well each separate classifier performs across a given dataset. ...
5
votes
2answers
766 views

How to speedup message passing between computing nodes

I'm developing a distributed application, and as it's been designed, there'll be a great load of communication during the processing. Since the communication is already as much spread along the entire ...
8
votes
3answers
122 views

How to compare experiments run over different infrastructures

I'm developing a distributed algorithm, and to improve efficiency, it relies both on the number of disks (one per machine), and on an efficient load balance strategy. With more disks, we're able to ...
25
votes
4answers
13k views

Is there a straightforward way to run pandas.DataFrame.isin in parallel?

I have a modeling and scoring program that makes heavy use of the DataFrame.isin function of pandas, searching through lists of facebook "like" records of ...
10
votes
4answers
187 views

Why is it hard to grant efficiency while using libraries?

Any small database processing can be easily tackled by Python/Perl/... scripts, that uses libraries and/or even utilities from the language itself. However, when it comes to performance, people tend ...
13
votes
1answer
188 views

When a relational database has better performance than a no relational

When a relational database, like MySQL, has better performance than a no relational, like MongoDB? I saw a question on Quora other day, about why Quora still uses MySQL as their backend, and that ...
6
votes
1answer
513 views

What is the Best NoSQL backend for a mobile game

What is the best noSQL backend to use for a mobile game? Users can make a lot of servers requests, it needs also to retrieve users' historical records (like app purchasing) and analytics of usage ...
87
votes
12answers
13k views

How big is big data?

Lots of people use the term big data in a rather commercial way, as a means of indicating that large datasets are involved in the computation, and therefore potential solutions must have good ...