5
$\begingroup$

By "large", I mean in the range of 100m to 10b rows.

I'm currently using both Hadoop MapReduce and Amazon RedShift. MapReduce has been a little disappointing here. Redshift works very well if the data is distributed well for the given query.

Are there other technologies that I should be looking at here? If so, what are the trade offs?

Improve this question
$\endgroup$
3
  • $\begingroup$ I think this might be a bit broad, and maybe better suited for StackOverflow. $\endgroup$
    – Sean Owen
    Nov 11, 2014 at 11:29
  • $\begingroup$ MapReduce is not really a technology for joining. Do you mean Hive? Impala is the closest analog to Redshift and things like Teradata $\endgroup$
    – Sean Owen
    Jun 14, 2015 at 9:39
  • $\begingroup$ consider monetdb $\endgroup$
    – Anthony Damico
    Jun 14, 2015 at 12:44

3 Answers 3

Reset to default
4
$\begingroup$

More importantly than the technology is the type of join you are using. For instance if the join keys are sorted, you can use sort merge joins and use join orders to get a better performance.

That being said, you can use in memory solutions for fastest joins if the size of your intermediate results will not blow up your cluster memory. Look at Spark SQL or Mem-SQL for instance.

Improve this answer
$\endgroup$
0
$\begingroup$

If you are willing to pay for a vendor solution, Teradata is designed to solve the problem of large scale joins with low latency.

Improve this answer
$\endgroup$
1
  • $\begingroup$ If price is no object you could check out Netezza... $\endgroup$
    – rawkintrevo
    Dec 5, 2014 at 23:27
0
$\begingroup$

I would say Teradata should be good choice. However, when you make join you have to take careful consideration of choosing right joining keys.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.