4
$\begingroup$

By "large", I mean in the range of 100m to 10b rows.

I'm currently using both Hadoop MapReduce and Amazon RedShift. MapReduce has been a little disappointing here. Redshift works very well if the data is distributed well for the given query.

Are there other technologies that I should be looking at here? If so, what are the trade offs?

$\endgroup$
  • $\begingroup$ I think this might be a bit broad, and maybe better suited for StackOverflow. $\endgroup$ – Sean Owen Nov 11 '14 at 11:29
  • $\begingroup$ MapReduce is not really a technology for joining. Do you mean Hive? Impala is the closest analog to Redshift and things like Teradata $\endgroup$ – Sean Owen Jun 14 '15 at 9:39
  • $\begingroup$ consider monetdb $\endgroup$ – Anthony Damico Jun 14 '15 at 12:44
4
$\begingroup$

More importantly than the technology is the type of join you are using. For instance if the join keys are sorted, you can use sort merge joins and use join orders to get a better performance.

That being said, you can use in memory solutions for fastest joins if the size of your intermediate results will not blow up your cluster memory. Look at Spark SQL or Mem-SQL for instance.

$\endgroup$
0
$\begingroup$

If you are willing to pay for a vendor solution, Teradata is designed to solve the problem of large scale joins with low latency.

$\endgroup$
  • $\begingroup$ If price is no object you could check out Netezza... $\endgroup$ – rawkintrevo Dec 5 '14 at 23:27
0
$\begingroup$

I would say Teradata should be good choice. However, when you make join you have to take careful consideration of choosing right joining keys.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.