0
$\begingroup$

I have 2 data files: the first one is a database, potentially very large; the second one contains queries I want to answer. My program pipeline is processing the database to get some information first, and then use that same information to answer the queries. Although the number of queries is not big, processing each query takes a long time. So I want to give each worker some queries to answer, then combine all the answers together into one.

This sounds like a MapReduce job. But to answer each query, the worker also needs to use the processed information from the database, and I'm not sure how to do this with MapReduce. I'm new to parallel programming and just heard of MPI and Spark.

Can you help me to choose an appropriate one?

Please ask if the description is not clear.

$\endgroup$
1
$\begingroup$

If I understand your question correctly. What you want to do is

  1. Write a sql query against to a database to get the processed/aggregated results first.
  2. Execute several queries in parallel.

The first thing first, if you want to run MapReduce programs on your data, you need to distribute your data. In addition, MapReduce doesn't run your queries to different nodes separately. If your $query_{A}$ runs on $data_{A}$, what we do here is to divide $data_{A}$ to different nodes and run $query_{A}$ on these nodes on $PartOfData_{A}$ in parallel. By the way, you can use Sqoop to migrate the data from your DB to HDFS(it is used to store data separately).

Here are steps for you:

  1. migrate all/part of your data to HDFS
  2. transform your queries to MapReduce version program
| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.