# MPI, MapReduce, or Spark for complex datasets and processing

I have 2 data files: the first one is a database, potentially very large; the second one contains queries I want to answer. My program pipeline is processing the database to get some information first, and then use that same information to answer the queries. Although the number of queries is not big, processing each query takes a long time. So I want to give each worker some queries to answer, then combine all the answers together into one.

This sounds like a MapReduce job. But to answer each query, the worker also needs to use the processed information from the database, and I'm not sure how to do this with MapReduce. I'm new to parallel programming and just heard of MPI and Spark.

Can you help me to choose an appropriate one?

The first thing first, if you want to run MapReduce programs on your data, you need to distribute your data. In addition, MapReduce doesn't run your queries to different nodes separately. If your $query_{A}$ runs on $data_{A}$, what we do here is to divide $data_{A}$ to different nodes and run $query_{A}$ on these nodes on $PartOfData_{A}$ in parallel. By the way, you can use Sqoop to migrate the data from your DB to HDFS(it is used to store data separately).