I have 2 data files: the first one is a database, potentially very large; the second one contains queries I want to answer. My program pipeline is processing the database to get some information first, and then use that same information to answer the queries. Although the number of queries is not big, processing each query takes a long time. So I want to give each worker some queries to answer, then combine all the answers together into one.
This sounds like a MapReduce job. But to answer each query, the worker also needs to use the processed information from the database, and I'm not sure how to do this with MapReduce. I'm new to parallel programming and just heard of MPI and Spark.
Can you help me to choose an appropriate one?
Please ask if the description is not clear.