3
$\begingroup$

Problem description

I have a data set about 10000 patients in a study. For each patient, I have a list of various measurements. Some information is scalar data (e.g. age), some information is time series of measurements, some other information can be even a bitmap. The individual record itself can be quite thick (10kB to 10MB). The data is to be processed practically in two steps:

  1. Preprocessing at the level of individual records (patients), i.e. to extract some features in raw data, store them. Calculate some slopes in time series etc. All this can be done on individual level and it can be very easily distributed.

  2. On top of the preprocessed data (extracted features), I will need to calculate some aggregated things such as e.g. average age, but also some machine learning tasks.

The question

Obviously, this is very suitable to be addressed in Apache Spark (or any map-reduce architecture). At the most general level, my question is: what is the most appropriate NoSQL database for this situation?

So far, I have considered two basic options:

  1. MongoDB - to take advantage of the document oriented storage where everything is on the same place. However, I am not sure about the performance on the larger binary data (pictures, time-series).
  2. Cassandra - this may have some better storage of binary data, but the joins will be necessary (even if optimized by indexing all data by "patient id").
$\endgroup$
3
$\begingroup$

I think you need to define the process and desired outcome a little more. It sounds like you need to:

  1. Define what features you want.
  2. Figure out how to extract those features.
  3. Figure out how to store those features.
  4. Pass that dataset to a ML model for training.

I would figure out what exactly you want for #3 in terms of the number and type of data elements and then choose a data storage method only then.

Unless you are going to be passing the unstructured documents directly to your model, you don't need MongoDB's capabilities. 10,000 records is small, but since you mentioned you want to calculate some aggregated statistics on your patient-level data, you could likely get by with something as simple as MySQL or SQLite.

Spark and Map/Reduce are actually competitors, with Spark stealing MR's spotlight lately. You might need one or the other for feature extraction but they're probably overkill for the rest of the stuff you described.

$\endgroup$
  • $\begingroup$ Thank you for your reply. The issue is that I do need the unstructured information for each patient (e.g. preprocessed image to disease classification). $\endgroup$ – Karel Macek Jun 5 '17 at 22:46
  • $\begingroup$ You need to keep it for what? Reference? If you aren't actively passing it to your ML algorithm, just store a pointer to it in a filesystem. $\endgroup$ – CalZ Jun 6 '17 at 11:16
  • $\begingroup$ I need it for the machine learning itself. $\endgroup$ – Karel Macek Jun 7 '17 at 8:26
  • $\begingroup$ Earlier you said you were extracting features from the thick records. Which is it? :) $\endgroup$ – CalZ Jun 7 '17 at 12:11
  • $\begingroup$ Both. There are various things to happen with the records. $\endgroup$ – Karel Macek Jun 7 '17 at 12:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.