any reason for this project to use hadoop/spark?

Question

I am setting up for a couple self study projects to explore machine learning techniques.

1st project has 10,000 time series with 24 float data points each day for 10 years (876 million points). I will be creating a bunch of calendar and weather features for the data, then trying to forecast using a variety of machine learning techniques.

2nd is some 13 million rows of text data (several paragraphs for each row) for classification. (currently in solr database)

My compute rig is 6-core, 32g ram, gforce GPU. I plan to install Ubuntu 14.2.

I expect to be using python for file processing, scilearn, pylearn2 and word2vec for general exploration and training. R for getting a taste of the language.

Clearly data set 1 will require joining weather and calendar data to date/time and aggregation across time and location. I know how to stuff it all into a MySQL database and do the aggregations and joins there, but I have been reading about spark and wondering.

......

If I take the time to simulate a cluster using virtual box/hadoop/spark (for my learning experience, not performance), can/should I do the aggregations there and write the results to the distributed data store?

Since deep learning can not be run on spark, does that mean I would need to copy the aggregated data back out to the local file system to use some of those techniques?

For data set 2, I want to run the word2vec algorithm as found in the kaggle tutorial https://www.kaggle.com/c/word2vec-nlp-tutorial/details/part-2-word-vectors. In that example, that is a deeplearning method, so I should just leave the data in the solr.. right?

In general I am looking for appropriate applications and insight into data flow from app to app to help me get to the part where I start trying various ML techniques.

Thanks for helping me along

Answer 1

Especially when your main goal is learning, I would break it into several phases:

1. Get familiar with pandas dataframes and visualization using matplotlib. Try loading subsets of your datasets using pandas and visualize them using matplotlib, for example plot the time-series, or word count histograms. This will come in handy lateron to be able to understand the predictions/clusters you will create using machine learning methods. Pandas also provides functions to clean, aggregate and resample the data, which will be useful to align multiple time-series (e.g. weather and time-series) to the same sampling rate.

2. Familiarize with scikit-learn using the articles and examples in the excellent online documentation. Play with the feature extraction, dimensionality reduction, classification and regression methods relevant to your data, apply them on smaller subsets and visualize the results. Learn about cross-validation and find a good algorithm and parameters that work well on a subset of your data.

3. One you have a pipeline setup and you want to run it on the full dataset, Spark can come in handy for feature extraction and cross-validation, as those tasks can be often run very well independently and in parallel. Install the Spark libraries on your machine, and reproduce the Pyspark examples (e.g. calculating Pi) in local mode. No need to setup a Hadoop cluster for that, Spark can make full use of your machine in local mode, reading from local files on disk. Once you have that running, try to express the expensive steps of your code from step 2 as Spark RDD operations. As Spark can make use of all cores of your machine, you should already see a speedup - debugging becomes much harder though compared to working with pure Python scripts.

4. Play with advanced stuff like deep learning, using the word2vec feature extraction. Compare with traditional feature extraction (bag of words).

5. (optional) If you want to experience the speedup you can get from Spark in a distributed setting, get an account in AWS, put your data on S3 and fire your Spark scripts against a multi-node cluster instantiated using Amazon EMR.

Regarding data storage, from my experience CSV files or similar work best to load the data into both Pandas and Spark. There are Spark connectors for databases (e.g. Cassandra), but I am not sure about MySQL and Solr. For processed data and intermediate results (e.g. models) I would always use Python's serialization framework pickle, so you can directly get the object instances back into Python without the need to parse file formats and reinstantiate objects.

Answer 2

Actually deep learning can be run in spark using h2o sparkling water feature.Also you can use h2o.deeplearning to run deeplearning on your data in cluster or single node.Spark is good for munging the data in cluster as it does so distributedly but in memory,otherwise h2o has limited functions for data munging and that too it can't distribute data munging.