I am setting up for a couple self study projects to explore machine learning techniques.
1st project has 10,000 time series with 24 float data points each day for 10 years (876 million points). I will be creating a bunch of calendar and weather features for the data, then trying to forecast using a variety of machine learning techniques.
2nd is some 13 million rows of text data (several paragraphs for each row) for classification. (currently in solr database)
My compute rig is 6-core, 32g ram, gforce GPU. I plan to install Ubuntu 14.2.
I expect to be using python for file processing, scilearn, pylearn2 and word2vec for general exploration and training. R for getting a taste of the language.
Clearly data set 1 will require joining weather and calendar data to date/time and aggregation across time and location. I know how to stuff it all into a MySQL database and do the aggregations and joins there, but I have been reading about spark and wondering.
......
If I take the time to simulate a cluster using virtual box/hadoop/spark (for my learning experience, not performance), can/should I do the aggregations there and write the results to the distributed data store?
Since deep learning can not be run on spark, does that mean I would need to copy the aggregated data back out to the local file system to use some of those techniques?
For data set 2, I want to run the word2vec algorithm as found in the kaggle tutorial https://www.kaggle.com/c/word2vec-nlp-tutorial/details/part-2-word-vectors. In that example, that is a deeplearning method, so I should just leave the data in the solr.. right?
In general I am looking for appropriate applications and insight into data flow from app to app to help me get to the part where I start trying various ML techniques.
Thanks for helping me along