The typical steps for solving a machine learning/pattern recognition problem:
- Data Analysis and splitting the data into test and train sets.
- Choosing a model.
- Training the model, and testing the model against the test set.
- If the accuracy of the model is not acceptable, start over with a new model.
Step (2) can be automated to some extent using a grid search, but are there ways of automating the whole process?
I'm thinking in particular of large scale data applications (for example when dealing with retail data, or customer analytics on a site like Netflix, ) where there are millions of instances of similar but distinct machine learning problems that each need to be trained and validated separately.
In such a situation it is impossible for a team of analysts or data scientists to perform the above steps and some sort of automated model development framework must be used.
What are the frameworks that allow for this?