2
$\begingroup$

The typical steps for solving a machine learning/pattern recognition problem:

  1. Data Analysis and splitting the data into test and train sets.
  2. Choosing a model.
  3. Training the model, and testing the model against the test set.
  4. If the accuracy of the model is not acceptable, start over with a new model.

Step (2) can be automated to some extent using a grid search, but are there ways of automating the whole process?

I'm thinking in particular of large scale data applications (for example when dealing with retail data, or customer analytics on a site like Netflix, ) where there are millions of instances of similar but distinct machine learning problems that each need to be trained and validated separately.

In such a situation it is impossible for a team of analysts or data scientists to perform the above steps and some sort of automated model development framework must be used.

What are the frameworks that allow for this?

$\endgroup$
1
$\begingroup$

I think, Rapidminer can be used for that. However, the free version can only load 10,000 rows of data.

$\endgroup$
1
$\begingroup$

There's a couple of tools that already do this. The ones that I'm familiar with are:

  1. IBM Watson Analytics
  2. Datarobot

Both of those frameworks will handle what you're looking for. Of course, they're not free, but they will do the work you seek.

$\endgroup$
0
$\begingroup$

I'd definitely recommend taking a look at the tooling/framework repository EthicalML put out - it lists a wealth of tools for productizing/scaling ML pipelines, broken up by categories and each with a few sentences description of what they do.

I'd imagine the Model & Data Versioning and some of the Commercial Platforms sections would be most germane to what you're looking for (the latter includes both options from I_Play_With_Data's answer).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.