14
$\begingroup$

I'm trying to understand how all the "big data" components play together in a real world use case, e.g. hadoop, monogodb/nosql, storm, kafka, ... I know that this is quite a wide range of tools used for different types, but I'd like to get to know more about their interaction in applications, e.g. thinking machine learning for an app, webapp, online shop.

I have vistors/session, transaction data etc and store that; but if I want to make recommendations on the fly, I can't run slow map/reduce jobs for that on some big database of logs I have. Where can I learn more about the infrastructure aspects? I think I can use most of the tools on their own, but plugging them into each other seems to be an art of its own.

Are there any public examples/use cases etc available? I understand that the individual pipelines strongly depend on the use case and the user, but just examples will probably be very useful to me.

Improve this question
$\endgroup$
2
  • $\begingroup$ Did you make any research about this? There are many youtube videos and slideshare presentations describing different architectures $\endgroup$
    – Stanpol
    Jun 17, 2014 at 10:53
  • 1
    $\begingroup$ Hey Stanpol, thanks for your response - I did some initial searches and didn't really find anything besides AWS and cloudera stuff - maybe if you can give me some search terms that a promising, I'll be happy to take it from there. $\endgroup$
    – chrshmmmr
    Jun 17, 2014 at 11:28

4 Answers 4

Reset to default
15
$\begingroup$

In order to understand the variety of ways machine learning can be integrated into production applications, I think it is useful to look at open source projects and papers/blog posts from companies describing their infrastructure.

The common theme that these systems have is the separation of model training from model application. In production systems, model application needs to be fast, on the order of 100s of ms, but there is more freedom in how frequently fitted model parameters (or equivalent) need to be updated.

People use a wide range of solutions for model training and deployment:

Improve this answer
$\endgroup$
0
8
$\begingroup$

One of the most detailed and clear explanations of setting up a complex analytics pipeline is from the folks over at Twitch.
They give detailed motivations of each of the architecture choices for collection, transportation, coordination, processing, storage, and querying their data.
Compelling reading! Find it here and here.

Improve this answer
$\endgroup$
7
  • $\begingroup$ That's pretty awesome, exactly what I was looking for! Thanks a lot :) $\endgroup$
    – chrshmmmr
    Jun 17, 2014 at 13:35
  • $\begingroup$ @chrshmmmr You're welcome. Don't forget to upvote/mark as accepted if this helped! $\endgroup$
    – tchakravarty
    Jun 17, 2014 at 13:37
  • 3
    $\begingroup$ These links seem very useful indeed, but then again, they are links, and I guess we should strive to maintain the answers independent of the stability of outer sources. Thus, it'd be nice if you could take some two or three minutes to add, for example, the diagram from this link, posting it along with a quick description. Something in the lines of: "For example, this is the workflow of a ... system. <img>. Further info may be found in <link>." $\endgroup$
    – Rubens
    Jun 17, 2014 at 13:40
  • 1
    $\begingroup$ @Rubens I will propose an edit in a bit. fgnu: Will do so, just need a bit more reputation to actually upvote answers, but I certainly will honor your contribution :) $\endgroup$
    – chrshmmmr
    Jun 17, 2014 at 13:42
  • $\begingroup$ @Rubens That would be no more than reproducing the information at the link. I would if there were something I felt would add to the explanation already given there. $\endgroup$
    – tchakravarty
    Jun 17, 2014 at 13:53
4
$\begingroup$

Airbnb and Etsy both recently posted detailed information about their workflows.

Improve this answer
$\endgroup$
2
$\begingroup$

Chapter 1 of Practical Data Science with R (http://www.manning.com/zumel/) has a great breakdown of the data science process, including team roles and how they relate to specific tasks. The book follows the models laid out in the chapter by referencing which stages/personnel this or that particular task would be performed by.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.