# How to connect data-mining with machine learner process

I want to write a data-mining service in Google Go which collects data through scraping and APIs.

However as Go lacks good ML support I would like to do the ML stuff in Python.

Having a web background I would connect both services with something like RPC but as I believe that this is a common problem in data science I think that there is some better solution.

For example most (web) protocols lack at:

• buffering between processes
• clustering over multiple instances

So what (type of libraries) do data scientists use to connect different languages/processes?

Bodo

• One solution which goes in the direction I am looking for is fluentd. – bodokaiser Dec 3 '14 at 16:36
• Are you interested only in libraries available within Google Go? – Hack-R Dec 5 '14 at 16:38
• @Hack-R if it is a more complex protocol which requires some heavy logic I would prefer that a library would be available in Go but I would even more prefer if the library would be a available for other languages too. What do you think of a message queue like nsq. – bodokaiser Dec 7 '14 at 11:22

The Data Science Toolkit is a powerful library (or collection of libraries, technically) which are available in a number of languages. For instance, I use the implementation called RDSTK in R.

In the case of your preferred language, Google Go, there's a list of web-related libraries here which looks very useful.

• The Data Science Toolkit is very interesting but not what I am looking for. I am looking for some high performant stream based protocol which allows me to stream (and buffer) data from n data-miners to m data-processors. – bodokaiser Dec 7 '14 at 11:21

If your only motivation for using Google Go is webscraping, and you want to do you ML in python, I would recommend the following stack:

Python requests for scraping data

MongoDB for caching data (MongoDB's page oriented format makes it a natural home for storing JSON objects commonly returned by APIs)

pymongo for interfacing python and mongodb

scikit-learn for doing your machine learning

This all happens in python and you can extend it multiple processors with multiprocessing or to multiple nodes with django

• This is a pure python solution? – bodokaiser Dec 7 '14 at 11:23
• No, using mongodb for caching. I think Mongodb is written in Java if that's what you mean? – rawkintrevo Dec 13 '14 at 16:40

I am not 100% if a message queue library will be the right tool for this job but so far it looks to me so.

With a messaging library like:

You can connect different processes operating on different environment through a TCP based protocol. As these systems run distributed it is possible to connect multiple nodes.

For nsq we even have a library in Python and Go!