8
$\begingroup$

I want to write a data-mining service in Google Go which collects data through scraping and APIs.

However as Go lacks good ML support I would like to do the ML stuff in Python.

Having a web background I would connect both services with something like RPC but as I believe that this is a common problem in data science I think that there is some better solution.

For example most (web) protocols lack at:

  • buffering between processes
  • clustering over multiple instances

So what (type of libraries) do data scientists use to connect different languages/processes?

Bodo

$\endgroup$
3
  • $\begingroup$ One solution which goes in the direction I am looking for is fluentd. $\endgroup$ – bodokaiser Dec 3 '14 at 16:36
  • $\begingroup$ Are you interested only in libraries available within Google Go? $\endgroup$ – Hack-R Dec 5 '14 at 16:38
  • $\begingroup$ @Hack-R if it is a more complex protocol which requires some heavy logic I would prefer that a library would be available in Go but I would even more prefer if the library would be a available for other languages too. What do you think of a message queue like nsq. $\endgroup$ – bodokaiser Dec 7 '14 at 11:22
4
$\begingroup$

The Data Science Toolkit is a powerful library (or collection of libraries, technically) which are available in a number of languages. For instance, I use the implementation called RDSTK in R.

In the case of your preferred language, Google Go, there's a list of web-related libraries here which looks very useful.

$\endgroup$
1
  • $\begingroup$ The Data Science Toolkit is very interesting but not what I am looking for. I am looking for some high performant stream based protocol which allows me to stream (and buffer) data from n data-miners to m data-processors. $\endgroup$ – bodokaiser Dec 7 '14 at 11:21
2
$\begingroup$

If your only motivation for using Google Go is webscraping, and you want to do you ML in python, I would recommend the following stack:

Python requests for scraping data

MongoDB for caching data (MongoDB's page oriented format makes it a natural home for storing JSON objects commonly returned by APIs)

pymongo for interfacing python and mongodb

scikit-learn for doing your machine learning

This all happens in python and you can extend it multiple processors with multiprocessing or to multiple nodes with django

$\endgroup$
2
  • $\begingroup$ This is a pure python solution? $\endgroup$ – bodokaiser Dec 7 '14 at 11:23
  • $\begingroup$ No, using mongodb for caching. I think Mongodb is written in Java if that's what you mean? $\endgroup$ – rawkintrevo Dec 13 '14 at 16:40
1
$\begingroup$

I am not 100% if a message queue library will be the right tool for this job but so far it looks to me so.

With a messaging library like:

You can connect different processes operating on different environment through a TCP based protocol. As these systems run distributed it is possible to connect multiple nodes.

For nsq we even have a library in Python and Go!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.