8
$\begingroup$

I want to write a data-mining service in Google Go which collects data through scraping and APIs.

However as Go lacks good ML support I would like to do the ML stuff in Python.

Having a web background I would connect both services with something like RPC but as I believe that this is a common problem in data science I think that there is some better solution.

For example most (web) protocols lack at:

  • buffering between processes
  • clustering over multiple instances

So what (type of libraries) do data scientists use to connect different languages/processes?

Bodo

Improve this question
$\endgroup$
3
  • $\begingroup$ One solution which goes in the direction I am looking for is fluentd. $\endgroup$
    – bodokaiser
    Commented Dec 3, 2014 at 16:36
  • $\begingroup$ Are you interested only in libraries available within Google Go? $\endgroup$
    – Hack-R
    Commented Dec 5, 2014 at 16:38
  • $\begingroup$ @Hack-R if it is a more complex protocol which requires some heavy logic I would prefer that a library would be available in Go but I would even more prefer if the library would be a available for other languages too. What do you think of a message queue like nsq. $\endgroup$
    – bodokaiser
    Commented Dec 7, 2014 at 11:22

3 Answers 3

Reset to default
4
$\begingroup$

The Data Science Toolkit is a powerful library (or collection of libraries, technically) which are available in a number of languages. For instance, I use the implementation called RDSTK in R.

In the case of your preferred language, Google Go, there's a list of web-related libraries here which looks very useful.

Improve this answer
$\endgroup$
1
  • $\begingroup$ The Data Science Toolkit is very interesting but not what I am looking for. I am looking for some high performant stream based protocol which allows me to stream (and buffer) data from n data-miners to m data-processors. $\endgroup$
    – bodokaiser
    Commented Dec 7, 2014 at 11:21
2
$\begingroup$

If your only motivation for using Google Go is webscraping, and you want to do you ML in python, I would recommend the following stack:

Python requests for scraping data

MongoDB for caching data (MongoDB's page oriented format makes it a natural home for storing JSON objects commonly returned by APIs)

pymongo for interfacing python and mongodb

scikit-learn for doing your machine learning

This all happens in python and you can extend it multiple processors with multiprocessing or to multiple nodes with django

Improve this answer
$\endgroup$
2
  • $\begingroup$ This is a pure python solution? $\endgroup$
    – bodokaiser
    Commented Dec 7, 2014 at 11:23
  • $\begingroup$ No, using mongodb for caching. I think Mongodb is written in Java if that's what you mean? $\endgroup$
    – rawkintrevo
    Commented Dec 13, 2014 at 16:40
1
$\begingroup$

I am not 100% if a message queue library will be the right tool for this job but so far it looks to me so.

With a messaging library like:

You can connect different processes operating on different environment through a TCP based protocol. As these systems run distributed it is possible to connect multiple nodes.

For nsq we even have a library in Python and Go!

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.