1
$\begingroup$

I have an extensive series of projects that require importing data from many different types of different sources (websites, web apis, sensors, legacy text files, etc). Is there a good framework (preferably open source) that has been built bottom up to tackle this type of problem? Preferably one leveraging an appropriate language (python, most likely) and already having an extensive plugin database.

$\endgroup$
6
  • $\begingroup$ I don't think there is such a wide-ranging framework, at least not on the data-connecting side, but most languages will have libraries supporting the data sources you suggest. You need to define "appropriate language" - probably best to simply state which languages you would accept, or at least what specific qualities count as "appropriate". You might instead try softwarerecs.stackexchange.com (but do read their help, and do fix the vague parts of your question first - such as which data sources are actually important to you, and which OS/language to support). $\endgroup$ Aug 30, 2015 at 6:59
  • $\begingroup$ Or are you looking for the middle-ware here for co-ordinating data movement, something like Microsoft's SQL Server Integration Services (was called Data Translation Services)? $\endgroup$ Aug 30, 2015 at 7:14
  • $\begingroup$ It's for IoT / Data Science project. The sources of data are specified as web scraping, web apis (rest request replying with json, blue tooth sensor data, and legacy text files). It's likely there will be more, but they haven't been nailed down yet. Language would probably be python. The assumption of the team is that this has happened many times before and that some framework would already exist. We don't mind writing one and open sourcing it, but it'd be a shame to re-invent the wheel unnecessarily. $\endgroup$
    – Blaze
    Aug 30, 2015 at 10:44
  • $\begingroup$ I think you need to define better what this imagined framework would do. Pretty much all software that handles data will "wrangle" or "munge" it, so you have not given enough information. For instance, is the goal to get all the data into a format suitable for specific ML algorithms (usually a matrix or sparse matrix)? Do you need support for data sets larger than RAM allocation for a single algorithm? $\endgroup$ Aug 30, 2015 at 11:16
  • $\begingroup$ Well, we don't want to get too specific as it will eliminate potential solutions. As long as the data can be processed into some internal format that is fine. We can deal with it from there. $\endgroup$
    – Blaze
    Aug 30, 2015 at 18:57

2 Answers 2

1
$\begingroup$

If you are interested in a very high-level (enterprise architecture) framework, I suggest you to take a look at the MIKE2.0 Methodology. Being an information management framework, MIKE2.0 has, certainly, much wider coverage than the domain of your interest, but it is a solid, interesting and open (licensed under the Creative Commons Attribution License) framework. A better fit for your focus is the Extract, transform, load (ETL) framework, which is extremely popular in contexts of Business Intelligence and Data Warehousing. On a more practical note, you might want to check my answer on Quora on open source master data management (MDM) solutions. Pay attention to the Talend solutions (disclaimer: I am not affiliated with this or any company), which cover a wide spectrum of MDM, ETL and data integration domains as open source and commercial offerings.

$\endgroup$
0
$\begingroup$

All of the scientific computing development going on around Python is a good option. Continuum's Anaconda is very easy to get up and running. Also well supported in distributed environments.

  • Numpy for fast matrix operations
  • Pandas for dataframes, munging, analytics, visualization
  • Statsmodels and Scikit for models, metrics, ML algs and so on
  • ...
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.