Hot answers tagged

3

This answer some of your questions from a Python perspective: Is Python any faster? This question is a bit tricky to answer, it will depend on your usage of Python, but Python is not a fast language per se. However, the pandas library in Python have been reported to handle tables of 33M-100M rows, see this. I myself have used to handle around 10M rows ...


3

in my opinion, you have 4 possibilities: You may treat the pdf directly using tabula You may convert the pdf to text using pdftotext, then parse text with python You may use an external tool, to convert your pdf file to excel or CSV, then use required python module to open the excel/CSV file. You may also convert pdf to an image file, then use any recent ...


2

First of all I just want to say that I am not a data engineer and there is definitely someone out there that can answer this better than me. I do think that there is a lot of theory behind data engineering. It is also very interesting. I too thought that it was boring and I was more interested in just data science/ machine learning. I am not sure if I can ...


1

All depends of partitioning of the input table. Here is 2 approaches: So if u have only one single partition then u will have a single task/job that will use single core from your cluster and that will ultimately require more than 50GB RAM, otherwise you’ll run OOM. In case u have read the data as multi partitioned table then that 50GB will be sufficient ...


1

There certainly is theory, or at least competing methodologies, behind ETL and Data Warehousing, for a start look at the Inmon vs Kimball methodologies. In a nutshell (I could talk for days on this subject), Bruce Inmon's (the Father of Data Warehousing) methodology revolved around building a large, loosely 3rd normalized data warehouse from multiple sources,...


1

I can think of a few ways besides using Apache Spark. There is a python script doing just that: https://github.com/rondunn/odbc2parquet. It utilizes pyodbc though. Depending on your use case you may find that https://github.com/blue-yonder/turbodbc would deliver a much speedier experience querying the data. turbodbc can also directly emit arrow arrays. ...


1

I would suggest a data fabric. That would meet your need for data acquisition, preprocessing, data quality, master data management, etc. Given I work for Talend, I would suggest our data fabric. =) Here’s a case study with the Panama Papers. https://www.talend.com/blog/2017/01/17/talend-data-masters-2016-icij-decoded-panama-papers-talend/ The concept in ...


1

Just about any ETL tool can manage fixed width, CSV, TSV, or PSV input, and just about any tool should be able to manage 100B records. The limiting part of the question really has to do with what your destination format is, and what disk throughput you need. Expected throughput on an i2.4xLarge is 250mb/s. If an 8xLarge is double that, times 32 machines, ...


1

I don't think you'll find anything that checks all of your requirements, but here are some things to look at: Automated ETL mapping: There is a tool called Karma started by a team at USC's Information Sciences Institute. It learns from your ETL mappings and helps automate future mappings. It's the only open source tool I'm aware of that helps automate the ...


Only top voted, non community-wiki answers of a minimum length are eligible