Questions tagged [etl]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
5 votes
1 answer
200 views

Data engineering good and bad practice?

I'm a Data Analyst in a pretty big company and I'm having a really bad time with the data I'm being given. I spend about 70% of my time thinking about where to find the data and how to pull it instead ...
user avatar
  • 83
0 votes
1 answer
10 views

Batch-driven or event-driven ETL

I am trying to come up with a data pipeline architecture. The data I deal with is event logging for labs requested, failed, succeeded etc with timestamps and some customer info for several different ...
user avatar
  • 1
1 vote
1 answer
108 views

PySpark for Big Data and RAM usage

I'm trying to figure out the best and most efficient method of handing ETL operations for big data. My question is this. Say I have a table that is ~50 GB in size. In order to effectively transfer the ...
user avatar
  • 11
0 votes
1 answer
38 views

Transitioning from a python script for data transformation to BigQuery

So I have a dataset spread over multiple and ever-growing excel files all of which looks like: email order_ID order_date user1@email.com 1234 23-Mar-2021 user2@email.com 1235 23-Mar-2021 user2@...
user avatar
  • 133
0 votes
0 answers
59 views

How to schedule importing data files from SFTP server located on compute engine instance into BigQuery?

What I want to achieve: Transfer hourly coming data files onto a SFTP file server located on a compute engine VM from several different feeds into Bigquery with real-time updates effectively & ...
user avatar
  • 133
5 votes
2 answers
223 views

ETL and Data Engineering - is it purely the knowledge of tools or is there theory behind it?

I would like to better understand what a good Data Englineer must know or what he does. Job descriptions mostly list tools that are required, such as Python. If it is possible to separate Data ...
user avatar
  • 51
2 votes
1 answer
1k views

Extracting and Mining PDF Data [closed]

I have a pdf file (admission application). I want to read/search the pdf and extract terms with similar meaning and then convert this data into a DataFrame to save as a xlsm file. HELP!
user avatar
  • 21
0 votes
0 answers
111 views

Integrating Dagster with Django ORM

I want to integrate Dagster into ongoing Django project. Dagster runs out of Django context and eventually there is no way to directly access django ORM without calling ...
user avatar
1 vote
2 answers
3k views

How to create a parquet file from a query to a mysql table

Updating a legacy ~ETL; on it's base it exports some tables of the prod DB to s3, the export contains a query. The export process generates a csv file using the following logic: ...
user avatar
4 votes
1 answer
1k views

What is the best practice to test a ETL pipeline?

In traditional software development practice, before going into production, a piece of code should go through various stages of testing (unit test, integration test, user acceptance test) to secure ...
user avatar
  • 41
3 votes
1 answer
3k views

How to make R or Python as fast as SAS for ODBC Oracle queries? [closed]

I want to use R or Python to query big structured SQL-type data, but they are very slow compared to SAS. I tried using R and Python to return a 1.3 million record Oracle ODBC passthrough query. The ...
user avatar
1 vote
1 answer
147 views

what ETL technique should i use for text documents using Hadoop?

I have a school Big Data project where basically the teacher is going to give us a large amount of text documents (from the Gutenberg project data set ) and he want us to give as output the document ...
user avatar
0 votes
1 answer
846 views

Convert date into number - Apache PIG

Imagine that I've a field called date in this format: "yyyy-mm-dd" and I want to convert to number like "yyymmdd". For that I'm trying to use this: ...
user avatar
2 votes
2 answers
291 views

Successful ETL Automation: Libraries, Review papers, Use Cases

I'm curious if anyone can point to some successful extract, transform, load (ETL) automation libraries, papers, or use cases for somewhat inhomogenious data? I would be interested to see any existing ...
user avatar
  • 6,638