Questions tagged [etl]
The etl tag has no usage guidance.
14
questions
5
votes
1
answer
200
views
Data engineering good and bad practice?
I'm a Data Analyst in a pretty big company and I'm having a really bad time with the data I'm being given. I spend about 70% of my time thinking about where to find the data and how to pull it instead ...
- 83
0
votes
1
answer
10
views
Batch-driven or event-driven ETL
I am trying to come up with a data pipeline architecture. The data I deal with is event logging for labs requested, failed, succeeded etc with timestamps and some customer info for several different ...
- 1
1
vote
1
answer
108
views
PySpark for Big Data and RAM usage
I'm trying to figure out the best and most efficient method of handing ETL operations for big data. My question is this.
Say I have a table that is ~50 GB in size. In order to effectively transfer the ...
- 11
0
votes
1
answer
38
views
Transitioning from a python script for data transformation to BigQuery
So I have a dataset spread over multiple and ever-growing excel files all of which looks like:
email
order_ID
order_date
user1@email.com
1234
23-Mar-2021
user2@email.com
1235
23-Mar-2021
user2@...
- 133
0
votes
0
answers
59
views
How to schedule importing data files from SFTP server located on compute engine instance into BigQuery?
What I want to achieve:
Transfer hourly coming data files onto a SFTP file server located on a compute engine VM from several different feeds into Bigquery with real-time updates effectively & ...
- 133
5
votes
2
answers
223
views
ETL and Data Engineering - is it purely the knowledge of tools or is there theory behind it?
I would like to better understand what a good Data Englineer must know or what he does. Job descriptions mostly list tools that are required, such as Python.
If it is possible to separate Data ...
- 51
2
votes
1
answer
1k
views
Extracting and Mining PDF Data [closed]
I have a pdf file (admission application). I want to read/search the pdf and extract terms with similar meaning and then convert this data into a DataFrame to save as a xlsm file. HELP!
- 21
0
votes
0
answers
111
views
Integrating Dagster with Django ORM
I want to integrate Dagster into ongoing Django project. Dagster runs out of Django context and eventually there is no way to directly access django ORM without calling ...
1
vote
2
answers
3k
views
How to create a parquet file from a query to a mysql table
Updating a legacy ~ETL; on it's base it exports some tables of the prod DB to s3, the export contains a query. The export process generates a csv file using the following logic:
...
4
votes
1
answer
1k
views
What is the best practice to test a ETL pipeline?
In traditional software development practice, before going into production, a piece of code should go through various stages of testing (unit test, integration test, user acceptance test) to secure ...
- 41
3
votes
1
answer
3k
views
How to make R or Python as fast as SAS for ODBC Oracle queries? [closed]
I want to use R or Python to query big structured SQL-type data, but they are very slow compared to SAS.
I tried using R and Python to return a 1.3 million record Oracle ODBC passthrough query. The ...
- 211
1
vote
1
answer
147
views
what ETL technique should i use for text documents using Hadoop?
I have a school Big Data project where basically the teacher is going to give us a large amount of text documents (from the Gutenberg project data set ) and he want us to give as output the document ...
0
votes
1
answer
846
views
Convert date into number - Apache PIG
Imagine that I've a field called date in this format: "yyyy-mm-dd" and I want to convert to number like "yyymmdd". For that I'm trying to use this:
...
- 159
2
votes
2
answers
291
views
Successful ETL Automation: Libraries, Review papers, Use Cases
I'm curious if anyone can point to some successful extract, transform, load (ETL) automation libraries, papers, or use cases for somewhat inhomogenious data?
I would be interested to see any existing ...
- 6,638