Questions tagged [etl]
The etl tag has no usage guidance.
12
questions
0
votes
1answer
24 views
Transitioning from a python script for data transformation to BigQuery
So I have a dataset spread over multiple and ever-growing excel files all of which looks like:
email
order_ID
order_date
user1@email.com
1234
23-Mar-2021
user2@email.com
1235
23-Mar-2021
user2@...
0
votes
0answers
10 views
How to schedule importing data files from SFTP server located on compute engine instance into BigQuery?
What I want to achieve:
Transfer hourly coming data files onto a SFTP file server located on a compute engine VM from several different feeds into Bigquery with real-time updates effectively & ...
0
votes
0answers
26 views
Create User Table from Google Firebase Analytics and Cloud Firestore Collections
I am exporting my snowplow collections collected on Firebase using the extention
Export Collections to BigQuery
firestore-bigquery-export
v0.1.10.
Whenever a document is created, updated, imported, or ...
6
votes
2answers
136 views
ETL and Data Engineering - is it purely the knowledge of tools or is there theory behind it?
I would like to better understand what a good Data Englineer must know or what he does. Job descriptions mostly list tools that are required, such as Python.
If it is possible to separate Data ...
2
votes
1answer
658 views
Extracting and Mining PDF Data [closed]
I have a pdf file (admission application). I want to read/search the pdf and extract terms with similar meaning and then convert this data into a DataFrame to save as a xlsm file. HELP!
0
votes
0answers
48 views
Integrating Dagster with Django ORM
I want to integrate Dagster into ongoing Django project. Dagster runs out of Django context and eventually there is no way to directly access django ORM without calling ...
1
vote
2answers
2k views
How to create a parquet file from a query to a mysql table
Updating a legacy ~ETL; on it's base it exports some tables of the prod DB to s3, the export contains a query. The export process generates a csv file using the following logic:
...
3
votes
1answer
708 views
What is the best practice to test a ETL pipeline?
In traditional software development practice, before going into production, a piece of code should go through various stages of testing (unit test, integration test, user acceptance test) to secure ...
3
votes
1answer
2k views
How to make R or Python as fast as SAS for ODBC Oracle queries? [closed]
I want to use R or Python to query big structured SQL-type data, but they are very slow compared to SAS.
I tried using R and Python to return a 1.3 million record Oracle ODBC passthrough query. The ...
1
vote
1answer
125 views
what ETL technique should i use for text documents using Hadoop?
I have a school Big Data project where basically the teacher is going to give us a large amount of text documents (from the Gutenberg project data set ) and he want us to give as output the document ...
0
votes
1answer
746 views
Convert date into number - Apache PIG
Imagine that I've a field called date in this format: "yyyy-mm-dd" and I want to convert to number like "yyymmdd". For that I'm trying to use this:
...
2
votes
2answers
279 views
Successful ETL Automation: Libraries, Review papers, Use Cases
I'm curious if anyone can point to some successful extract, transform, load (ETL) automation libraries, papers, or use cases for somewhat inhomogenious data?
I would be interested to see any existing ...
