Questions tagged [etl]
The etl tag has no usage guidance.
14
questions
1
vote
1answer
13 views
PySpark for Big Data and RAM usage
I'm trying to figure out the best and most efficient method of handing ETL operations for big data. My question is this.
Say I have a table that is ~50 GB in size. In order to effectively transfer the ...
0
votes
0answers
9 views
GCloud AI Platform for an ETL process is the right approach?
I'm involved in a project consisting in:
Extracting from the internet a document (file of 8/10 pages)
Processing it (extract text, clean, tokenize and lemmatizing)
Loading it to a Relational data ...
0
votes
1answer
30 views
Transitioning from a python script for data transformation to BigQuery
So I have a dataset spread over multiple and ever-growing excel files all of which looks like:
email
order_ID
order_date
user1@email.com
1234
23-Mar-2021
user2@email.com
1235
23-Mar-2021
user2@...
0
votes
0answers
16 views
How to schedule importing data files from SFTP server located on compute engine instance into BigQuery?
What I want to achieve:
Transfer hourly coming data files onto a SFTP file server located on a compute engine VM from several different feeds into Bigquery with real-time updates effectively & ...
0
votes
0answers
38 views
Create User Table from Google Firebase Analytics and Cloud Firestore Collections
I am exporting my snowplow collections collected on Firebase using the extention
Export Collections to BigQuery
firestore-bigquery-export
v0.1.10.
Whenever a document is created, updated, imported, or ...
6
votes
2answers
160 views
ETL and Data Engineering - is it purely the knowledge of tools or is there theory behind it?
I would like to better understand what a good Data Englineer must know or what he does. Job descriptions mostly list tools that are required, such as Python.
If it is possible to separate Data ...
2
votes
1answer
866 views
Extracting and Mining PDF Data [closed]
I have a pdf file (admission application). I want to read/search the pdf and extract terms with similar meaning and then convert this data into a DataFrame to save as a xlsm file. HELP!
0
votes
0answers
67 views
Integrating Dagster with Django ORM
I want to integrate Dagster into ongoing Django project. Dagster runs out of Django context and eventually there is no way to directly access django ORM without calling ...
1
vote
2answers
2k views
How to create a parquet file from a query to a mysql table
Updating a legacy ~ETL; on it's base it exports some tables of the prod DB to s3, the export contains a query. The export process generates a csv file using the following logic:
...
3
votes
1answer
862 views
What is the best practice to test a ETL pipeline?
In traditional software development practice, before going into production, a piece of code should go through various stages of testing (unit test, integration test, user acceptance test) to secure ...
3
votes
1answer
3k views
How to make R or Python as fast as SAS for ODBC Oracle queries? [closed]
I want to use R or Python to query big structured SQL-type data, but they are very slow compared to SAS.
I tried using R and Python to return a 1.3 million record Oracle ODBC passthrough query. The ...
1
vote
1answer
135 views
what ETL technique should i use for text documents using Hadoop?
I have a school Big Data project where basically the teacher is going to give us a large amount of text documents (from the Gutenberg project data set ) and he want us to give as output the document ...
0
votes
1answer
781 views
Convert date into number - Apache PIG
Imagine that I've a field called date in this format: "yyyy-mm-dd" and I want to convert to number like "yyymmdd". For that I'm trying to use this:
...
2
votes
2answers
283 views
Successful ETL Automation: Libraries, Review papers, Use Cases
I'm curious if anyone can point to some successful extract, transform, load (ETL) automation libraries, papers, or use cases for somewhat inhomogenious data?
I would be interested to see any existing ...
