Questions tagged [etl]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1
vote
1answer
33 views

PySpark for Big Data and RAM usage

I'm trying to figure out the best and most efficient method of handing ETL operations for big data. My question is this. Say I have a table that is ~50 GB in size. In order to effectively transfer the ...
0
votes
0answers
10 views

GCloud AI Platform for an ETL process is the right approach?

I'm involved in a project consisting in: Extracting from the internet a document (file of 8/10 pages) Processing it (extract text, clean, tokenize and lemmatizing) Loading it to a Relational data ...
0
votes
1answer
35 views

Transitioning from a python script for data transformation to BigQuery

So I have a dataset spread over multiple and ever-growing excel files all of which looks like: email order_ID order_date user1@email.com 1234 23-Mar-2021 user2@email.com 1235 23-Mar-2021 user2@...
0
votes
0answers
19 views

How to schedule importing data files from SFTP server located on compute engine instance into BigQuery?

What I want to achieve: Transfer hourly coming data files onto a SFTP file server located on a compute engine VM from several different feeds into Bigquery with real-time updates effectively & ...
0
votes
0answers
42 views

Create User Table from Google Firebase Analytics and Cloud Firestore Collections

I am exporting my snowplow collections collected on Firebase using the extention Export Collections to BigQuery firestore-bigquery-export v0.1.10. Whenever a document is created, updated, imported, or ...
6
votes
2answers
170 views

ETL and Data Engineering - is it purely the knowledge of tools or is there theory behind it?

I would like to better understand what a good Data Englineer must know or what he does. Job descriptions mostly list tools that are required, such as Python. If it is possible to separate Data ...
2
votes
1answer
1k views

Extracting and Mining PDF Data [closed]

I have a pdf file (admission application). I want to read/search the pdf and extract terms with similar meaning and then convert this data into a DataFrame to save as a xlsm file. HELP!
0
votes
0answers
76 views

Integrating Dagster with Django ORM

I want to integrate Dagster into ongoing Django project. Dagster runs out of Django context and eventually there is no way to directly access django ORM without calling ...
1
vote
2answers
3k views

How to create a parquet file from a query to a mysql table

Updating a legacy ~ETL; on it's base it exports some tables of the prod DB to s3, the export contains a query. The export process generates a csv file using the following logic: ...
3
votes
1answer
980 views

What is the best practice to test a ETL pipeline?

In traditional software development practice, before going into production, a piece of code should go through various stages of testing (unit test, integration test, user acceptance test) to secure ...
3
votes
1answer
3k views

How to make R or Python as fast as SAS for ODBC Oracle queries? [closed]

I want to use R or Python to query big structured SQL-type data, but they are very slow compared to SAS. I tried using R and Python to return a 1.3 million record Oracle ODBC passthrough query. The ...
1
vote
1answer
139 views

what ETL technique should i use for text documents using Hadoop?

I have a school Big Data project where basically the teacher is going to give us a large amount of text documents (from the Gutenberg project data set ) and he want us to give as output the document ...
0
votes
1answer
790 views

Convert date into number - Apache PIG

Imagine that I've a field called date in this format: "yyyy-mm-dd" and I want to convert to number like "yyymmdd". For that I'm trying to use this: ...
2
votes
2answers
284 views

Successful ETL Automation: Libraries, Review papers, Use Cases

I'm curious if anyone can point to some successful extract, transform, load (ETL) automation libraries, papers, or use cases for somewhat inhomogenious data? I would be interested to see any existing ...