Questions tagged [etl]
The etl tag has no usage guidance.
9
questions
0
votes
1answer
16 views
How to create a parquet file from a query to a mysql table
Updating a legacy ~ETL; on it's base it exports some tables of the prod DB to s3, the export contains a query. The export process generates a csv file using the following logic:
...
0
votes
0answers
13 views
How can I preview AWS Glue jobs?
I´m want to use Glue to extract data from an RDS PostgresDB, transform/clean it and load into an S3 Bucket so I can use Athena and Quicksight to visualize the data and create reports.
I´m currently ...
0
votes
0answers
84 views
Python data cleaning for CSV using Pandas and PySpark and mapping to JSON in AWS ETL Glue Environment
I have recently started working on some ETL work and wanted some guidance in this area related to data cleaning from CSV to JSON mapping using AWS Glue, Python (pandas, pyspark).
Question
Need a ...
2
votes
1answer
38 views
What is the best practice to test a ETL pipeline?
In traditional software development practice, before going into production, a piece of code should go through various stages of testing (unit test, integration test, user acceptance test) to secure ...
1
vote
0answers
12 views
Can data aggregation (and statistics) be done using an ETL solution?
I am doing a stock trend prediction model. I use MySQL and I have a table that has the following colums:
date | open | high | low | close | volume
For every day ...
3
votes
1answer
2k views
How to make R or Python as fast as SAS for ODBC Oracle queries? [closed]
I want to use R or Python to query big structured SQL-type data, but they are very slow compared to SAS.
I tried using R and Python to return a 1.3 million record Oracle ODBC passthrough query. The ...
1
vote
1answer
74 views
what ETL technique should i use for text documents using Hadoop?
I have a school Big Data project where basically the teacher is going to give us a large amount of text documents (from the Gutenberg project data set ) and he want us to give as output the document ...
0
votes
1answer
588 views
Convert date into number - Apache PIG
Imagine that I've a field called date in this format: "yyyy-mm-dd" and I want to convert to number like "yyymmdd". For that I'm trying to use this:
...
2
votes
2answers
258 views
Successful ETL Automation: Libraries, Review papers, Use Cases
I'm curious if anyone can point to some successful extract, transform, load (ETL) automation libraries, papers, or use cases for somewhat inhomogenious data?
I would be interested to see any existing ...
