Episode #125 of the Stack Overflow podcast is here. We talk Tilde Club and mechanical keyboards. Listen now

Questions tagged [etl]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
1answer
16 views

How to create a parquet file from a query to a mysql table

Updating a legacy ~ETL; on it's base it exports some tables of the prod DB to s3, the export contains a query. The export process generates a csv file using the following logic: ...
0
votes
0answers
13 views

How can I preview AWS Glue jobs?

I´m want to use Glue to extract data from an RDS PostgresDB, transform/clean it and load into an S3 Bucket so I can use Athena and Quicksight to visualize the data and create reports. I´m currently ...
0
votes
0answers
84 views

Python data cleaning for CSV using Pandas and PySpark and mapping to JSON in AWS ETL Glue Environment

I have recently started working on some ETL work and wanted some guidance in this area related to data cleaning from CSV to JSON mapping using AWS Glue, Python (pandas, pyspark). Question Need a ...
2
votes
1answer
38 views

What is the best practice to test a ETL pipeline?

In traditional software development practice, before going into production, a piece of code should go through various stages of testing (unit test, integration test, user acceptance test) to secure ...
1
vote
0answers
12 views

Can data aggregation (and statistics) be done using an ETL solution?

I am doing a stock trend prediction model. I use MySQL and I have a table that has the following colums: date | open | high | low | close | volume For every day ...
3
votes
1answer
2k views

How to make R or Python as fast as SAS for ODBC Oracle queries? [closed]

I want to use R or Python to query big structured SQL-type data, but they are very slow compared to SAS. I tried using R and Python to return a 1.3 million record Oracle ODBC passthrough query. The ...
1
vote
1answer
74 views

what ETL technique should i use for text documents using Hadoop?

I have a school Big Data project where basically the teacher is going to give us a large amount of text documents (from the Gutenberg project data set ) and he want us to give as output the document ...
0
votes
1answer
588 views

Convert date into number - Apache PIG

Imagine that I've a field called date in this format: "yyyy-mm-dd" and I want to convert to number like "yyymmdd". For that I'm trying to use this: ...
2
votes
2answers
258 views

Successful ETL Automation: Libraries, Review papers, Use Cases

I'm curious if anyone can point to some successful extract, transform, load (ETL) automation libraries, papers, or use cases for somewhat inhomogenious data? I would be interested to see any existing ...