Data Science

Questions tagged [etl]

Ask Question

The tag has no usage guidance.

12 questions
Newest
Active
Bountied
Unanswered
Filter by
Sorted by
Tagged with
1
vote
1answer
19 views

Extracting and Mining PDF Data [closed]

I have a pdf file (admission application). I want to read/search the pdf and extract terms with similar meaning and then convert this data into a DataFrame to save as a xlsm file. HELP!
0
votes
0answers
11 views

Integrating Dagster with Django ORM

I want to integrate Dagster into ongoing Django project. Dagster runs out of Django context and eventually there is no way to directly access django ORM without calling ...
0
votes
0answers
80 views

Viewing TensorFlow datasets vs. pandas DataFrames: Differences?

A simple preview of a pandas.DataFrame is typically done by calling head(k) and will simply return the first ...
0
votes
1answer
82 views

How to create a parquet file from a query to a mysql table

Updating a legacy ~ETL; on it's base it exports some tables of the prod DB to s3, the export contains a query. The export process generates a csv file using the following logic: ...
0
votes
0answers
16 views

How can I preview AWS Glue jobs?

I´m want to use Glue to extract data from an RDS PostgresDB, transform/clean it and load into an S3 Bucket so I can use Athena and Quicksight to visualize the data and create reports. I´m currently ...
0
votes
0answers
157 views

Python data cleaning for CSV using Pandas and PySpark and mapping to JSON in AWS ETL Glue Environment

I have recently started working on some ETL work and wanted some guidance in this area related to data cleaning from CSV to JSON mapping using AWS Glue, Python (pandas, pyspark). Question Need a ...
2
votes
1answer
89 views

What is the best practice to test a ETL pipeline?

In traditional software development practice, before going into production, a piece of code should go through various stages of testing (unit test, integration test, user acceptance test) to secure ...
1
vote
0answers
12 views

Can data aggregation (and statistics) be done using an ETL solution?

I am doing a stock trend prediction model. I use MySQL and I have a table that has the following colums: date | open | high | low | close | volume For every day ...
3
votes
1answer
2k views

How to make R or Python as fast as SAS for ODBC Oracle queries? [closed]

I want to use R or Python to query big structured SQL-type data, but they are very slow compared to SAS. I tried using R and Python to return a 1.3 million record Oracle ODBC passthrough query. The ...
1
vote
1answer
81 views

what ETL technique should i use for text documents using Hadoop?

I have a school Big Data project where basically the teacher is going to give us a large amount of text documents (from the Gutenberg project data set ) and he want us to give as output the document ...
0
votes
1answer
619 views

Convert date into number - Apache PIG

Imagine that I've a field called date in this format: "yyyy-mm-dd" and I want to convert to number like "yyymmdd". For that I'm trying to use this: ...
2
votes
2answers
264 views

Successful ETL Automation: Libraries, Review papers, Use Cases

I'm curious if anyone can point to some successful extract, transform, load (ETL) automation libraries, papers, or use cases for somewhat inhomogenious data? I would be interested to see any existing ...