What's the deal with Deno? We talk with a major contributor to find out. Listen now.
Data Science

Questions tagged [etl]

Ask Question

The tag has no usage guidance.

10 questions
Newest
Active
Bountied
Unanswered
Filter by
Sorted by
Tagged with
6
votes
2answers
116 views

ETL and Data Engineering - is it purely the knowledge of tools or is there theory behind it?

I would like to better understand what a good Data Englineer must know or what he does. Job descriptions mostly list tools that are required, such as Python. If it is possible to separate Data ...
2
votes
1answer
230 views

Extracting and Mining PDF Data [closed]

I have a pdf file (admission application). I want to read/search the pdf and extract terms with similar meaning and then convert this data into a DataFrame to save as a xlsm file. HELP!
0
votes
0answers
28 views

Integrating Dagster with Django ORM

I want to integrate Dagster into ongoing Django project. Dagster runs out of Django context and eventually there is no way to directly access django ORM without calling ...
0
votes
0answers
246 views

Viewing TensorFlow datasets vs. pandas DataFrames: Differences?

A simple preview of a pandas.DataFrame is typically done by calling head(k) and will simply return the first ...
0
votes
1answer
760 views

How to create a parquet file from a query to a mysql table

Updating a legacy ~ETL; on it's base it exports some tables of the prod DB to s3, the export contains a query. The export process generates a csv file using the following logic: ...
3
votes
1answer
400 views

What is the best practice to test a ETL pipeline?

In traditional software development practice, before going into production, a piece of code should go through various stages of testing (unit test, integration test, user acceptance test) to secure ...
3
votes
1answer
2k views

How to make R or Python as fast as SAS for ODBC Oracle queries? [closed]

I want to use R or Python to query big structured SQL-type data, but they are very slow compared to SAS. I tried using R and Python to return a 1.3 million record Oracle ODBC passthrough query. The ...
1
vote
1answer
109 views

what ETL technique should i use for text documents using Hadoop?

I have a school Big Data project where basically the teacher is going to give us a large amount of text documents (from the Gutenberg project data set ) and he want us to give as output the document ...
0
votes
1answer
677 views

Convert date into number - Apache PIG

Imagine that I've a field called date in this format: "yyyy-mm-dd" and I want to convert to number like "yyymmdd". For that I'm trying to use this: ...
2
votes
2answers
273 views

Successful ETL Automation: Libraries, Review papers, Use Cases

I'm curious if anyone can point to some successful extract, transform, load (ETL) automation libraries, papers, or use cases for somewhat inhomogenious data? I would be interested to see any existing ...