Skip to main content

Questions tagged [data-engineering]

Data Engineering is the process of extracting data, (cleaning and) transforming it into a proper format, and loading it into a location where other users can access it, which is commonly abbreviated as ETL. This data is later on used by Data Scientists for their analysis.

Filter by
Sorted by
Tagged with
0 votes
0 answers
10 views

What's an OLAP cube and how does it differ from star schema?

There are 2, I guess related concepts: star schema and olap cube, what are their differences? When should one be chosen over the other?
Pavel Orekhov's user avatar
0 votes
0 answers
16 views

What to include in fact and dimension table from election database

I am working with election dataset of India of year 2014 and data for 2019 I also have table for party names and descriptions and finally state name with code. I am not getting how do i create a ...
Sugam Sharma's user avatar
0 votes
0 answers
33 views

Best way to deal with a large dataset?

I am working on a project with a team of 4 people, on a relatively large dataset (around 500Gb). I cleaned the data and did some feature engineering, and now I want everyone to have simple access to ...
Mordechai's user avatar
1 vote
1 answer
34 views

Machine Learning on tabular data with spherical coordinates

I am wondering if any work has been done on machine learning with high dimensional spherical coordinates for tabular data. Because with tabular data, upon encoding the data for machine learning, we ...
timmy1691's user avatar
0 votes
0 answers
17 views

Are there any data processing documentation tools?

I was responsible of generating a table report to a commercial area of my job. It is currently being updated daily by DAG in airtomation that runs a python script. The problem is that now I have to ...
Gustavo's user avatar
2 votes
0 answers
60 views

How to build this data pipeline?

I don't have much experience in data engineering, so I'm here to ask for advice. I am working on a project which consists of building a dashboard for the IT department of a bank. the dashboard should ...
Wissem Boujlida's user avatar
0 votes
1 answer
85 views

Is it possible to use the SODA tool as open source?

I was reading about the SODA tool and was wondering if it was free to use. As far as I understand, it has a SODA library and a SODA core, but they are all just clients for connecting to a cloud server ...
JustUser's user avatar
0 votes
1 answer
59 views

Storing XML data from thousands of API calls

I am working on a data pipeline that pulls document metadata from an electronic document management system (EDMS). The end goal is to visualize document statuses (overdue, submitted etc.) in a PowerBI ...
spiralini's user avatar
2 votes
2 answers
189 views

How to deal with high data volumes? (Tools, techniques, concepts, etc.)

I have some doubts about how to deal with high volumes of data. I'm currently working in the data analysis/data science field, so I've had the chance to perform calculations, manipulate data, and ...
tms's user avatar
  • 21
0 votes
0 answers
11 views

What are the implications of lethal use of military technology developed by industrial engineers with the use of data science?

I’m currently in my first year of collage in engineering and I was wondering about the implications of the use of technology in military applications, with the combination of data science and ...
SantAlt33's user avatar
-1 votes
1 answer
23 views

How to Data Engineer a dataset to get the best featurres to predict a target class?

In my dataset, I have data of IDs that don't create any meaningful relationship with each other and when I test that dataset on different models I am not getting accuracy more than 40%. Anyone can ...
Farhan Aslam's user avatar
0 votes
1 answer
387 views

How would I design a database structure for a feature store?

I have a personal project to create predictions for tennis matches. It currently consists of a Python application and a MySQL database. I extract data from various websites and APIs and store it in ...
Jossy's user avatar
  • 103
1 vote
2 answers
303 views

Preprocessing vs Data Engineering vs Feature Engineering

I'm having a very hard categorizing different methods of data preparation into the 3 categories Preprocessing, Data Engineering, Feature Engineering. A somewhat common definition describes Feature ...
GondorCallsForAid's user avatar
1 vote
0 answers
415 views

Airbyte docker container not able to read local json file on MacOSX

I am just trying to test out airbyte, I am loading up a json, I want to do some data manipulation and export it back out to json. All done locally with while running on docker. ...
user3738936's user avatar
0 votes
3 answers
6k views

How to upload a Pandas DataFrame into a Postgres table in Airflow?

I need to create a task in Airflow which downloads data from one Postgres database and uploads it to another. My code looks like this: ...
frogseer's user avatar
1 vote
1 answer
87 views

Migrating legacy data in Kafka to use a schema registry to support a streaming data pipeline

(If this is not the correct Stack site for this question, please let me know - it seemed the best fit) We currently have a 'legacy' system whose output are events coming in from a distributed ...
user991710's user avatar
2 votes
1 answer
254 views

How to partition data effectively?

I have a pipeline which outputs model scores to s3. I need to partition the data by model_type and date. Which is the most efficient way to partition the data from the below: s3://bucket/data/...
CyberPunk's user avatar
  • 141
5 votes
1 answer
393 views

Data engineering good and bad practice?

I'm a Data Analyst in a pretty big company and I'm having a really bad time with the data I'm being given. I spend about 70% of my time thinking about where to find the data and how to pull it instead ...
Marc's user avatar
  • 222
0 votes
1 answer
231 views

Batch-driven or event-driven ETL

I am trying to come up with a data pipeline architecture. The data I deal with is event logging for labs requested, failed, succeeded etc with timestamps and some customer info for several different ...
alyonushka's user avatar
1 vote
2 answers
238 views

When Does Feature Selection Takes Place?

I have a dataset where there are categorical features as well as numeric features, and I have to perform OneHotEncoding, Normalization and feature selection on it. In what order should I perform these ...
Jainam Shroff's user avatar
1 vote
0 answers
20 views

Best Technologies opening Large Sets of Sensor Time-Series Data to Analytics

My team is exploring options to create a robust "analytics" capability that is well-suited for our large quantities of sensor test data. I'd appreciate any suggestions for technologies that ...
CrashLandon's user avatar
2 votes
1 answer
174 views

Alternative to EC2 for running ML batch training jobs on AWS

We are building an ML pipeline on AWS, which will obviously require some heavy-compute components including preprocessing and batch training. Most the the pipeline is on Lambda, but Lambda is known to ...
Cybernetic's user avatar
0 votes
2 answers
993 views

How do I replace NaN values using group by pivot_table in pandas DataFrame?

I am working on a machine learning practice problem, from https://datahack.analyticsvidhya.com/contest/practice-problem-big-mart-sales-iii/#ProblemStatement I want to replace the null values in the ...
Amandeep Singh's user avatar
3 votes
2 answers
932 views

Storing Large dataset for processing and analysis of data

I am new to data engineering and wanted to know , what is the best way to store more than 3000 GB of data for further processing and analysis ? I am specifically looking for open source resources . I ...
user14519285's user avatar
0 votes
1 answer
75 views

Transitioning from a python script for data transformation to BigQuery

So I have a dataset spread over multiple and ever-growing excel files all of which looks like: email order_ID order_date [email protected] 1234 23-Mar-2021 [email protected] 1235 23-Mar-2021 user2@...
Hamza's user avatar
  • 143
0 votes
0 answers
127 views

How to schedule importing data files from SFTP server located on compute engine instance into BigQuery?

What I want to achieve: Transfer hourly coming data files onto a SFTP file server located on a compute engine VM from several different feeds into Bigquery with real-time updates effectively & ...
Hamza's user avatar
  • 143
0 votes
1 answer
48 views

How to Include Features that Apply to Specific Classes

I'm predicting hours that will be worked for building tasks. Due to the overall low sample size, I've stacked multiple related tasks together into a single model. (There may be 100 total samples in a ...
phatgamer69's user avatar
1 vote
3 answers
387 views

Advice on where to continue in the field of data engineering and machine learning

I finished a 28 hours Machine learning with python (Basic course) on Udemy, and it was very beneficial. My aim, is to be able to understand what is ML and how to use its concepts while working with ...
alim1990's user avatar
  • 163
5 votes
3 answers
390 views

ETL and Data Engineering - is it purely the knowledge of tools or is there theory behind it?

I would like to better understand what a good Data Englineer must know or what he does. Job descriptions mostly list tools that are required, such as Python. If it is possible to separate Data ...
MindYB's user avatar
  • 51
0 votes
1 answer
330 views

The OLAP (On-Line Analytical Processing) cube with 4 dimensions [closed]

A typical OLAP cube looks like this: As I can see, this cube can work with 2 or 3 dimensions, but what if I have 4 dimensions to produce facts? Should I use star schema instead when having more than ...
Dan D.'s user avatar
  • 125
4 votes
2 answers
128 views

Data Engineering Stack - collect, transform and visualize geospatial data

I'm making a side project, where I collect geospatial data by web scrapping and from OSM API. I've started with simple Java application, however, I would like to make it as a data flow, purely for ...
Forin's user avatar
  • 141
1 vote
0 answers
78 views

Efficient way of adding new columns to datamart without reprocessing complete pipeline

I have a software engineering background and relatively new to data engineering. I am building out tables/datamarts in our datalake for data scientists and analysts to use. We use Airflow for ...
Abdul R's user avatar
  • 11
0 votes
1 answer
93 views

Creating more than one worker nodes for local windows machine [closed]

I am using windows laptop. And I installed apache spark for my laptop. And I try to measure spark performance by changing spark components. because of that I want to create more than one worker nodes ...
randunu galhena's user avatar
0 votes
1 answer
468 views

How to run Spark python code in Jupyter Notebook via command prompt

I am trying to import a data frame into spark using Python's pyspark module. For this, I used Jupyter Notebook and executed the code shown in the screenshot below After that I want to run this in CMD ...
randunu galhena's user avatar
0 votes
1 answer
363 views

Can we add other features along with Text in sentiment analysis

Can we add other features along with Text to a ML model . Like giving text and other features as one input combining them and predict the output value. As model can learn some more better if given ...
sai prasanna Kumar's user avatar
2 votes
3 answers
2k views

Is there a cost associated with converting Koalas dataframe to Spark dataframe?

I know that pandas works "under the hood" with numpy arrays stored in dictionaries. In contrast, Koalas works with the underlying Spark framework. Does that mean that there is no extra cost associated ...
DataBach's user avatar
  • 165
2 votes
2 answers
120 views

Loading models from external source

I have a 500MB model which I am commiting to Git. That is a really bad practice since for newer model versions the repository will be huge. As well, It will slow down all builds for deployments. I ...
room13's user avatar
  • 133
1 vote
3 answers
6k views

How to sort a multi-level pandas data-frame by a particular column?

I would like to sort a multi-index pandas dataframe by a column, but do not want the entire dataframe to be sorted at once. But rather would like to sort by one of the indices. Here is an example of ...
user62198's user avatar
  • 1,091