Questions tagged [data-engineering]
The data-engineering tag has no usage guidance.
28
questions
0
votes
0
answers
36
views
Time series prediction of individual customer (transaction count and gross value) using LSTM/NARX
Problem:
I'm trying to rewrite Matlab code that goal is prediction of gross value using historical data of 2 features: gross value of next week. The data is about customer transaction in retail where ...
0
votes
1
answer
39
views
Preprocessing vs Data Engineering vs Feature Engineering
I'm having a very hard categorizing different methods of data preparation into the 3 categories Preprocessing, Data Engineering, Feature Engineering.
A somewhat common definition describes Feature ...
0
votes
0
answers
19
views
Cleaning data using a schema
I'm ingesting JSON data and as part of the my data pipeline, I'm hoping to clean the data to match a schema that can only be known at runtime.
Which tools should I be looking into? So far from my ...
- 101
0
votes
0
answers
140
views
Airbyte docker container not able to read local json file on MacOSX
I am just trying to test out airbyte, I am loading up a json, I want to do some data manipulation and export it back out to json. All done locally with while running on docker.
...
- 101
0
votes
1
answer
968
views
How to upload a Pandas DataFrame into a Postgres table in Airflow?
I need to create a task in Airflow which downloads data from one Postgres database and uploads it to another.
My code looks like this:
...
- 1
1
vote
1
answer
61
views
Migrating legacy data in Kafka to use a schema registry to support a streaming data pipeline
(If this is not the correct Stack site for this question, please let me know - it seemed the best fit)
We currently have a 'legacy' system whose output are events coming in from a distributed ...
- 71
2
votes
1
answer
73
views
How to partition data effectively?
I have a pipeline which outputs model scores to s3. I need to partition the data by model_type and date. Which is the most efficient way to partition the data from the below:
s3://bucket/data/...
- 141
5
votes
1
answer
316
views
Data engineering good and bad practice?
I'm a Data Analyst in a pretty big company and I'm having a really bad time with the data I'm being given. I spend about 70% of my time thinking about where to find the data and how to pull it instead ...
- 222
0
votes
1
answer
123
views
Batch-driven or event-driven ETL
I am trying to come up with a data pipeline architecture. The data I deal with is event logging for labs requested, failed, succeeded etc with timestamps and some customer info for several different ...
1
vote
2
answers
122
views
When Does Feature Selection Takes Place?
I have a dataset where there are categorical features as well as numeric features, and I have to perform OneHotEncoding, Normalization and feature selection on it.
In what order should I perform these ...
1
vote
0
answers
18
views
Best Technologies opening Large Sets of Sensor Time-Series Data to Analytics
My team is exploring options to create a robust "analytics" capability that is well-suited for our large quantities of sensor test data. I'd appreciate any suggestions for technologies that ...
- 11
1
vote
1
answer
101
views
Alternative to EC2 for running ML batch training jobs on AWS
We are building an ML pipeline on AWS, which will obviously require some heavy-compute components including preprocessing and batch training.
Most the the pipeline is on Lambda, but Lambda is known to ...
- 770
0
votes
2
answers
736
views
How do I replace NaN values using group by pivot_table in pandas DataFrame?
I am working on a machine learning practice problem, from https://datahack.analyticsvidhya.com/contest/practice-problem-big-mart-sales-iii/#ProblemStatement
I want to replace the null values in the ...
3
votes
2
answers
307
views
Storing Large dataset for processing and analysis of data
I am new to data engineering and wanted to know , what is the best way to store more than 3000 GB of data for further processing and analysis ? I am specifically looking for open source resources . I ...
- 41
0
votes
1
answer
56
views
Transitioning from a python script for data transformation to BigQuery
So I have a dataset spread over multiple and ever-growing excel files all of which looks like:
email
order_ID
order_date
user1@email.com
1234
23-Mar-2021
user2@email.com
1235
23-Mar-2021
user2@...
- 133
0
votes
0
answers
88
views
How to schedule importing data files from SFTP server located on compute engine instance into BigQuery?
What I want to achieve:
Transfer hourly coming data files onto a SFTP file server located on a compute engine VM from several different feeds into Bigquery with real-time updates effectively & ...
- 133
0
votes
1
answer
41
views
How to Include Features that Apply to Specific Classes
I'm predicting hours that will be worked for building tasks. Due to the overall low sample size, I've stacked multiple related tasks together into a single model. (There may be 100 total samples in a ...
- 28
1
vote
3
answers
288
views
Advice on where to continue in the field of data engineering and machine learning
I finished a 28 hours Machine learning with python (Basic course) on Udemy, and it was very beneficial.
My aim, is to be able to understand what is ML and how to use its concepts while working with ...
- 163
5
votes
2
answers
300
views
ETL and Data Engineering - is it purely the knowledge of tools or is there theory behind it?
I would like to better understand what a good Data Englineer must know or what he does. Job descriptions mostly list tools that are required, such as Python.
If it is possible to separate Data ...
- 51
0
votes
1
answer
157
views
The OLAP (On-Line Analytical Processing) cube with 4 dimensions [closed]
A typical OLAP cube looks like this:
As I can see, this cube can work with 2 or 3 dimensions, but what if I have 4 dimensions to produce facts?
Should I use star schema instead when having more than ...
- 125
4
votes
2
answers
110
views
Data Engineering Stack - collect, transform and visualize geospatial data
I'm making a side project, where I collect geospatial data by web scrapping and from OSM API. I've started with simple Java application, however, I would like to make it as a data flow, purely for ...
- 141
1
vote
0
answers
36
views
Efficient way of adding new columns to datamart without reprocessing complete pipeline
I have a software engineering background and relatively new to data engineering. I am building out tables/datamarts in our datalake for data scientists and analysts to use. We use Airflow for ...
- 11
0
votes
1
answer
43
views
Creating more than one worker nodes for local windows machine [closed]
I am using windows laptop. And I installed apache spark for my laptop. And I try to measure spark performance by changing spark components. because of that I want to create more than one worker nodes ...
0
votes
1
answer
295
views
How to run Spark python code in Jupyter Notebook via command prompt
I am trying to import a data frame into spark using Python's pyspark module. For this, I used Jupyter Notebook and executed the code shown in the screenshot below
After that I want to run this in CMD ...
0
votes
1
answer
306
views
Can we add other features along with Text in sentiment analysis
Can we add other features along with Text to a ML model . Like giving text and other features as one input combining them and predict the output value. As model can learn some more better if given ...
2
votes
3
answers
1k
views
Is there a cost associated with converting Koalas dataframe to Spark dataframe?
I know that pandas works "under the hood" with numpy arrays stored in dictionaries. In contrast, Koalas works with the underlying Spark framework. Does that mean that there is no extra cost associated ...
- 165
2
votes
2
answers
62
views
Loading models from external source
I have a 500MB model which I am commiting to Git. That is a really bad practice since for newer model versions the repository will be huge. As well, It will slow down all builds for deployments.
I ...
- 133
1
vote
3
answers
4k
views
How to sort a multi-level pandas data-frame by a particular column?
I would like to sort a multi-index pandas dataframe by a column, but do not want the entire dataframe to be sorted at once. But rather would like to sort by one of the indices. Here is an example of ...
- 1,067