Questions tagged [data-engineering]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
1answer
33 views

How to Include Features that Apply to Specific Classes

I'm predicting hours that will be worked for building tasks. Due to the overall low sample size, I've stacked multiple related tasks together into a single model. (There may be 100 total samples in a ...
1
vote
2answers
37 views

Which Language Is Best for Data Scientists/Machine Learning Practitioners to Learn? [closed]

Working as a data scientist after a Phd in computer science. After spending almost almost 3 years in the industry, I love the software engineering component of my job (data pipeline development, model ...
1
vote
1answer
1k views

how to sort a multi-level pandas data-frame by a particular column?

I want to sort a multi-index pandas dataframe by a column but don't want the entire dataframe to be sorted at once. But rather want to sort by one of the indices. Here is an example of what I mean: ...
-1
votes
2answers
62 views

How to deal with nan values after merging / joining two dataframes? [closed]

A lot of time after merging two pandas dataframes, I end up with NaNs in the new dataframe, that's just how the way it is, because one csv does not have all the ID's that the other has (Two dataframes ...
4
votes
2answers
66 views

Data Engineering Stack - collect, transform and visualize geospatial data

I'm making a side project, where I collect geospatial data by web scrapping and from OSM API. I've started with simple Java application, however, I would like to make it as a data flow, purely for ...
6
votes
2answers
122 views

ETL and Data Engineering - is it purely the knowledge of tools or is there theory behind it?

I would like to better understand what a good Data Englineer must know or what he does. Job descriptions mostly list tools that are required, such as Python. If it is possible to separate Data ...
0
votes
1answer
28 views

How to run Spark python code in Jupyter Notebook via command prompt

I am trying to import a data frame into spark using Python's pyspark module. For this, I used Jupyter Notebook and executed the code shown in the screenshot below After that I want to run this in CMD ...
1
vote
3answers
61 views

Advice on where to continue in the field of data engineering and machine learning

I finished a 28 hours Machine learning with python (Basic course) on Udemy, and it was very beneficial. My aim, is to be able to understand what is ML and how to use its concepts while working with ...
0
votes
1answer
24 views

The OLAP (On-Line Analytical Processing) cube with 4 dimensions [closed]

A typical OLAP cube looks like this: As I can see, this cube can work with 2 or 3 dimensions, but what if I have 4 dimensions to produce facts? Should I use star schema instead when having more than ...
1
vote
0answers
14 views

Efficient way of adding new columns to datamart without reprocessing complete pipeline

I have a software engineering background and relatively new to data engineering. I am building out tables/datamarts in our datalake for data scientists and analysts to use. We use Airflow for ...
0
votes
1answer
20 views

Creating more than one worker nodes for local windows machine [closed]

I am using windows laptop. And I installed apache spark for my laptop. And I try to measure spark performance by changing spark components. because of that I want to create more than one worker nodes ...
1
vote
3answers
636 views

Is there a cost associated with converting Koalas dataframe to Spark dataframe?

I know that pandas works "under the hood" with numpy arrays stored in dictionaries. In contrast, Koalas works with the underlying Spark framework. Does that mean that there is no extra cost associated ...
0
votes
1answer
37 views

Can we add other features along with Text in sentiment analysis

Can we add other features along with Text to a ML model . Like giving text and other features as one input combining them and predict the output value. As model can learn some more better if given ...
2
votes
1answer
30 views

Loading models from external source

I have a 500MB model which I am commiting to Git. That is a really bad practice since for newer model versions the repository will be huge. As well, It will slow down all builds for deployments. I ...