Questions tagged [data-engineering]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1
vote
0answers
13 views

Is there a cost associated with converting Koalas dataframe to Spark dataframe?

I know that pandas works "under the hood" with numpy arrays stored in dictionaries. In contrast, Koalas works with the underlying Spark framework. Does that mean that there is no extra cost associated ...
2
votes
1answer
28 views

Loading models from external source

I have a 500MB model which I am commiting to Git. That is a really bad practice since for newer model versions the repository will be huge. As well, It will slow down all builds for deployments. I ...
2
votes
1answer
87 views

how to sort a multi-level pandas data-frame by a particular column?

I want to sort a multi-index pandas dataframe by a column but don't want the entire dataframe to be sorted at once. But rather want to sort by one of the indices. Here is an example of what I mean: ...