Skip to main content
Data Science

Return to Answer

edited body
Source Link
Benoit Descamps
Benoit Descamps
  • 341
  • 2
  • 6

Pandas unfortunately does not allow for the type of conditional join you wish to do, without copying a lot of unnecessary data before processing.

Your best solution is pandas to explode the teaches columns like you did and broadcast join the teachers. IdIf you are dealing with memory issues, and still want to deal with pandas then you will ave to multiprocess the join by splitting the courses dataframe.

Pandas unfortunately does not allow for the type of conditional join you wish to do, without copying a lot of unnecessary data before processing.

Your best solution is pandas to explode the teaches columns like you did and broadcast join the teachers. Id you are dealing with memory issues, and still want to deal with pandas then you will ave to multiprocess the join by splitting the courses dataframe.

Pandas unfortunately does not allow for the type of conditional join you wish to do, without copying a lot of unnecessary data before processing.

Your best solution is pandas to explode the teaches columns like you did and broadcast join the teachers. If you are dealing with memory issues, and still want to deal with pandas then you will ave to multiprocess the join by splitting the courses dataframe.

Source Link
Benoit Descamps
Benoit Descamps
  • 341
  • 2
  • 6

Pandas unfortunately does not allow for the type of conditional join you wish to do, without copying a lot of unnecessary data before processing.

Your best solution is pandas to explode the teaches columns like you did and broadcast join the teachers. Id you are dealing with memory issues, and still want to deal with pandas then you will ave to multiprocess the join by splitting the courses dataframe.