3
$\begingroup$

I have been searching for methods to plot in PySpark.

I couldn't find any resource on plotting data residing in DataFrame in PySpark. The only methods which are listed are:

  • through method collect() which brings data into 'local' Python session and plot
  • through method toPandas() which converts data to 'local' Pandas Dataframe.

The problem is that these both are very time-consuming functions. Is there any method by which we can plot data residing in Spark session directly (not importing it into the local session)?

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
4
$\begingroup$

No, there is no such method, I have found out.

The reason is, plotting libraries run on a single machine and expect a rather sample dataset. Data on Spark is distributed among its clusters and hence needs to be brought to a local session first, from where it can be plotted.

That is why methods such as collect(), toPandas() are needed.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Could you show where you found that information? $\endgroup$
    – diogo
    Jul 1, 2019 at 22:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.