We trained a model in a single server using pandas, dataframe = 2.000.000 rows(I run it later in my own laptop), now we are migrating the code to the cloud and in that case to Databricks. In databricks we can run pandas withoud problems(we use it for other projects) but for any reason that I can not understand why they want that we translate our panda code to pyspark, I do not find any reason to do that. Scalability could be a reason but once you have a good moderl as far as I know more data doesn´t mean a better model.

So there is any good reasong to translate our code from Pandas to Pyspark(and invest several weeks doing that) ?

  • $\begingroup$ any chance that your dataframe is produced from raw data placed in a Hadoop file system? $\endgroup$ Jan 26, 2022 at 16:59
  • $\begingroup$ Hi @RaymondKwok the data will be in a Delta Lake table(because right now was in csv in the server), the standar way to read that is in spark, spark.read .. .. but the think is that after the first read you would just need to tranlate to pandas (ok it mabe take a minute) df.toPandas() but after that you are done and that is something that you dont run every day just maybe 1 a month, so why invest a lot of time tranlsating it to pyspark ? $\endgroup$ Jan 27, 2022 at 16:14
  • $\begingroup$ I dont see the advanjate to tranlate something that I probram in a laptob and was working toa pyspark that its optimal for parallel processing when you have a lot of GB of data $\endgroup$ Jan 27, 2022 at 16:17


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.