I know that pandas works "under the hood" with numpy arrays stored in dictionaries. In contrast, Koalas works with the underlying Spark framework. Does that mean that there is no extra cost associated with switching back and forth between Koalas and PySpark dataframes?

#convert to pyspark dataframe

#convert to kolas frame
koalas_df = ks.DataFrame(df)

Edit: With cost I mean, does it ks.Dataframe(ks) create additional overhead? For example, toPandas() results in the collection of all records in the DataFrame to the driver program. Therefore we can only do toPandas() on a small subset of data.

Since I am switching between Koalas and Spark I am wondering if there is any such overhead or if Koalas "interprets" Spark dataframes without collecting records on the driver. At the moment I am working with a small subset of the data, but I am interested in any drawbacks when using larger amounts.


3 Answers 3


As you said, since the Koalas is aiming for processing the big data, there is no such overhead like collecting data into a single partition when ks.DataFrame(df).

However, the overheads are occurred when creating a default columns for creating the _InternalFrame which internally manages the metadata between pandas and PySpark.

Koalas is internally using immutable frame named _InternalFrame, so you can refer the /koalas/databricks/koalas/internal.py if you want to more detail.

Here is a short code examples from internal.py that creating the default index if given Spark DataFrame has no index information.


        if index_map is None:
            assert not any(SPARK_INDEX_NAME_PATTERN.match(name) for name in spark_frame.columns), (
                "Index columns should not appear in columns of the Spark DataFrame. Avoid "
                "index column names [%s]." % SPARK_INDEX_NAME_PATTERN

            # Create default index.
            spark_frame = _InternalFrame.attach_default_index(spark_frame)
            index_map = OrderedDict({SPARK_DEFAULT_INDEX_NAME: None})

        if NATURAL_ORDER_COLUMN_NAME not in spark_frame.columns:
            spark_frame = spark_frame.withColumn(
                NATURAL_ORDER_COLUMN_NAME, F.monotonically_increasing_id()

I believe Kolas is the Databricks DF equivalent of a Python DF and the equivalent of a Spark DF (I think Kolas is very,very new; released just a few months ago). I don't know what you mean by cost, but you can easily switch between Spark DF and Pandas DF. See the examples below.

# Convert Koala dataframe to Spark dataframe
df = kdf.to_spark(kdf)

# Create a Spark DataFrame from a Pandas DataFrame
df = spark.createDataFrame(pdf)

# Convert the Spark DataFrame to a Pandas DataFrame
df = df.select("*").toPandas(sdf)

If you are asking how much you will be billed for the time used, it's just pennies, really. Personally, I think some things are a lot easier to do in Python, vs other languages. So, if you have some non-Python DF, and you want to convert it into a Python DF, to do a merge, or whatever, just make the conversion and do a merge.

result = pd.concat([df1, df2], axis=1)

Does that make sense?

  • $\begingroup$ hey @ryguy72. Thank you for your answer. I am using Koalas to do data preprocessing and for visualization. It is quite convenient. For the actual machine learning I am using still Spark though. What I meant in my question is conversion cost. I will edit my question, hopefully it will be more clear afterwards. $\endgroup$
    – DataBach
    Mar 2, 2020 at 7:34

A little late to the party on this one, but here is some code and output (run on my moderate laptop):

startTime = datetime.now()
spark_medium = koalas_medium.to_spark()
print('Convert Medium Size DF to spark: (3 x 10,000): ' + str (datetime.now() - 

startTime = datetime.now()
spark_large = koalas_large.to_spark()
print('Convert Large Size DF to spark: (23 x Millions of rows): ' + str 
(datetime.now() - startTime))


  • Convert Medium Size DF to spark: (3 x 10,000): 0:00:00.689036

  • Convert Large Size DF to spark: (23 x Millions of rows): 0:00:00.078039

Why it's slower to convert the medium DF I can't tell you, but running it several times the results were always roughly the same. Hope this helps someone.

  • $\begingroup$ It is slower for a smaller DF because distributed computing has it sole purpose in big data processing.In fact, if you have a small data-set and partition it across multiple machines for processing, I/O operations & aggregation will generate more overhead than simply processing the data on a single machine. The data-set you choose as medium is still "too small". Hence you large data-set is converted faster. Thank you for your answer, but it still does not answer my question if it should be avoided to convert between Koalas and sparks DFs frequently. $\endgroup$
    – DataBach
    Mar 13, 2020 at 12:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.