1
$\begingroup$

I have a requirement to store a nested list of json objects in a column by doing a JOIN between two datasets related by one-to-many relation. Example: stackoverflow posts (each question can have one or many answers), answers should be populated against each question as a nested list of dict.

I am able to achieve this perfectly using pandas. I can store the output as parquet and also load it back again using pandas. However, due to performance reasons I am using pyspark.

But, when I store the nested list of objects column using pyspark, I am not able to load it back using pandas, which makes me wonder if I can store it differently such a way that I can load it back using pandas.

The error I get is this:

pyarrow.lib.ArrowNotImplementedError: Not implemented type for Arrow list to pandas: map<string, string>

If I convert the pyspark dataframe to pandas and store it as parquet file, I don't run into this error. So, I believe parquet has the necessary support to be able to store the data that way I want (so that I can load these list(dict) columns using pandas).

Below is my pyspark code

        answers_dicted = answers_df.withColumn("dict",
        create_map(create_map_args(answers_df))
    ).select(["Id", "ParentId", "dict"])

    # group answers by questions
    print("Grouping answers by questions")
    answers_grouped = answers_dicted.groupby("ParentId").agg(collect_list("dict").alias("answers"))

    # populate questions with answers 
    print("Adding answers to questions")
    questions_df = questions_df.join(answers_grouped, questions_df.Id == answers_grouped.ParentId, "left").select(questions_df["*"], answers_grouped["answers"])
$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.