1
$\begingroup$

I am pretty new to polars and have been working around it just to get acquainted.

import polars as pl

filename_data = 'endomondoHR.json'
pl.read_json(filename_data)

Data -> kaggle data

The error I am getting is :

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
c:\Users\shiv\Desktop\test\test.ipynb Cell 13 in <cell line: 4>()
      1 import polars as pl
      2 filename_data = 'endomondoHR.json'
----> 4 pl.read_json(filename_data)

File c:\Users\shiv\miniconda3\lib\site-packages\polars\io.py:985, in read_json(file)
    971 def read_json(file: str | Path | IOBase) -> DataFrame:
    972     """
    973     Read into a DataFrame from a JSON file.
    974 
   (...)
    983 
    984     """
--> 985     return DataFrame._read_json(file)

File c:\Users\shiv\miniconda3\lib\site-packages\polars\internals\dataframe\frame.py:959, in DataFrame._read_json(cls, file)
    956     file = normalise_filepath(file)
    958 self = cls.__new__(cls)
--> 959 self._df = PyDataFrame.read_json(file, False)
    960 return self

RuntimeError: BindingsError: "ComputeError(Owned(\"InvalidEOF\"))"

Useful informations :

Python version : 3.9.13 Windows 11, 16GB RAM, Intel Core i5-11400H, NVIDIA-GeForce GTX 1650 with 4GB GDDR6 dedicated VRAM Data size 6.6 GB

What are the reasons for the error can anybody help? Thanks

$\endgroup$

2 Answers 2

1
$\begingroup$

It seems that there are two issues with the file you are trying to read that cause the file to not be a valid JSON file:

  1. The files uses single quotes instead of double quotes.
  2. The file contains multiple JSON objects concatenated by the newline character (\n).

To be able to read in the data correctly in JSON format you therefore need to first replace the single quotes by double quotes, after which you can read in the JSON objects in the file separately instead of all at once.

$\endgroup$
4
  • $\begingroup$ Thanks for the Answer. Though the file can be easily read using pyspark read.json . I doubt the issue is with the file, though I must think there can be some tweaks with in Polars that can be utilized but I am still novice wrt polars $\endgroup$ Feb 1, 2023 at 15:08
  • $\begingroup$ @user18559875 can you shed some light? I happen to follow your comments and answer on this stackoverflow.com/questions/73154392/… $\endgroup$ Feb 1, 2023 at 15:33
  • $\begingroup$ Based on this github issue, it seems that polars uses a very stricht json dialect. You could use pl.from_pandas(pd.read_json("data.json")) to read in the data using pandas and then convert it to a polars dataframe. $\endgroup$
    – Oxbowerce
    Feb 1, 2023 at 16:25
  • $\begingroup$ Thank you for the suggestion. Reading as pandas Dataframe won't work as I am getting memory error with the same $\endgroup$ Feb 2, 2023 at 10:28
0
$\begingroup$

It seems the json uses newline as separator, you can try this:

import polars as pl
df = pl.read_ndjson(json_file)

polars provide read_ndjson for reading json files with newline separator.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.