1
$\begingroup$

I am pretty new to polars and have been working around it just to get acquainted.

import polars as pl

filename_data = 'endomondoHR.json'
pl.read_json(filename_data)

Data -> kaggle data

The error I am getting is :

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
c:\Users\shiv\Desktop\test\test.ipynb Cell 13 in <cell line: 4>()
      1 import polars as pl
      2 filename_data = 'endomondoHR.json'
----> 4 pl.read_json(filename_data)

File c:\Users\shiv\miniconda3\lib\site-packages\polars\io.py:985, in read_json(file)
    971 def read_json(file: str | Path | IOBase) -> DataFrame:
    972     """
    973     Read into a DataFrame from a JSON file.
    974 
   (...)
    983 
    984     """
--> 985     return DataFrame._read_json(file)

File c:\Users\shiv\miniconda3\lib\site-packages\polars\internals\dataframe\frame.py:959, in DataFrame._read_json(cls, file)
    956     file = normalise_filepath(file)
    958 self = cls.__new__(cls)
--> 959 self._df = PyDataFrame.read_json(file, False)
    960 return self

RuntimeError: BindingsError: "ComputeError(Owned(\"InvalidEOF\"))"

Useful informations :

Python version : 3.9.13 Windows 11, 16GB RAM, Intel Core i5-11400H, NVIDIA-GeForce GTX 1650 with 4GB GDDR6 dedicated VRAM Data size 6.6 GB

What are the reasons for the error can anybody help? Thanks

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

It seems that there are two issues with the file you are trying to read that cause the file to not be a valid JSON file:

  1. The files uses single quotes instead of double quotes.
  2. The file contains multiple JSON objects concatenated by the newline character (\n).

To be able to read in the data correctly in JSON format you therefore need to first replace the single quotes by double quotes, after which you can read in the JSON objects in the file separately instead of all at once.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for the Answer. Though the file can be easily read using pyspark read.json . I doubt the issue is with the file, though I must think there can be some tweaks with in Polars that can be utilized but I am still novice wrt polars $\endgroup$
    – shivanshu dhawan
    Feb 1 at 15:08
  • $\begingroup$ @user18559875 can you shed some light? I happen to follow your comments and answer on this stackoverflow.com/questions/73154392/… $\endgroup$
    – shivanshu dhawan
    Feb 1 at 15:33
  • $\begingroup$ Based on this github issue, it seems that polars uses a very stricht json dialect. You could use pl.from_pandas(pd.read_json("data.json")) to read in the data using pandas and then convert it to a polars dataframe. $\endgroup$
    – Oxbowerce
    Feb 1 at 16:25
  • $\begingroup$ Thank you for the suggestion. Reading as pandas Dataframe won't work as I am getting memory error with the same $\endgroup$
    – shivanshu dhawan
    Feb 2 at 10:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.