1
$\begingroup$

Arrow is an in-memory format, so I can have a dataframe in Python backed by the arrow format. So the dataframe sits in memory, can I use that dataframe directly from R without making a copy of the dataframe? Imagine if the dataframe is 100G in size so copying is very inefficient.

I read in the documentation of Arrow that there is zero-copy streaming, but there isn't a way to make the whole dataframe available.

$\endgroup$
2
$\begingroup$

Not currently, though hopefully very soon. https://issues.apache.org/jira/browse/ARROW-3750 is in progress and hopefully will resolve in the coming weeks.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.