1
$\begingroup$

Arrow is an in-memory format, so I can have a dataframe in Python backed by the arrow format. So the dataframe sits in memory, can I use that dataframe directly from R without making a copy of the dataframe? Imagine if the dataframe is 100G in size so copying is very inefficient.

I read in the documentation of Arrow that there is zero-copy streaming, but there isn't a way to make the whole dataframe available.

$\endgroup$

2 Answers 2

2
$\begingroup$

Not currently, though hopefully very soon. https://issues.apache.org/jira/browse/ARROW-3750 is in progress and hopefully will resolve in the coming weeks.

$\endgroup$
0
$\begingroup$

Yes we can according to the latest Arrow documentation.

  • If Python is your “primary” language and R is the “secondary” language. To use R variables and functions inside a Python process, there's rpy2 package
# this is an example to call an R function with a python pandas data frame argument, and then convert the result R dataframe back into python pandas

# in R: suppose there is an R function 'risk_summary' in an R package 'BCRA', both input and result are data frames
library(BCRA)
result = risk_summary(input)

# in python:
import rpy2.robjects as ro
from rpy2.robjects.packages import importr
from rpy2.robjects import pandas2ri
from rpy2.robjects.conversion import localconverter

# start an R child process in python, import BCRA package in the R process
BCRA = importr('BCRA')

# create the input data frame
input = pandas.DataFrame(#SOME DATA#)

# a wrapping python function to call the 'BCRA.risk_summary' R function
def r_rcra(input):
    with localconverter(ro.default_converter + pandas2ri.converter):
        input_r = ro.conversion.py2rpy(input)

    result_r = BCRA.risk_summary(input_r)

    with localconverter(ro.default_converter + pandas2ri.converter):
        result = ro.conversion.rpy2py(result_r)

    # NA in R will be translated in a special "rpy2.robjects.NA_Character" object in python, so we need to manually translate it into None in Python
    result[result == ro.NA_Character] = None
    return result

result_df = r_rcra(input)
  • If R is your “primary” language and Python is the “secondary” language. To use Python variables and functions inside an R process, there's reticulated package
    • with pandas version 2.0.0, you can create pandas data frames directly in Arrow (instead of converting a pandas data frame into arrow object) and expose it to R
# in python: suppose you would like to create a pandas data frame df from some CSV file
import pandas as pd
df = pd.read_csv(#SOME CSV FILE#, dtype_backend='pyarrow')

# in R:
library(reticulate)
library(arrow)

pd <- import("pandas")
df = pd$read_csv(#SOME CSV FILE#, dtype_backend='pyarrow')


Both use Arrow as the underline in memory data object format.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.