Encoding, Decoding, and Cross-Language Data Transfer

Introduction

Data pipelines in rixpress often require controlling how objects are stored and restored, especially when dealing with:

1. Non-standard R objects (e.g., machine learning models, large tables).
2. Multiple file formats (CSV, qs compressed files, etc.).
3. Cross-language workflows mixing R and Python.

This vignette focuses on encoding and decoding in R, and on transferring data between R and Python using rxp_py2r() and rxp_r2py().

Custom Encoding and Decoding in R

By default, rixpress uses saveRDS() and readRDS(). You can override this to handle different formats or complex objects:

library(rixpress)

# Encode output as CSV instead of RDS
d2 <- rxp_r(
  mtcars_head,
  my_head(mtcars_am, 100),
  user_functions = "my_head.R",
  nix_env = "default.nix",
  encoder = write.csv
)

# Encode as qs, decode input from CSV
d3 <- rxp_r(
  mtcars_tail,
  my_tail(mtcars_head),
  user_functions = "my_tail.R",
  nix_env = "default2.nix",
  encoder = qs::qsave,
  decoder = read.csv
)

# Decode multiple upstream objects with different decoders
d4 <- rxp_r(
  mtcars_mpg,
  full_join(mtcars_tail, mtcars_head),
  nix_env = "default2.nix",
  decoder = c(
    mtcars_tail = "qs::qread",
    mtcars_head = "read.csv"
  )
)

Key points:

• encoder controls how this step’s output is stored.
• decoder specifies how to read inputs from upstream derivations.
• You can assign different decoders per upstream object using a named vector.

As shown in the examples above, you can pass a function or a string representation of the function to encoder and decoder.

By encoding the object in a cross-language format, it is possible to pass it to another language. For example, read a csv file using Julia, encode it to Arrow and read it back in R:

library(rixpress)

list(
  rxp_jl_file(
    mtcars,
    # Assume here that mtcars.csv is separated by "|" instead of ","
    path = "data/mtcars.csv",
    read_function = "read_csv",
    user_functions = "functions.jl",
    encoder = "write_arrow"
    # read_csv and write_arrow are both
    # defined in the functions.jl script
    # and looks like this:

    #function write_arrow(df::DataFrame, filename::String)
    #    Arrow.write(filename, df)
    #end

    #function read_csv(path::String)
    #    df = CSV.read(path, DataFrame; delim="|")
    #return df
    #end

  ),

  rxp_r(
    mtcars2,
    select(mtcars, am, cyl, mpg),
    decoder = "read_feather"
  )
) |>
  rxp_populate()

You can find this example here. You can use the same approach to transfer data to Python (well, from and to any of the three supported languages).

Cross-Language Data Transfer: R ↔︎ Python

In the specific case of transferring objects (data, lists, vectors, arrays, etc.) between R and Python, it also possible to use reticulate’s built-in conversion by using rxp_py2r() and rxp_r2py(). These functions enable seamless movement of objects between R and Python:

library(rixpress)

# Python step producing pandas DataFrame
d1 <- rxp_py(
  name = mtcars_pl_am,
  expr = "mtcars_pl.filter(polars.col('am') == 1).to_pandas()"
)

# Transfer Python -> R
d2 <- rxp_py2r(
  name = mtcars_am,
  expr = mtcars_pl_am
)

# R step processing the data
d3 <- rxp_r(
  name = mtcars_head,
  expr = my_head(mtcars_am),
  user_functions = "functions.R"
)

# Transfer R -> Python
d3_1 <- rxp_r2py(
  name = mtcars_head_py,
  expr = mtcars_head
)

For this to work, you need to add reticulate to the pipeline’s execution environment.

Summary

• Use encoder/decoder for non-RDS objects (CSV, qs, Keras models) and to pass data to and from different languages.
• Explicitly set decoders per upstream object to avoid mismatches.
• Use rxp_py2r() and rxp_r2py() if you want to re-use reticulate’s bulit-in conversion (useful for more complex objects).