[R] R arrow cannot handle labelled data in arrow tables

[EinMaulwurf]

Describe the bug, including details regarding any error messages, version, and platform.

Arrow seems to have issues with labelled data, as it might come from STATA datasets.

Filter labelled data in an arrow table does not work and throws an error. So far, so good, I can deal with that.

library(haven)
library(arrow)
library(tibble)
library(dplyr)

d <- tibble(
  a = labelled(x = 1:5, label = "example variable a"),
  b = labelled(x = 11:15, label = "example variable b")
)

d
#> # A tibble: 5 × 2
#>   a         b        
#>   <int+lbl> <int+lbl>
#> 1 1         11       
#> 2 2         12       
#> 3 3         13       
#> 4 4         14       
#> 5 5         15

d %>%
  as_arrow_table() %>%
  filter(a > 3) %>%
  collect()
#> Error in `compute.arrow_dplyr_query()`:
#> ! NotImplemented: Function 'greater' has no kernel matching input types (<labelled<integer>[0]>: example variable a, <labelled<integer>[0]>: example variable a)

But when leaving out the final collect() to execute the query, the R session crashes completely:

d %>%
  as_arrow_table() %>%
  filter(a > 5)
# R crashes....

Component(s)

R

[amoeba]

Thanks for the report, this needs to be fixed. For the crash, we're crashing when just trying to print:

    --->8--- 
    frame #6: 0x000000011b7ea404 arrow.so`arrow::internal::InvalidValueOrDie(arrow::Status const&) + 244
    frame #7: 0x000000011b7ef470 arrow.so`arrow::Scalar::ToString() const + 1008
    frame #8: 0x000000011ba76668 arrow.so`arrow::compute::Expression::ToString() const::$_4::operator()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>) const + 260
    frame #9: 0x000000011ba75a38 arrow.so`arrow::compute::Expression::ToString() const + 1008
    frame #10: 0x000000011b267b6c arrow.so`_arrow_compute___expr__ToString + 108
    --->8--- 

and with extra error context turned on I see:

ValueOrDie called on an error: NotImplemented: construction from scalar of type <labelled[0]>: example variable a

We could probably also make arrow work seamlessly with labeled dataframes, is that what you were expecting when you ran into this? We could drop the labels when converting to an Arrow table which would be functionally equivalent to this:

> d %>%
+   zap_labels() %>%
+   as_arrow_table() %>%
+   filter(a > 3) %>%
+   collect()
# A tibble: 2 × 2
      a     b
* <int> <int>
1     4    14
2     5    15

[EinMaulwurf]

Yes exactly, I don't care about the labels but could not use open_dataset() for querying my data because of it. I had to load it into RAM and remove the labels as you showed.

Having open_dataset() or as_arrow_table() dropping the labels would be great. Maybe just add a message that labels were dropped, so the users knows what's going on.

Oh and also, there are different kind of labels, see here, so it would probably be best to deal with them all at once.

[thisisnic]

Here's an even more minimal reprex which strips away the context of haven labels:

library(arrow)
x <- 1
class(x) <- "mynewclass"
Expression$scalar(x)

[thisisnic]

Hi @EinMaulwurf, I'm curious whether this temporary fix would work for you on the kind of datasets you're working with or if it's too slow? You can use map_batches() to convert the labelled column to a data type that Arrow can work with, though it'll be slower as it's doing the conversion in R and not Arrow. Here's an example with a small dataset.

library(haven)
library(arrow)
library(tibble)
library(dplyr)

d <- tibble(
  a = labelled(x = 1:5, label = "example variable a"),
  b = labelled(x = 11:15, label = "example variable b")
)

tf <- tempfile()
write_parquet(d, tf)
library(arrow)
open_dataset(tf) %>%
  map_batches(~mutate(., a = as.integer(a))) %>%   # remove labels
  filter(a > 3) %>%
  collect() %>%
  mutate(a = labelled(a, , label = "example variable a")) # restore labels on output data

My best guess is that it would be slower than if it "just worked" keeping the computation in Arrow, but it avoids having to pull everything into RAM at once, as it's applying the conversion to each batch which is being read at a time, rather than the whole dataset in one go.