Repository navigation
[R] Parquets do not support list-columns of ordered factors (ordered dictionaries) #49689
Description
Describe the bug, including details regarding any error messages, version, and platform.
With arrow 23.0.1.2:
data <- tibble::tibble(
x = list(
factor(c("a", "b"), levels = c("c", "b", "a"), ordered = TRUE),
factor(c("b", "c"), levels = c("c", "b", "a"), ordered = TRUE)
)
)
arrow::write_parquet(data, tempfile())
#> Error: Invalid: Column data for field 0 with type list<item: dictionary<values=string, indices=int8, ordered=0>> is inconsistent with schema list<item: dictionary<values=string, indices=int8, ordered=1>>No issues if x is a list-column of factors (coerced to unsorted dictionaries).
Upstream bug of ropensci/targets#1291.
Component(s)
R
Activity
aviralgarg05 commentedon Apr 9, 2026
Hi I would like to solve this issue if someone could assign me
thisisnic commentedon Apr 9, 2026
@aviralgarg05 - currently this is an untriaged bug ticket. Before we assign it to anyone, we need to work out if it can be replicated, if it's a bug or not (did this work in previous versions of Arrow - is it a bug or an unimplemented feature?), and then, if it is a bug, work out why it occurs.
Feel free to post your hypothesis about the cause here, and we can walk through your ideas.
lgaborini commentedon Apr 9, 2026
Hello, it was probably an unimplemented or never properly tested feature.
I encountered this first in 2024 with arrow 16.1.0 (forgot to submit the bug report), then never actually checked again.
thisisnic commentedon Apr 10, 2026
I did a little investigation with Claude, and it looks a bit more complex than it first appeared; will paste the output at the bottom of this comment shortly.
Root cause: the C++
DictionaryBuilderdoesn't track theorderedflag.When R converts an ordered factor to an Arrow array, the type is correctly inferred as
dictionary<values=string, indices=int8, ordered=1>. However, the C++DictionaryBuilderreconstructs the dictionary type onFinish()using:return ::arrow::dictionary(indices_builder_.type(), value_type_);The
orderedparameter defaults tofalse, so the built array always hasordered=0regardless of the input type.For top-level ordered factor columns, there's already a workaround in the R package —
RDictionaryConverter::ToChunkedArray()detects the mismatch and patches the type back toordered=true.However, when an ordered factor is inside a list column,
ListBuilder::Finish()internally finishes its childDictionaryBuilder, producingordered=0. TheRDictionaryConverter::ToChunkedArray()workaround is never called for child converters — only the list converter'sToChunkedArray()runs. So the schema sayslist<dictionary<..., ordered=1>>but the data haslist<dictionary<..., ordered=0>>, and validation catches the mismatch.This has likely been broken since ordered dictionary support was added — as the reporter notes, it reproduces as far back as Arrow 16.1.0.
Proposed fix:
The fix belongs in C++, since adding another R-side workaround would be fragile (the
ListBuilderalso reconstructs its own type from the child builder's type viaListBuilder::type(), so both the list type and child type end up wrong).We should open a separate C++ ticket for this, but here's the full picture of what needs to change:
MakeDictionaryBuildercurrently decomposes theDictionaryTypeinto separateindex_typeandvalue_typebefore passing them to the builder viaDictionaryBuilderCase. Theorderedflag is lost at this point. It should pass the fullDictionaryType(or at minimum, theorderedflag) through to the builder.DictionaryBuilderBasestoresvalue_type_but has noordered_member. It needs to store the ordered flag (or the full dictionary type).DictionaryBuilderBase::type()currently returns::arrow::dictionary(indices_builder_.type(), value_type_)which defaultsorderedtofalse. It needs to pass the stored flag:::arrow::dictionary(indices_builder_.type(), value_type_, ordered_).DictionaryBuilderBase::FinishInternal()sets the output type via(*out)->type = type(), so fixingtype()would fixFinish()automatically.Once the C++ fix is in place, the R workaround and its TODO can be removed.
Note:
DictionaryBuilderBasehas multiple constructor overloads that would all need to accept the flag (or a default offalsefor backward compatibility).GH-49689: [R][C++] Parquets do not support list-columns of ordered fa…
thisisnic commentedon Jun 11, 2026
Issue resolved by pull request 49937
#49937