Skip to content

GH-51659: [R] Expose ignore_extra_columns and pad_short_rows CSV pars… - #51662

Open
rajaryan2007 wants to merge 2 commits into
apache:mainfrom
rajaryan2007:r-csv-parse-extra-columns-options
Open

rajaryan2007 wants to merge 2 commits into
apache:mainfrom
rajaryan2007:r-csv-parse-extra-columns-options

Conversation

@rajaryan2007

@rajaryan2007 rajaryan2007 commented Sep 30, 2026 •

Copy link
Copy Markdown

Rationale for this change

Arrow C++ 26.0.0 added two new arrow::csv::ParseOptions fields:

Neither could be set from R, so read_csv_arrow() and open_csv_dataset()
error on ragged CSV with no way to opt into the C++ behaviour.

# before: Error: CSV parse error: Expected 3 columns, got 2
read_csv_arrow(I("a,b,c\n1,2,9\n3"))

# after
read_csv_arrow(I("a,b,c\n1,2,9\n3"),
               parse_options = csv_parse_options(pad_short_rows = TRUE))
#>   a  b  c
#> 1 1  2  9
#> 2 3 NA NA

Are these changes tested?

Yes. test-csv.R gains pad_short_rows and ignore_extra_columns parse options, covering each option alone, both together, the plain-list form, and the CsvParseOptions$create() entry point. It also asserts ragged input still errors by default, pinning the new defaults.

Verified locally with R 4.4.3: the full testthat suite passes (6,561 assertions, 0 failures, 0 errors), and the new test was confirmed to fail against the pre-change r/src/csv.cpp.

Are there any user-facing changes?

Additive: csv_parse_options() and CsvParseOptions$create() accept two new arguments, both defaulting to FALSE to match the C++ defaults, so existing code is unaffected.

Was AI used for this PR?

i used ai for build and some someother help .

PR code and description written by:

  • Human
  • AI

Reviewed before submission by:

  • Human
  • AI
  • Not reviewed

@github-actions github-actions Bot added the awaiting review Awaiting review label Sep 30, 2026
@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #51659 has been automatically assigned in GitHub to PR creator.

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #51659 has been automatically assigned in GitHub to PR creator.

…V parse options

Arrow C++ 26.0.0 adds pad_short_rows and ignore_extra_columns to
arrow::csv::ParseOptions. Neither could be set from R, so add them to
csv_parse_options() and pass them through from csv.cpp.
CsvParseOptions$create is an alias of csv_parse_options(), so both entry
points accept the new arguments.
@rajaryan2007
rajaryan2007 force-pushed the r-csv-parse-extra-columns-options branch from b52067b to 23a8d17 Compare September 30, 2026 20:32
@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #51659 has been automatically assigned in GitHub to PR creator.

4 similar comments
@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #51659 has been automatically assigned in GitHub to PR creator.

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #51659 has been automatically assigned in GitHub to PR creator.

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #51659 has been automatically assigned in GitHub to PR creator.

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #51659 has been automatically assigned in GitHub to PR creator.

@rajaryan2007
rajaryan2007 marked this pull request as ready for review September 30, 2026 20:47
@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #51659 has been automatically assigned in GitHub to PR creator.

2 similar comments
@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #51659 has been automatically assigned in GitHub to PR creator.

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #51659 has been automatically assigned in GitHub to PR creator.

Comment thread r/NEWS.md Outdated

@thisisnic thisisnic left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking the time to make this PR! This is almost there, just a few changes to suggest. I've put some of them inline.

One more thing is that the CsvReadOptions docs block above has its own list of the arguments CsvParseOptions$create() takes, so please could you add pad_short_rows and ignore_extra_columns there too?

Let me know if you have any questions!

Comment thread r/R/csv.R
Comment thread r/tests/testthat/test-csv.R Outdated
- Credit contributor in NEWS.md
- Document pad_short_rows and ignore_extra_columns in CsvParseOptions
- Regenerate .Rd documentation files
- Simplify test to avoid bare expect_error
@rajaryan2007

Copy link
Copy Markdown
Author

done : )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants