Feature Request: Add support for semicolon-delimited CSV responses (European locales)

[mrkskk]

Problem Description

When using REDCapR with European REDCap servers, the API returns CSV data with semicolons (;) as field separators instead of commas (,). This causes REDCapR functions to fail because REDCapR functions use readr::read_csv() which is hardcoded to expect comma separators.

Current Behavior

# This fails with European REDCap servers
redcap_variables(redcap_uri = uri, token = token)
# Error: parsing issues due to semicolon separators

The raw CSV returned from European REDCap servers looks like:

field_name;form_name;section_header;field_type;field_label
record_id;demographics;;text;Study ID
name_first;demographics;;text;First Name

But REDCapR expects:

field_name,form_name,section_header,field_type,field_label
record_id,demographics,,text,Study ID
name_first,demographics,,text,First Name

So of course this leads to parsing errors

Why the existing locale parameter doesn't solve this

REDCapR functions accept a locale parameter, but this does not fix the semicolon issue because:

1. readr::read_csv() is hardcoded for commas - The locale parameter only affects decimal separators, date formats, and encoding, not the field delimiter
2. Field delimiter is not part of locale settings - readr::locale() cannot change the fundamental CSV structure that read_csv() expects

Proposed Solution

REDCapR should use rreadr::read_delim() (for a configurable delimiter that would work regardless of locale)

and add a delimiter parameter to REDCapR functions that allows users to specify the CSV delimiter:

# Option 1: Add delimiter parameter
redcap_variables(redcap_uri = uri, token = token, delimiter = ";")

# Option 2: Auto-detect based on locale
redcap_variables(redcap_uri = uri, token = token, locale = readr::locale(encoding = "UTF-8"))

Affected Functions

This issue affects all REDCapR functions that parse CSV responses:

• redcap_read() and related functions
• redcap_variables()
• redcap_instruments()
• redcap_event_instruments()
• redcap_event_read()
• redcap_metadata_read()
• redcap_project_info_read()

Current Workaround

Currently, users must intercept and override REDCapR functions using assignInNamespace() to replace semicolons with commas before CSV parsing. This requires:

1. Creating custom wrapper functions for each affected REDCapR function
2. Using assignInNamespace() to override the original functions
3. Maintaining custom code that duplicates REDCapR's internal logic
4. Regex manipulation of raw CSV text before parsing

This workaround is very cumbersome.

---

[wibeasley]

That's a good idea. What do you think about REDCapR allowing the user to set the csvDelimiter API parameter instead? Here's a screenshot of the documentation and of the API playground.

I'm not opposed to accepting a readr object --I think the idea.

However I'm very hesitant to introduce any feature that I can't write unit tests for. In other words, I can't mimic & test how servers behave in different parts of Europe.

[mrkskk]

I don't particularly like the idea of supporting the CSV delimiter API. For several reasons:

In Europe, commas are often used as decimal markers. If the REDCap database has fields that use commas as decimals, it'll be problematic to use a comma-separated CSV export. That's the reason we often use semicolon-separated files as a "standard" in Europe. Perhaps setting the locale could circumvent this? I don't know.

For me personally, using the CSV delimiter API would not fix anything. The set delimiter CSV API has not been exposed to me as an API user. It's been disabled by the REDCap server admins at my workplace.

I think replacing read_csv() with read_delim() would be the easiest fix. It could default to comma separator, but it would give me the option to set it to semicolon.

I understand that you wont be able to test it, unless of course you can set the API in Redcap to use another delimiter. But is there any reason to think it would break anything? redcap_read (and the other functions) would of course give parsing errors if one set's the wrong delimiter.

This is an aside, but I feel the need to mention it in hope, that if you do change the pckages, that they will be pushed to the stable (CRAN) version swiftly:

I'm entirely dependend on the packages releases to CRAN. My workplace do not allow me to build from source from github or any other repository.

It's a pain to work on these admin-blocked PC. But if I were to work on my own computer with the data, I have would violate data protection laws.

[BlairCooper]

I would expect that REDCap would correctly support using a comma as the separator for European locales. This would be similar to the way it would need to support the comma as a separator for data that contains commas.

The inherent problem seems to be that REDCapR assumes that all servers use the comma as the separator so it does not specify the delimiter when requesting CSV data. A simple solution would seem to be that REDCapR include the csvDelimiter parameter (set to comma) when requesting data in CSV format. Below is the post body from the redcap_variables function with the additional parameter.

post_body <- list(
token = token,
content = "exportFieldNames",
format = "csv",
csvDelimiter = ","
)

[mrkskk]

Adding a csvDelimiter option to REDCapR could be a viable solution—if this ensures the API call actually returns CSV with commas as separators.
My API calls currently default to semicolons.

Does REDCap format comma-separated CSV files correctly when field values themselves contain commas?

My understanding is that readr::read_csv() handles embedded commas properly, but only if the CSV is well-formed—meaning, values containing commas must be enclosed in quotes.

I'm not sure whether REDCap consistently puts quotes around such values in its comma-separated exports. If it doesn't, parsing will break with read_csv().

I can test it out of course.

If REDCap does not reliably generate correctly quoted comma-separated files, I think its best for REDCapR to switch from read_csv() to read_delim(), allowing explicit delimiter control and more robust parsing.

From my prior use of the two readr functions, read_delim with sep=',' functions identically to read_csv()