Skip to content

Fix iconv error on CSV export with special characters (RES-2058)#828

Open
edwh wants to merge 214 commits intodevelopfrom
RES-2058_dash_export_problem
Open

Fix iconv error on CSV export with special characters (RES-2058)#828
edwh wants to merge 214 commits intodevelopfrom
RES-2058_dash_export_problem

Conversation

@edwh
Copy link
Collaborator

@edwh edwh commented Jan 23, 2026

Summary

  • Add //IGNORE flag to all iconv calls using //TRANSLIT in ExportController
  • This prevents "Detected an illegal character in input string" errors on servers with older glibc (2.27) and POSIX locale
  • Add test for CSV export with special characters (emdash, accented characters)

Problem

The network event export was failing on production with:

iconv(): Detected an illegal character in input string

We think this occurred when exporting events for a network containing a group with an emdash (—) in the name. But we can't reproduce the actual problem.

Solution

Nevertheless there is a likely fix to change all iconv('UTF-8', 'ASCII//TRANSLIT', ...) calls to iconv('UTF-8', 'ASCII//TRANSLIT//IGNORE', ...). The //IGNORE flag tells iconv to skip characters that cannot be transliterated rather than throwing an error.

edwh and others added 30 commits June 23, 2025 10:01
…ssions we don't have in Docker Compose environment.
Since PHP 5.4 the short array syntax `[]` may be used instead of `array()`.
edwh and others added 26 commits December 18, 2025 15:49
- Add fr and fr-BE translations for new network strings
- Remove unused translation keys (general.count, show.about_modal_header)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Use "Etiquette" (without accent) to match existing usage
- Use "Repair Café" instead of "groupe" for consistency

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Removed fwrite(STDERR) which was causing TeamCity to mark the test
as failed even when assertions passed. Debug info now only appears
in assertion failure messages.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- /api/v2/networks/{id}/tags returns empty for unauthenticated users
- /api/v2/groups/tags returns empty for unauthenticated users
- Added tests for both endpoints

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The test occasionally fails in CI with all_confirmed_restarters_count
being 1 instead of 0. Added debug info to show group membership state
when assertion fails.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
After making unauthenticated API calls return empty tags, the test
needs to authenticate first to see the tags.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The test expected unauthenticated users to see network tags, but we
changed the API to return empty for unauthenticated users.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Must be authenticated to see tags after unauthenticated API change.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Center title and intro text on landing page
- Reduce line-height on h1 for tighter title spacing
- Add white-space: nowrap to stat headers to prevent line breaks
- Add missing description field to skills seeder (required NOT NULL column)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Use more specific selector (label.btn.btn-checkbox) for higher specificity
- Override background color to gray (#E4E4E4)
- Remove uppercase text-transform
- Add background-image: none to prevent gradient overrides

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add docker:vite, docker:vite:start, docker:vite:stop tasks
- Change HMR host from hardcoded www.example.com to env variable with localhost default
- Update CLAUDE.md with critical Docker task command documentation

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Create EmailValidation.vue component that checks if email already
exists on blur and displays validation message. Replaces inline
HTML with proper Vue component following codebase patterns.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The templates use kebab-case (<categories-table>, <roles-table>) but
components were registered without hyphens. Vue's template-to-component
resolution requires matching names.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Past events were missing timezone in the expanded event data, causing
date/time display issues on group pages. Now explicitly passing
timezone alongside event_date_local, start_local, and end_local.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The route was incorrectly placed in the ensureAPIToken middleware group
but requires authentication since it calls auth()->user(). Moved to the
auth+verifyUserConsent+ensureAPIToken group where it belongs.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Hide hosts section in EventDetails when no hosts assigned
- Update StatsShare to use __() instead of $lang.get/choice methods

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The checkbox in the event invite modal was not functional - clicking it
did nothing. This fix:

1. Adds server-side code to pass group member emails as a data attribute
   on the checkbox element
2. Adds JavaScript handler to populate the email textarea when the
   checkbox is checked, and remove them when unchecked
3. Properly enables the submit button when emails are present

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Create EventInviteModal.vue component to replace blade modal
- Remove jQuery handler from app.js and use Vue event handling
- Add conditional email field to Volunteer API resource (only visible
  to hosts, network coordinators, and admins for privacy)
- Update EventAttendance.vue to use new Vue modal component
- Remove blade modal include from events/view.blade.php
- Add translation for manual_invite_placeholder in en/fr/fr-BE
- Add tests for volunteer email visibility based on user role

Fixes the "Add invites for group members" checkbox not populating
emails when clicked.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Event invite modal: Add Tab key support for email tags, fix CSRF token
  retrieval from meta tag, update placeholder translations
- Notifications: Fix mark as read using event delegation for dynamic elements
- Profile skills: Add 'Ctrl-click to select multiple' hint, hide dropdown arrow
- Landing page: Fix 'Waste prevented' text wrapping with proper styling

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Brings in post-release fixes:
- Event invite modal converted to Vue component with Tab key support
- Fix notifications mark as read (event delegation)
- Fix skills list dropdown arrow and add Ctrl-click hint
- Fix landing page 'Waste prevented' layout wrapping

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add edit functionality for network tags (PUT endpoint, modal, tests)
- Filter global tags from NC view on group edit page
- Preserve global/other network tags when NC saves group
- Add GroupTagsBadges component to display tags on group view
- Rename API parameter 'tag' to 'group_tag' for clarity
- Fix memory exhaustion in loginRegisterStats (use COUNT query)
- Fix stats calculation in Network model (aggregate SQL queries)
- Add translations for tag editing UI
- Update tests and regenerate OpenAPI spec

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add group_tags to Group model's $with array for eager loading
- Update GroupTagsBadges to handle both 'tags' and 'group_tags' keys
- Handle both 'name' and 'tag_name' property names for display

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
NCs should not see global tags from the API - this is a privacy issue.
The getFilteredTagsForUser method now filters the tags returned by the
Group resource based on the authenticated user's role.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Override toArray() to filter group_tags based on user permissions when
the model is serialized to JSON in Blade templates. This ensures NCs
don't see global tags on the group view page.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add //IGNORE flag to all iconv calls using //TRANSLIT in ExportController.
This prevents "Detected an illegal character in input string" errors on
servers with older glibc (2.27) and POSIX locale, which lack transliteration
tables for certain Unicode characters like emdash (—).

The error was triggered when exporting events for a network containing a
group with special characters in the name. The fix ensures untransliterable
characters are skipped rather than causing an error.

Note: The test documents this issue but cannot reproduce the actual error
because the Docker test environment uses glibc 2.41 which has better
transliteration support than the production server's glibc 2.27.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Dockerfile.fly: multi-stage build with PHP 8.2-FPM, nginx, supervisord
- fly.toml / fly-mysql.toml: app and MySQL machine configs
- nginx-fly.conf: nginx config with unix socket and Tigris proxy
- supervisord-fly.conf: process manager for nginx, php-fpm, cron
- startup.sh: background DB migrations, immediate supervisord start
- TrustProxies: trust all proxies (Fly terminates TLS)
- Disable Discourse/Wiki features when URLs not configured
- Add try-catch around Discourse notification fetch
- Remove zz-docker.conf override in Dockerfile

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@edwh edwh force-pushed the RES-2058_dash_export_problem branch from 34f74c3 to 20b4913 Compare February 2, 2026 22:35
name: Setup application
command: |
# Grant timezone access - run directly on MySQL container
docker exec restarters_db mysql -u root -ps3cr3t -e "GRANT SELECT ON mysql.time_zone_name TO 'restarters'@'%';"

Check failure

Code scanning / SonarCloud

MySQL database passwords should not be disclosed High

Make sure this MySQL password gets changed and removed from the code. See more on SonarQube Cloud
- run: sed -i 's/HONEYPOT_DISABLE=.*$/HONEYPOT_DISABLE=TRUE/g' .env
# Setup additional configuration needed for CI (most setup already done by docker_run.sh)
# Set MySQL function creators using session variable (compatible with MySQL 5.7)
docker exec restarters_db mysql -u root -ps3cr3t -e "SET GLOBAL log_bin_trust_function_creators = 1;"

Check failure

Code scanning / SonarCloud

MySQL database passwords should not be disclosed High

Make sure this MySQL password gets changed and removed from the code. See more on SonarQube Cloud
- run: npx playwright install-deps
- run: npm install -D @playwright/test
# Disable ONLY_FULL_GROUP_BY for compatibility with getItemTypes() query
docker exec restarters_db mysql -u root -ps3cr3t -e "SET GLOBAL sql_mode=(SELECT REPLACE(@@sql_mode,'ONLY_FULL_GROUP_BY',''));"

Check failure

Code scanning / SonarCloud

MySQL database passwords should not be disclosed High

Make sure this MySQL password gets changed and removed from the code. See more on SonarQube Cloud
echo "Waiting for services in profile: {{.PROFILE}}"
echo ""

# Wait for core services (always needed)

Check failure

Code scanning / SonarCloud

MySQL database passwords should not be disclosed High

Make sure this MySQL password gets changed and removed from the code. See more on SonarQube Cloud
…r/fr-BE tag translations

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@sonarqubecloud
Copy link

sonarqubecloud bot commented Feb 2, 2026

Quality Gate Failed Quality Gate failed

Failed conditions
18 Security Hotspots
7.8% Duplication on New Code (required ≤ 3%)
E Security Rating on New Code (required ≥ A)
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants