Skip to content

added a function to handle decoding as utf-16 as a fallback if decodi…#917

Merged
yuanzhou merged 1 commit intodev-integratefrom
Derek-Furst/fix-utf-bug
Apr 15, 2026
Merged

added a function to handle decoding as utf-16 as a fallback if decodi…#917
yuanzhou merged 1 commit intodev-integratefrom
Derek-Furst/fix-utf-bug

Conversation

@DerekFurstPitt
Copy link
Copy Markdown
Contributor

…ng with utf-8 fails

@DerekFurstPitt DerekFurstPitt requested a review from yuanzhou April 15, 2026 16:45
Comment thread src/app.py Dismissed
Comment thread src/app.py Dismissed
@DerekFurstPitt
Copy link
Copy Markdown
Contributor Author

@yuanzhou the alert its showing seems to have an issue with this:

def open_tsv(path):
    try:
        f =  open(path, newline='', encoding='utf-8-sig')
        f.read()
        f.seek(0)
    except UnicodeDecodeError:
        return open(path, newline='', encoding='utf-16')

particularly "path". It seems to think that this is a user provided value. However path actually comes from

 file_location = commons_file_helper.ensureTrailingSlash(
        app.config['FILE_UPLOAD_TEMP_DIR']) + temp_id + os.sep + file.filename
    tsvfile = open_tsv(file_location)

So not a user defined value unless i'm misunderstanding it

@yuanzhou yuanzhou merged commit a9ff1d7 into dev-integrate Apr 15, 2026
4 checks passed
@yuanzhou yuanzhou deleted the Derek-Furst/fix-utf-bug branch April 21, 2026 14:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants