Improvements to utf8 handling.#36
Conversation
…ues I was having with this version of tf not talking utf8 correctly to my utf8 enabled talker, plus it addresses some of the byte-versus-character spacing issues in input processing mentioned in the change logs. It does change the default character to utf8, which I appreciate is an invasive change. I'm supporting that because, at this point in history, it feels more indicated than precipitous.
maths mixed with transpose being a tf function rather than a C function. The results of the limited testing I've done seems pretty nice - transposing of superscripts all works like it should, and edits happen in the right place, look right and have the expected outcome
| * is preserved. */ | ||
| if ((unsigned char)buf[i] < 0x80) { | ||
| if (istrip) buf[i] &= 0x7F; | ||
| if (buf[i] & 0x80) { |
There was a problem hiding this comment.
How would this be true if line 149 is true? This already doesn't apply when WIDECHAR is defined, so this should be reverted.
There was a problem hiding this comment.
Indeed. Having this logic fire anyway appeared to solve a problem, but I'll go and look at this again - either it didn't really solve anything, and was just "solution adjacent", or the WIDECHAR conditional here isn't working as intended. I'll check it out and get back to you. Thanks for engaging.
There was a problem hiding this comment.
I'm on mobile now so I haven't reviewed the other code. but sometimes other places say #ifdef and maybe only this one says #if ?
There was a problem hiding this comment.
It might have been before you enabled WIDECHAR ? Anyway, this will break non-UTF-8 versions, so it should be reverted.
2 participants
Hello,
I had some problems getting your code to talk to a utf8 talker of mine, so I set about making some changes.
Firstly, my apologies for the mass-change pull request; I appreciate that's bad form. Even worse, this is in part because there's been some AI involved. I can understand if you want to tell me where to stick my vibe-coded drivel, but I can at least offer up that I've >30 years experience of doing it the old way :)
Anyway, these changes improve compatibility, but also fix some of (all of?) the issues you mentioned in the release notes, regarding byte-verses-character width processing on input and output, so I thought you might like to take a look. I'm keen to get utf8 support into the core tinyfugue offering, so if you'd like to work together to make that happen, I'm up for that. I think it's a bit silly to be viewing utf8 as new fangled fancy feature at this point in time.