Case changing commands over multi-line ranges lose newlines

[donaldcallen]

Problem

Assume these lines are present

now is the time
for all good men

If I position the cursor on the first character of the first line and then invoke 'shift-v' to go into VISUAL-LINE mode and then invoke 'j' to extend the VISUAL-LINE range to both lines and then issue the command 'gU', I end up with this:

NOW IS THE TIMEFOR ALL GOOD MEN

Something similar happens if use the 'gu' command. So whether converting to upper or lower case, this sequence of commands results in the loss of the newlines.

This works correctly in vim, nvim, kakoune, etc.

Steps to reproduce

No response

vis version (vis -v)

vis v0.9-384-g93f2c297 +curses +lua +tre +acl

Terminal name/version

No response

TERM environment variable

No response

[fischerling]

I can not reproduce this behaviour on my arch system using GNU coreutils.

vis uses tr(1) to change the case of all characters in the selection.
What version of tr(1) are you using?

Can you confirm that echo -e "now is the time\nfor all good men" | tr '[:lower:]' '[:upper:]', does keep the newline on your system?

[donaldcallen]

dca@giovanni:/home/dca$ echo -e "now is the time\nfor all good men" | tr '[:lower:]' '[:upper:]'
NOW IS THE TIME
FOR ALL GOOD MEN

dca@giovanni:/tmp$ tr --version
tr (GNU coreutils) 9.10

My initial post was based on the behavior of vis as compiled from the sources a few days ago. So I installed the released version on my system and tested that. Same problem.

[donaldcallen]

I should add that my system is up-to-date as of a few days ago.

[fischerling]

Hm I build the same version as mentioned in your report vis v0.9-384-g93f2c297 +curses +lua +tre +acl from 93f2c29.
And could not reproduce the behavior.
If you use gU is the last command in the command buffer :|tr '[:lower:]' '[:upper:]'?
You can verify by pressing : and <Up>.

Have you tried it without your local configuration?

[fischerling]

So it was my local config.h file still using :|tr '[:lower:]' '[:upper:]' instead of |awk '{printf \"%s\", toupper($0)}'<Enter>" introduced in d3e4af1.

So this should definitly be fixed.

[donaldcallen]

My (default) config.h uses awk for gu and gU.

Good work identifying the problem. Pending a fix in the git repository, I can fix it in my own config.h.

[donaldcallen]

It seems odd that this change was made for gu and gU for unicode-support reasons, but not for g~.

[donaldcallen]

And the documentation for Gnu Awk's toupper and tolower functions says that they only change alphabetic characters.

dca@giovanni:/home/dca/Software/vis$ echo "now is the time\nfor all" | awk -e '{print toupper($0)}' -
NOW IS THE TIME
FOR ALL
dca@giovanni:/home/dca/Software/vis$ echo 'now is the time\nfor all' | awk -e '{print toupper($0)}' -
NOW IS THE TIME
FOR ALL

So I think some work needs to be done to understand why this change was unsuccessful. It also suggests expanding the tests to include multi-line cases.

[fischerling]

The problem is that awk is line-based. It strips the newline from the input lines and print always appends a newline back.
However we need to preserve all newlines in the input, even a trailing one but we must not add a trailing one if none was present in the input.

now is the time\nfor all -> NOW IS THE TIME\nFOR ALL
now is the time\nfor all\n -> NOW IS THE TIME\nFOR ALL\n

Additionally we want to be POSIX compatible and not rely on GNU awk features.

[donaldcallen]

Yes, I understand that vis can't assume GNU Awk. I mentioned it because on an Arch Linux system like mine, GNU Awk is what you get and contrary to its documentation and my little tests, which handled the newlines correctly, vis still exhibited the issue I raised. So clearly something more was going on in how vis was making use of awk.

[fischerling]

Yes, I understand that vis can't assume GNU Awk. I mentioned it because on an Arch Linux system like mine, GNU Awk is what you get and contrary to its documentation and my little tests, which handled the newlines correctly, vis still exhibited the issue I raised. So clearly something more was going on in how vis was making use of awk.

But your tests do not test the input without trailing newline, which does not work using awk print.

[donaldcallen]

Now I am really confused. Why are you fixated on the fact that awk print always appends a newline to its output when the change you cited that broke things uses awk printf, not print, which does not unconditionally append newlines to its output?

If you were referring to my use of print in my tests, those tests were intended to see if (GNU) awk damaged an interior newline, which was the original problem I raised? And it turns out it doesn't, so, as I said previously, something more complex than my simple tests is happening between vis and awk.

[fischerling]

Now I am really confused. Why are you fixated on the fact that awk print always appends a newline to its output when the change you cited that broke things uses awk printf, not print, which does not unconditionally append newlines to its output?

Both versions are broken.
We can not use print because in vis the selection we use gU on can be arbitrary.

foo
bar

In this example the selected text is "oo\nba" if we replace this with the output of awk which appends a newline `"OO\nBA\n" the resulting file content would be:

fOO
BA
r

Both versions are broken. The current version swallows interior newlines. All other versions using POSIX awk will append a new line or will not preserve a leading one if present.

I have no good solution for this problem. Maybe we should implement this in vis.

[TwoF1nger]

Related: #1238