Character sequences used for string representation of boxes should be treated as single characters in string character-wise manipulation operations.

[mmatera]

The escaped sequences lije \!, \), and \( in WMA are encoded as the special characters with Unicode encoding:

sequence	character	meaning
")"	"\uf7c0"	End raw boxes
"!"	"\uf7c1"	Parse what follows as a raw box expression
"@"	"\uf7c2"	SqrtBox
"%"	"\uf7c5"	last argument (like SqrtBox[A,B] or UnderOverScriptBox[A,B,C]
"^"	"\uf7c6"	SuperscriptBox
"&"	"\uf7c7"	UnderscriptBox
"*"	"\uf7c8"	Construct boxes from input
"("	"\uf7c9"	Open Raw Boxes
"_"	"\uf7cA"	SubscriptBox
"+"	"\uf7cB"	UnderscriptBox
"/"	"\uf7cC"	FractionBox
"`"	"\uf7cD"	FormBox

(see https://reference.wolfram.com/language/tutorial/TextualInputAndOutput.html#28564)

When some of these sequences appear in a string, they are stored as a single character, disregarding the expression is a valid box structure. For example, in WMA,

In[1]:= StringLength["\!A"]
Out[1] = 2

In[2]:= StringTake["\!A",{1}]
Out[2] = \!

In[3]:= StringLength["\!\(A\)"]
Out[3] = 4

In Mathics3, we get

In[1]:=  StringLength["\!A"]
Out[1]= 2

In[2]:=  StringTake["\!A",{1}]
Out[2]= "!"

In[3]:=  StringLength["\!\(A\)"]
Out[3]= 6

When a valid sequence of these characters appears inside a String, the string is converted to boxes as if blocks ... \!\( ... \) ... where a RowBox expression:

In[5]:= "This is a GridBox with a subscript: \!\(\*GridBox[{{a,b},{c,d}}]\_n\), 
yes, it is!"                                                                    

Out[5]= This is a GridBox with a subscript: a   b , yes, it is!
                                                 n
                                            c   d

Nothing of this currently works in Mathics3.

The ability to generate and render strings in this way would be useful to serialize box expressions in the same way that WMA does. Also, this is related to the expected output in ToString expressions (see #1678).

[rocky]

This is probably related to #1622. Both have a part in the mathics scanner. Here, there is also something going on outside of mathics-scanner inside strings as well.

[rocky]

The escaped sequences lije \!, \), and \( in WMA are encoded as the special characters with Unicode encoding:

sequence character meaning
")" "\uf7c0" End raw boxes
"!" "\uf7c1" Parse what follows as a raw box expression
"@" "\uf7c2" SqrtBox
"%" "\uf7c5" last argument (like SqrtBox[A,B] or UnderOverScriptBox[A,B,C]
"^" "\uf7c6" SuperscriptBox
"&" "\uf7c7" UnderscriptBox
"*" "\uf7c8" Construct boxes from input
"(" "\uf7c9" Open Raw Boxes
"_" "\uf7cA" SubscriptBox
"+" "\uf7cB" UnderscriptBox
"/" "\uf7cC" FractionBox
"`" "\uf7cD" FormBox
(see https://reference.wolfram.com/language/tutorial/TextualInputAndOutput.html#28564)

The Unicode characters above are defined in a Wolfram-specific character encoding that does not generally have meaning otherwise. In a terminal, I see a glyph with the hex code one nybble on each of two lines where the character would appear.

I don't know what we're going to do in Mathics3 for terminal devices.

[mmatera]

These characters are not used for show, but for the internal string representation. When the string is rendered, if the string is not a valid box expression, they are replaced back with an ASCII sequence.

[mmatera]

This is probably related to #1622. Both have a part in the mathics scanner. Here, there is also something going on outside of mathics-scanner inside strings as well.

Indeed, this is similar to that case, and the case of using named characters with the notation \[...]. The escaped sequence should be replaced by the corresponding Unicode character.

[rocky]

. The escaped sequence should be replaced by the corresponding Unicode character.

I am not totally sure about this. Another possibility might be to run the contents of the string through the scanner or a component of the scanner again.

[rocky]

These characters are not used for show, but for the internal string representation.

That is what tokens are for. They take characters or letters from a sequence of characters and combine or compress them into word-like, single-symbol "tokens".

[rocky]

I looked at and thought about this some more.

...
When some of these sequences appear in a string, they are stored as a single character, disregarding the expression is a valid box structure. For example, in WMA,

In[1]:= StringLength["\!A"]
Out[1] = 2

In[2]:= StringTake["\!A",{1}]
Out[2] = \!

In[3]:= StringLength["\!\(A\)"]
Out[3] = 4

The function Characters[] gives more detailed information:

In[3]:= Characters["\!\(A\)"]                                            

Out[3]= {\!, \(, A, \)}

Note that while "\!" and "\(" appear as a single entity, there is not any Unicode shown. Internally, there might be a unicode symbol involved as a way to implement this, but from what I have gleaned from how Mathematica works, although the scanning and parsing is home grown, it does follow conventional scanning and parsing principles. This suggests to me that some sort of more conventional tokenization process may be going on.

Here is how the CodeParser package handles:

In[3]:= CodeTokenize["\!\(A\)"]                                          

Out[3]= {LeafNode[Token`LinearSyntax`Bang, \!,  
>     <|Source -> {{1, 1}, {1, 2}}|>],  
>    LeafNode[Token`LinearSyntaxBlob, \(A\), 
>     <|Source -> {{1, 2}, {1, 5}}|>]}
In[4]:= CodeTokenize["\"\!\(A\)\""]                                      

Out[4]= {LeafNode[String, "A", <|Source -> {{1, 1}, {1, 7}}|>]}

Whether this is correct or not, I don't know. If it is correct, string operations would have to unpack LinearSyntaxBlob into 3 parts.

This may be why this package is still marked "Experimental" and probably not used internally as the main scanner and parser.

When a valid sequence of these characters appears inside a String, the string is converted to boxes as if blocks ... \!\( ... \) ... where a RowBox expression:

In[5]:= "This is a GridBox with a subscript: \!\(\*GridBox[{{a,b},{c,d}}]\_n\), 
yes, it is!"                                                                    

Out[5]= This is a GridBox with a subscript: a   b , yes, it is!
                                                 n
                                            c   d

Nothing of this currently works in Mathics3.

Yes, we don't handle \* yet which most probably makes another pass through both a scanner and a parser.

The ability to generate and render strings in this way would be useful to serialize box expressions in the same way that WMA does. Also, this is related to the expected output in ToString expressions (see #1678).

I think we need to separate the Characters[] (or StringLength[]) aspect in isolation from the \* aspect. \* which involves parsing probably builds on the scanning part of #1622.

So let's work on #1622 first, and then come back to what remains here.

There is a bug part of issue which is valid. That part might just be that \* is not implemented.

The issue title however suggests an implementation which is probably not correct.

[mmatera]

The function Characters[] gives more detailed information:

In[3]:= Characters["!(A)"]

Out[3]= {!, (, A, )}

Note that while "!" and "(" appear as a single entity, there is not any Unicode shown. Internally, there might be a unicode symbol involved as a way to implement this, but from what I have gleaned from how Mathematica works, although the scanning and parsing is home grown, it does follow conventional scanning and parsing principles. This suggests to me that no Unicode symbol is used, but rather some sort of more conventional tokenization process is going on.

The point is that \!, \( and \) are treated as single characters that, when they are printed, they are shown as two characters, in a similar way that \[Rule] is stored as a single character ("\:f522"), and when it is printed in ASCII it is converted into ->:

In[1]:= "A \[Rule] B"                                                          

Out[1]= A  B

In[2]:= ToString["A \[Rule] B"]                                                

Out[2]= A  B

In[3]:= ToString["A \[Rule] B", CharacterEncoding->"ASCII"]                    

Out[3]= A -> B

We could reimplement string-handling functions to treat these special characters as a single character, but that would be reinventing Unicode.

So let's work on #1622 first, and then come back to what remains here.

Looks like a good plan

There is a bug part of issue which is valid. That part might just be that \* is not implemented.

As I see, there are two parts. One is the lack of implementation of certain symbols. The other is how these sequences are treated in character-wise string functions.

The issue title however suggests an implementation which is probably not correct.

Indeed, it does. I guess there are other possibilities, but what looks more natural to me -because it does not require rewriting all the character-wise string functions to handle special cases- is to store these character sequences as special characters.

[rocky]

... Indeed, it does. I guess there are other possibilities, but what looks more natural to me -because ...

We've encountered this kind of thing several times before.

Suppose one doesn't understand group theory and tries to solve a Rubik's Cube. It looks natural to solve this one face at a time and on a surface, by consecutive corner and edge cubes.

However, that is not how anyone except a beginner approaches the situation.

What can be useful here is to understand when there might be something deeper going on, and ask how this kind of thing is typically handled, and then, if possible, hook into the concepts, ideas, and technology of existing art.

The difference between using Unicode symbols to represent a token may not be all that different here. But following this implementation as a proxy for an unstated concept can lead to dead ends in handling harder things, should it arise.

That kind of thing was certainly the situation when attempting to handle the special syntax for boxing operators and expressions.

[rocky]

The point is that !, ( and ) are treated as single characters that, when they are printed,

An issue title involving that, then, would make an excellent issue title. Note that it is not about being "printed" it is about them being considered atoms of the string. The StringLength[] example shows more clearly that it is not about printing since there is no display. But in both Characters and StringLength, the issue concerns components of the internal string object.

And when framed this way, this is also the underlying issue behind #1622.

[mmatera]

... Indeed, it does. I guess there are other possibilities, but what looks more natural to me -because ...

We've encountered this kind of thing several times before.

Suppose one doesn't understand group theory and tries to solve a Rubik's Cube. It looks natural to solve this one face at a time and on a surface, by consecutive corner and edge cubes.

I know that. I just observed that it looks like when WMA stores a string that was input, including these special sequences, it behaves as if it stores one special character for each of these sequences. And that mimics such behavior looks like a reimplementation of how Unicode strings are handled at the low level. With this, I am not proposing a concrete implementation in the Mathics3 scanner/tokenizer/parser.

[rocky]

Investigating further, it does look like the Unicode codes are used:

In[42]:= ToCharacterCode["\!A"]                                           
Out[42]= {63425, 65}

And 63425 is hexadecimal '0xf7c1'.

So you were correct in your approach to use Unicode inside the value of String, and I am wrong.

We are already doing this kind of thing for operators like \[Nand]. (I picked an example where our Unicode matches WMAs. We mismatch unicode for Rule, but the same idea occurs.)

To misquote Magritte: sometimes a pipe is a pipe.

[rocky]

Continuing... if \! were a named character, then simply adding this named character would go through without further modification. The problem is that this is an unnamed character. Google suggests that the common (un)name for this is "Raw Linear Syntax Marker." which I later come to understand is a bit vague.

CodeTokenize uses a similar, but more specific name LinearSyntax`Bang which I think I like better.

One approach, then, is to add a table for unnamed combinations of characters in mathics-scanner. And treat those similar to named characters in mathics-core.

@mmatera thoughts?

[rocky]

Ok. I think I now understand the idea behind the CodeParser package. What is seen using CodeTokenize[] is detecting any sort of \( ... \) syntax, and then passing that off as one token, a LinearSyntaxBlob. The parser then has to deal with this as a separate parse if it wants to break it down.

And when I run CodeParse["\(\"a\"\)"], all sorts of complicated things involving ConfidenceLevel and CodeActions start to appear.

It looks like we may run into complications in the future. We may need to think this through more carefully.