Add X++ support

[andrewschmidt-a]

X++ is a language developed and maintained by Microsoft. It compiles down to CIL. It is considered a .NET language. This is used exclusively within Microsoft's ERP product called Dynamics 365 and its partner ecosystem.

Reference links:

• https://learn.microsoft.com/en-us/dynamics365/fin-ops-core/dev-itpro/dev-ref/xpp-language-reference
• https://learn.microsoft.com/en-us/dynamicsax-2012/developer/x-syntax

[Anteru]

Thanks for the contribution. Can you please add an example file as well and generate "golden" test output for it? Reviewing regexes without seeing them run is hard :)

[andrewschmidt-a]

@Anteru, thanks for the feedback I have included a test and its output

[andrewschmidt-a]

@Anteru , I tried to run mapping for my change because the build failed with this error
Please run "make mapfiles" and add the changes to a commit.

Not sure how it was different than my manual change but I committed it.

[jeanas]

`git diff` should show that.

[jeanas]

This fails CI, apparently a missing r string prefix.

[andrewschmidt-a]

I found the CI pipeline and ran the commands myself, I think this should be covered. RegexLint didnt give me an affirmative output but I think it passed

[jeanas]

Overall this looks quite well thought out to me, with two reservations:

• I'm not sure the Unicode level configurability is worth the code, see comment inline.

• While testing on large files (basically with for i in {1..1000}; do echo tests/examplefiles/xpp/test.xpp >> bigtest.xpp; done), I experienced RecursionErrors. This is because the lexer sometimes recurses with using(this) for way too much code. For example, in r'(\s*)\b(else|if)\b(.*)', the .* part will match everything after the if in the file (since you have DOTALL). Please fix this and check that it works on a large file.

[andrewschmidt-a]

@jeanas , thanks for the suggestions. I have tested with a large file and am not experiencing a recursion error.

[andrewschmidt-a]

Any suggestions to resolve the REGEXLint issue? I use the full set and it complains about null characters not being supported. This is the same regex used in the CSharpLexer, but I think it does not complain about it cause the basic level is default. If I set mine to the basic level, the regex errors go away.

[jeanas]

I've submitted a fix to regexlint for this. See thatch/regexlint#49

[jeanas]

Ah, I just noticed that CI uses the fork https://github.com/pygments/regexlint, not the original https://github.com/thatch/regexlint. @birkenfeld What is the history behind this?

[birkenfeld]

Probably just switched to a repo I had access to while I was working on the updates for thatch/regexlint#40 - after that was merged I should have switched back to upstream.

[jeanas]

In the meantime, I've added a workaround so we can merge this. It only affects code with null characters, which is quite unimportant to get right.

[birkenfeld]

NUL chars can be escaped on the regex level with r"\x00" anyway, no?

[jeanas]

Ah, right, I forgot about this. Well, eventually, we'll use regex anyway, which supports matching on Unicode character classes 😉