Render graphical POUs as readable text on export

[kehindepeters]

Ladder and Function Block Diagram POUs export as native CODESYS xml, which git can store but nobody can review. This renders them to text alongside that xml, so graphical logic gets the same diffable history the .st files already have.

A graphical POU now exports Main.xml as before, plus Main.txt:

PROGRAM LD_TEST
VAR
	Sensor1: BOOL;
	PowerOn: BOOL;
	TON_0: TON;
	CTU_0: CTU;
	PowerOff: BOOL;
END_VAR

(* Network 1: Try me Codesys I swear *)
(* comment without backslash *)
│    Sensor1   Sensor2  PowerOn
├──┬───┤ ├───┬───┤/├──────(S)─────┤
│  │ sensor3 │
│  └───┤ ├───┘

(* Network 3: header text *)
(* Comment *)
│                   TON_0 : TON              CTU_0 : CTU
│   PowerOn        ┌───────────┐            ┌───────────┐  PowerOff
├─────┤ ├──────────┤IN        Q├────────────┤CU        Q├────(R)──────┤
│            T#5S──┤PT       ET│  PowerOff──┤RESET    CV│
│                  └───────────┘  10────────┤PV         │
│                                           └───────────┘

Each network is headed by its title, as the editor heads it, with the network's comment on the line below. Variables are greppable for the first time: Ctrl+Shift+F for a tag now finds every network that touches it.

Numbering matches the editor

A network in the file lines up with the one in CODESYS, so opening "Network 5" in the editor and reading "Network 5" in the file gives you the same logic.

That takes reading the native xml as well as the PLCopen export. CODESYS leaves out of the PLCopen export every network that carries no elements: an out-commented one goes entirely, its comment with it, and so does an empty one. Numbering what survives 1..n drifts from what the editor shows, silently.

So the native xml written beside the rendering is the authority for the structure — how many networks there are, their order, and each one's comment, title and label — and the PLCopen export supplies only the logic. The two are joined by matching the networks that carry logic, in order, so there is no position to guess at. Networks the export left out keep their number and say why they have no diagram:

(* Network 2: Safely power off PLC when ignition is lower than 5V *)
(* out-commented in CODESYS - does not execute; diagram not exported, see the native xml *)

If the two cannot be lined up, the file says so at the top and falls back to numbering in export order, rather than showing numbers that quietly disagree with the editor.

The contract

The .txt is derived and read-only. The native xml stays the only thing Import From Files reads, so the round trip is untouched and editing the .txt achieves nothing.

That safety is incidental — import_directory_child dispatches on .xml and .st, so .txt matches no branch. Incidental safety breaks silently, so test_export.py pins it down, with a control asserting the native xml still imports (otherwise the test would pass even if dispatch were broken outright).

Design notes

• Layout comes from topology, not coordinates. Moving a block in the CODESYS editor produces no diff. A test shifts every <position x=> by 500px and asserts byte-identical output. Real CODESYS exports write x="0" y="0" on everything anyway, so coordinates were never usable.
• Rendering goes via export_xml (PLCopen), not the native format. PLCopen has a published schema for graphical bodies; the native format does not. The native xml is read only for the network list described above.
• A rendering failure is reported, not raised. The native xml is already written and complete; a diagram we cannot draw is not a reason to fail an otherwise good export. This bends the repo's usual loud-failure stance deliberately, because the artifact is optional. It paid off: the first real-project run failed on every POU and still produced a correct export. The summary counts how many POUs were rendered, how many failed, and which were skipped for having no renderable body.
• Glyphs are escape sequences, never literal characters. IronPython 2.7 enforces PEP 263 and would refuse to load a source file containing a non-ASCII byte, so a literal box character anywhere in src/ stops CODESYS loading the scripts at all. --charset ascii is built and tested for terminals that mangle box drawing.
• The declaration is taken verbatim from CODESYS, not rebuilt. It comes from obj.textual_declaration.text, which is the only form carrying comments, pragmas and attributes — and a pragma like qualified_only changes what the code means. The structured interface has nowhere to put any of the three, so it is kept only as a fallback for builds that do not offer the plaintext declaration. A rendering that falls back says so on its first line, and a variable whose type the export omits reads UNKNOWN rather than being assumed to be a BOOL. Confirmed on a real project: 22 of 25 POUs carry their header block, inline comments and the literals as the author typed them (T#10S, not the normalised TIME#10s0ms).
• A block read through two of its output pins is one block. It is called once in the program, so it is drawn once and called once in the rendering, with its readers branching off the pin they read. Where a reader sits behind another box, the box is still drawn once and the branch runs to it.
• Values and stores are drawn outside the box. A value feeding a side pin, and a store written on an output pin, sit on wires either side of the box the way the editor draws them, rather than being written inside among the pin names.
• Jumps and labels are written the way ST writes them — JMP ByeBye; and ByeBye: — not inside comment delimiters. They are program structure: something jumps to the label, and removing it changes what runs.
• An equivalent-ST rendering is available but not written by the export. The ST says things a single-wire diagram cannot, but showing each network twice reads worse than showing it once. tools/ladder/write_st.py writes it on demand, and its docstring says how to put it back on the export path.

What only a real project could find

None of these could have been caught by CI. CODESYS puts its own ScriptLib ahead of the standard library, and that XML parser is stricter and slower than anything runnable outside CODESYS.

• A UTF-8 BOM broke every POU. export_xml writes one; CODESYS's bundled ElementTree rejects it as illegal data at start of file. CPython's expat and stock IronPython both accept it silently.
• A single non-ASCII character broke one POU. That parser works byte-wise, so one degree sign in a comment lost the whole POU. Non-ASCII is now rewritten as numeric character references, with a test asserting the character survives the round trip rather than merely that parsing stopped failing. A character outside the Basic Multilingual Plane needs its surrogate pair recombining first, or the two halves are written separately and no parser outside CODESYS accepts the result.
• Jumps, inline ST and negated inputs were silently dropped — the rendering looked complete while a guard clause and a body of inline ST were simply absent. Worse than an error.
• Every fan-out was split into two networks. A block driving two outputs is one network in the editor; treating each output as its own network duplicated the shared expression and left the numbering disagreeing with CODESYS.
• An action rendered its parent POU. PLCopen has no top-level element for an action, so CODESYS exports the parent with the member nested inside it, parent body included. Rendering the POU's own body drew the parent's networks under the action's filename — a file named for one POU describing another.

Tests now assert on the bytes handed to the parser, not on whether a parse succeeds, so this class of bug is checkable in CI.

Performance

Measured on a real 25-POU project, then reduced from 6.8s to 1.7s:

	first measurement	now
CODESYS export_xml	0.2s	0.2s
parsing	6.4s	1.2s
drawing	0.3s	0.3s

Every export prints that split. It is worth saying that the first guess — that the CODESYS-side export_xml would dominate — was wrong by an order of magnitude, and so was the second; only the measurements were right, which is why the instrumentation ships rather than being a debugging aid that got stripped out.

Parsing goes through .NET's System.Xml under CODESYS, because the ElementTree shipped in ScriptLib parses in pure Python. test_xmlbackend.py compares the two backends element for element, attribute for attribute, and on rendered output — it can only run where both exist, which is the IronPython CI job, and it prints a SKIPPED banner rather than passing quietly anywhere else. It caught two real divergences on its first two runs, neither of which changed output yet: whitespace nodes being dropped, and CR surviving line-ending normalisation.

Not covered

• SFC and CFC are not rendered. They parse to nothing renderable and are skipped rather than writing an empty file, and the export summary names them. SFC is the obvious follow-up and SFCTesting.xml is already a fixture.
• A library reference pinned at version * shows the * rather than what it resolves to. A placeholder reference does resolve.

Tests

439 checks under CPython and 511 under IronPython 2.7, across four suites, run in CI under both. Fixtures include real CODESYS V3.5 SP11 exports for LD, FBD and SFC, plus hand-authored ones for shapes the real exports happen not to contain. The GraphicalTesting folder is a worked example of what the export writes, and a test reads its native xml and asserts the committed renderings still agree with it, so a renderer change cannot rewrite those files unnoticed.

src/script_diagnose_xml.py is included deliberately: it diagnosed the BOM in a single run, and CODESYS's parser has surprised us three times. Happy to drop it if you would rather not carry it.

[kehindepeters]

Pushed nine more commits (20eeb1f..1ff84d3) from reviewing a fresh re-export of the test project. No changes to the ST side; these are all about the ladder drawing matching what the CODESYS editor shows.

What changed

• A contact wired to a block output without naming the pin now tees the box. CODESYS often writes that connection without a formalParameter. The renderer took "no pin named" to mean "nothing reads this box", so the box's wall stayed closed while a rung carried on from it, and two rungs off one ENO (a RETURN beside a coil) were drawn as two copies of the box. They now build identically and merge into one branched rung.
• A contact feeding a block's side pin is drawn as that contact. A reset or enable read off the rail used to be flattened into the pin caption (R(PowerOff)──┤RESET); it is now drawn as PowerOff ┤P├──┤RESET, the way the editor draws it. Literals and expressions on side pins are unchanged. The ST keeps the flattened form, since it cannot draw a second wire.
• A pin's own bubble or edge marker is kept. Follow-up to the above: where the pin itself carries negated or edge in the PLCopen (a bubble or P/N on the pin, separate from the contact), the drawn contact had dropped it while the ST still said NOT. It now goes on the box wall (o, P or N), in the same place the power pin's already goes; a pin carrying both keeps its text caption, which has room for the pair.
• Branches that all end in outputs run to the right rail on their own. Two coils off one contact, or a RETURN beside a coil, were drawn as a loop that rejoined on the right. Each is a rung end, so each now reaches the rail separately, and the branch splits on the left only.
• README sample network updated to match; test project re-exported.

Tests: four new fixtures/checks in test_ladder.py covering each case; all four suites pass under CPython 3 and IronPython 2.7.12. The golden CODESYS fixture changed only where the RESET pin is now drawn as a contact.

One thing still to come: LD_TEST.txt in the tree was exported before the last fix, so its network 3 RESET pin reads ┤RESET where the renderer now writes PRESET (that pin has its own rising edge on top of the P contact). I'll re-export and push that.

[gsokoll]

Done