CI fails on Windows

[cwalther]

As discovered in a CI run of #440, some tests now fail on Windows. They still worked last week.

The failing tests are:

• "after each" hook for attach remote ▸ can detach from a running program, in all scenarios where it runs (defaults, remote, gdb-async-off, remote gdb-async-off).
• launch ▸ works with unicode in file names, in all scenarios where it runs (defaults, gdb-async-off).

What I have found out so far:

• The difference is that the OS image changed from Windows Server 2022 to Windows Server 2025.
• Apparently we are not using the GDB that is downloaded as part of the job, but one that is supplied with the OS image. That was version 11.2 before and 16.2 now.
• We could work around the CI failures by reverting to the Windows-2022 image, but if there are actual problems with GDB 16, we should probably fix them anyway.
• I can reproduce the problem that GDB 16.2 becomes unresponsive when the gdbserver that it is attached to is killed while the program is running, after outputting warning: Exception condition detected on fd.

[jreineckearm]

Thanks for investigating, @cwalther . I'd suggest to switch back to 2022 to unblock the tests until we have time to look into 16.2. This should unblock ongoing development. I've started draft #447 for the first.

[jreineckearm]

Reviewed a couple of logs from a failing CI run that hadn't expired yet. All failing cases show the same error/warnings

launch ▸ works with unicode in file names, in all scenarios where it runs (defaults, gdb-async-off).

Looks like something changed in the new GDB version for handling unicode characters.

[17:12:17.632 UTC] GDB result: 5 error,msg="No source file named D:\\a\\cdt-gdb-adapter\\cdt-gdb-adapter\\src\\integration-tests\\test-programs\\bug275-\346\265\213\350\257\225.c."
[17:12:17.632 UTC] GDB command: 5 -break-insert --source "D:\\a\\cdt-gdb-adapter\\cdt-gdb-adapter\\src\\integration-tests\\test-programs\\bug275-测试.c" --line 3 failed with 'No source file named D:\a\cdt-gdb-adapter\cdt-gdb-adapter\src\integration-tests\test-programs\bug275-测试.c.'

"after each" hook for attach remote ▸ can detach from a running program, in all scenarios where it runs (defaults, remote, gdb-async-off, remote gdb-async-off).

Getting $_gthread seems to fail in this scenario. Timestamps don't indicate problems with timeouts, the remote session states don't show a completed disconnect. Maybe a crash somewhere?

2025-9-16, 17:11:00.737 UTC
[17:11:00.737 UTC] From client: initialize({"supportsRunInTerminalRequest":true,"adapterID":"gdb","linesStartAt1":true,"columnsStartAt1":true,"pathFormat":"path"})
[17:11:00.737 UTC] To client: {"seq":0,"type":"response","request_seq":1,"command":"initialize","success":true,"body":{"supportsConfigurationDoneRequest":true,"supportsEvaluateForHovers":true,"supportsSetVariable":true,"supportsConditionalBreakpoints":true,"supportsHitConditionalBreakpoints":true,"supportsLogPoints":true,"supportsFunctionBreakpoints":true,"supportsDisassembleRequest":true,"supportsReadMemoryRequest":true,"supportsWriteMemoryRequest":true,"supportsSteppingGranularity":true,"supportsInstructionBreakpoints":true,"supportsTerminateRequest":true,"supportsDataBreakpoints":true,"breakpointModes":[{"label":"Hardware Breakpoint","mode":"hardware","appliesTo":["source"]},{"label":"Software Breakpoint","mode":"software","appliesTo":["source"]}]}}
....
[17:11:01.459 UTC] To client: {"seq":0,"type":"event","event":"initialized"}
[17:11:01.459 UTC] To client: {"seq":0,"type":"response","request_seq":3,"command":"attach","success":true}
[17:11:01.459 UTC] GDB Remote session: State 'CONNECTED' => 'SESSION_READY'
[17:11:01.460 UTC] From client: configurationDone(undefined)
[17:11:01.460 UTC] To client: {"seq":0,"type":"event","event":"output","body":{"category":"console","output":"\nIn the Debug Console view you can interact directly with GDB.\nTo display the value of an expression, type that expression which can reference\nvariables that are in scope. For example type '2 + 3' or the name of a variable.\nArbitrary commands can be sent to GDB by prefixing the input with a '>',\nfor example type '>show version' or '>help'.\n\n"}}
[17:11:01.460 UTC] GDB command: 5 -exec-continue
[17:11:01.460 UTC] GDB write command: 5 -exec-continue
[17:11:01.478 UTC] GDB result: 5 running
[17:11:01.478 UTC] GDB command: 5 -exec-continue completed with data
[17:11:01.479 UTC] GDB exec async: running,thread-id="all"
[17:11:01.479 UTC] To client: {"seq":0,"type":"event","event":"continued","body":{"threadId":1,"allThreadsContinued":true}}
[17:11:01.479 UTC] To client: {"seq":0,"type":"response","request_seq":4,"command":"configurationDone","success":true}
[17:11:01.480 UTC] From client: disconnect(undefined)
[17:11:01.480 UTC] GDB Remote session: State 'SESSION_READY' => 'EXITING'
[17:11:01.481 UTC] GDB command: 6 -data-evaluate-expression "$_gthread"
[17:11:01.481 UTC] GDB write command: 6 -data-evaluate-expression "$_gthread"
[17:11:01.482 UTC] To client: {"seq":0,"type":"event","event":"output","body":{"category":"log","output":"warning: Exception condition detected on fd 680\n"}}

[jreineckearm]

Looks like now also windows-2022 runner images got updated with new GNU toolchain parts. Which makes this pretty much a blocker unless one can explicitly select runner images/runners with specific images... 😒

[cwalther]

Or make sure the explicitly installed GDB is used instead of the one that comes with the image?

(I am on vacation so no contributions from me for another week.)

[jreineckearm]

Thanks for the hint. Enjoy your vacation, we'll get it sorted somehow. 🙂

[jreineckearm]

Found how to make CI use the mingw installation from chocolatey.
But the "newest" version that works now has an even older GDB in it (10.2).
Mingw 12.x versions from chocolatey seem to contain already a GDB 12.1. Which is different from what was installed on the GH runners with mingw 12.x (GDB 11.2) ?
GDB 12.1 gives me errors about broken string tables.

Will have another look tomorrow. But may have to live with either that, or with disabling some Windows tests on new GDBs until we get to the bottom of that. Not entirely sure at this point what's the lesser evil.

[jreineckearm]

Did some investigation and had a chat with a trusted AI:
It might be related to https://sourceware.org/bugzilla/show_bug.cgi?id=31577

In older Windows GDBs like 11.2, detecting a closed remote threw an exception causing GDB to close. See log from a working test run with 11.2 and set debug remote 1. gdbserver got terminated before the disconnect call.

[14:53:20.784 UTC] From client: disconnect(undefined)
[14:53:20.784 UTC] GDB Remote session: State 'SESSION_READY' => 'EXITING'
[14:53:20.784 UTC] GDB command: 7 disconnect
[14:53:20.784 UTC] GDB write command: 7 disconnect
[14:53:20.785 UTC] To client: {"seq":0,"type":"event","event":"output","body":{"category":"log","output":"warning: Exception condition detected on fd 768\n"}}
[14:53:20.785 UTC] To client: {"seq":0,"type":"event","event":"output","body":{"category":"log","output":"[remote] wait: enter\n"}}
[14:53:20.786 UTC] GDB notify async: thread-exited,id="1",group-id="i1"
[14:53:20.786 UTC] GDB notify async: thread-exited,id="2",group-id="i1"
[14:53:20.786 UTC] GDB notify async: thread-group-exited,id="i1"
[14:53:20.788 UTC] To client: {"seq":0,"type":"event","event":"output","body":{"category":"log","output":"[remote] wait: exit\n"}}
[14:53:20.788 UTC] To client: {"seq":0,"type":"event","event":"output","body":{"category":"log","output":"Remote communication error.  Target disconnected.: No such file or directory.\n"}}
[14:53:20.788 UTC] To client: {"seq":0,"type":"event","event":"output","body":{"category":"log","output":"disconnect\n"}}
[14:53:20.788 UTC] To client: {"seq":0,"type":"event","event":"output","body":{"category":"log","output":"You can't do that when your target is `exec'\n"}}
[14:53:20.788 UTC] GDB result: 7 error,msg="You can't do that when your target is `exec'"
[14:53:20.788 UTC] GDB command: 7 disconnect failed with 'You can't do that when your target is `exec''
[14:53:20.788 UTC] To client: {"seq":0,"type":"event","event":"output","body":{"category":"server","output":"gdb connection lost\n"}}
[14:53:20.788 UTC] GDB Remote session: State 'EXITING' => 'EXITED'
[14:53:20.788 UTC] GDB Remote session: skip stopping GDB server, already down
[14:53:20.788 UTC] To client: {"seq":0,"type":"response","request_seq":5,"command":"disconnect","success":true}

The fix in newer GDBs seems to catch the exception and keeps GDB alive.

[15:26:47.467 UTC] From client: disconnect(undefined)
[15:26:47.467 UTC] GDB Remote session: State 'SESSION_READY' => 'EXITING'
[15:26:47.467 UTC] GDB command: 7 disconnect
[15:26:47.467 UTC] GDB write command: 7 disconnect
[15:26:47.468 UTC] To client: {"seq":0,"type":"event","event":"output","body":{"category":"log","output":"warning: Exception condition detected on fd 520\n"}}
[15:26:47.468 UTC] To client: {"seq":0,"type":"event","event":"output","body":{"category":"log","output":"[remote] wait: enter\n"}}

Our MI command queue gets stuck here. The disconnect command actually ends.
But we don't get a proper result from the MI stream. And the writeCallback in GDBBackend.sendCommand is called but without an error since there was no MI pipe error. All I see is the log stream message about the file descriptor....

Need to think about a (probably version-specific) heuristic for Windows when we encounter this file descriptor warning. Maybe something that checks if a remote is still connected. Unless an option was introduced somewhere to escalate to a GDB shutdown....