Skip to content

fix: read the redirect log as bytes, not text (CII-240) - #426

Merged
hfudev merged 1 commit into
espressif:mainfrom
hacker-cb:fix/forward-redirect-log-as-bytes
Aug 21, 2026
Merged

fix: read the redirect log as bytes, not text (CII-240)#426
hfudev merged 1 commit into
espressif:mainfrom
hacker-cb:fix/forward-redirect-log-as-bytes

Conversation

@hacker-cb

Copy link
Copy Markdown
Contributor

Fixes #425.

_PopenRedirectProcess._forward_io read the redirect log in text mode while it was still
being appended to. Every read reaches a temporary EOF, which finalizes the incremental
decoder, so a UTF-8 sequence straddling that boundary raises UnicodeDecodeError. The
surrounding except Exception swallows it — the process exits 0 with an empty stderr and
DUT output stops for the rest of the session, surfacing later as a pexpect timeout that
names no cause.

Why binary rather than a decoder

The decode was never needed here:

  • MessageQueue.put accepts str | bytes and re-encodes through to_bytes(), which is
    the identity for bytes;
  • PexpectProcess's buffer is bytes;
  • the serial transport already forwards read_all() undecoded
    (pytest-embedded-serial/pytest_embedded_serial/serial.py:244).

So today's path is a bytes → str → bytes round trip. Reading in binary removes the failure
mode and makes the two transports consistent.

One behaviour change

Text mode applied universal-newline translation, so a DUT sending CRLF was forwarded as LF.
Binary mode preserves CR — which is what the serial transport already delivers, and both
_listen and the Unity parser handle CRLF already. test_forward_io_preserves_carriage_returns
pins it so it stays deliberate.

Tests

Two unit tests next to the existing test_listen_no_data_loss_*, in the same shape.

The reproduction condition is worth noting: the multi-byte sequence has to be split
between two appends, which a byte-at-a-time UART/QEMU stream does naturally. Writing a
whole line in a single write() does not reproduce it.

On unmodified main both fail:

AssertionError: forwarder died on a split sequence (exitcode=0)
AssertionError: assert b'first\nsecond\n' == b'first\r\nsecond\r\n'

With this change both pass.

Test runs on this machine

  • pytest-embedded/tests/ — 36 passed, 1 skipped, 2 failed.
  • The other eight packages — 47 failed, 24 passed, 9 skipped.

All of those failures reproduce on unmodified main: I ran the same selection on both refs
and the sorted FAILED lists are identical (empty diff), with the same 47/24/9 totals.
They need QEMU binaries, serial ports, an Arduino CLI or NuttX images, none of which are
present here. test_temp_disable_packages fails for a separate reason — it expects
ImportError for the sibling packages, which the editable install from CONTRIBUTING makes
importable.

Not included

Narrowing except Exception. MessageQueue.put already swallows a closed queue itself, and
the only escape I observed at teardown was BrokenPipeError, but I have not shown that list
to be exhaustive — so this PR stays one change. Happy to add it here if you would rather have
it together.

`_PopenRedirectProcess._forward_io` opened the redirect log in text mode while
it was still being appended to. Every read reaches a temporary EOF, which
finalizes the incremental decoder, so a UTF-8 sequence straddling that boundary
raises UnicodeDecodeError. The surrounding `except Exception` swallows it: the
process exits 0 with an empty stderr and DUT output stops for the rest of the
session, surfacing much later as a pexpect timeout that names no cause.

Decoding there was never needed - `MessageQueue.put` re-encodes through
`to_bytes`, the pexpect buffer is bytes, and the serial transport already
forwards `read_all()` undecoded - so reading in binary removes a failure mode
and makes the two transports consistent.

One behaviour change: text mode applied universal-newline translation, so a DUT
sending CRLF was forwarded as LF. Binary mode preserves CR, which is what the
serial transport already delivers and what the console writer and the Unity
parser already handle.
@github-actions github-actions Bot changed the title fix: read the redirect log as bytes, not text fix: read the redirect log as bytes, not text (CII-240) Aug 6, 2026
@github-actions

github-actions Bot commented Aug 21, 2026

Copy link
Copy Markdown
Title Coverage Tests Skipped Failures Errors Time
Qemu Coverage 7 0 💤 0 ❌ 0 🔥 25.154s ⏱️
3.14 X64 Coverage 119 16 💤 0 ❌ 0 🔥 15m 37s ⏱️
3.10 ARM64 Coverage 119 18 💤 0 ❌ 0 🔥 12m 46s ⏱️

@hfudev
hfudev merged commit 0d6502d into espressif:main Aug 21, 2026
8 of 9 checks passed
@hfudev

hfudev commented Aug 21, 2026

Copy link
Copy Markdown
Member

LGTM. Thank you for the fix. I'll do the release next Monday

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

_forward_io dies silently when a multi-byte UTF-8 char is split across writes, truncating DUT output (CII-239)

2 participants