Describe the bug
_PopenRedirectProcess._forward_io (pytest-embedded/pytest_embedded/log.py:155-162, identical on v2.8.1 and main) reads the redirect log in text mode:
with open(logfile) as fr: # text mode, errors='strict'
while True:
try:
msg_queue.put(fr.read()) # msg_queue may be closed
except Exception:
breakThe file is being appended to concurrently, so the reader keeps hitting a temporary EOF. When that end falls inside a multi-byte sequence, the decoder raises:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 43: unexpected end of data
except Exception swallows it: the reader exits with exitcode=0, empty stderr, and DUT output simply stops mid-stream. Downstream this surfaces as pexpect.exceptions.TIMEOUT on whatever expect() was waiting for, with nothing anywhere mentioning decoding.
Not specific to read() without a size — read(n) and readline() fail the same way; only binary mode avoids it. The text-mode open exists in 1.x too, so the decode issue predates 387b877 (#374); that commit only made it silent (1.18.x prints a traceback and exits 1).
Configuration Files
Not relevant — no project config needed, and no ESP-IDF: the defect is in the pure-Python forwarding path.
To Reproduce
The trigger is a multi-byte character split across two writes to the log, which a byte-at-a-time UART/QEMU stream does naturally. Writing whole lines with a single write() does not reproduce.
The decode itself:
p = 'x.log'
open(p, 'wb').write(b'hello \xe2\x80\x94 world\n[SKIP] tail \xe2')
open(p).read() # UnicodeDecodeError: ... unexpected end of data
Against the real class: start _PopenRedirectProcess(msg_queue, logfile), append an ASCII chunk (arrives), then append a chunk ending mid-em-dash — is_alive=False, exitcode=0, child stderr empty, and the text before the split byte is lost too (read() raises before returning anything). End to end in my run, one em-dash (U+2014) in device output stopped the stream at line 40 and the run failed with TIMEOUT: Not found "Press ENTER to see the list of tests" after 30 s.
Expected behavior
Forwarding should not depend on where chunk boundaries fall.
Suggested fix:
with open(logfile, 'rb') as fr:
The decode looks unnecessary here: MessageQueue.put accepts str | bytes and calls to_bytes() (identity for bytes), PexpectProcess's buffer is bytes, and the serial transport already puts raw bytes on the same queue (pytest-embedded-serial/pytest_embedded_serial/serial.py:244, self._q.put(s) with s = self._s.read_all(), no decode) — so today's path is a bytes → str → bytes round trip, and 'rb' makes the two transports consistent.
One honest caveat: text mode does universal-newline translation, so 'rb' stops swallowing CR (CRLF sample: 45 bytes written → 43 received before, 45 after). That is the same raw \r\n the serial path already delivers, and _listen's \r\n → \n plus splitlines() in the Unity parser already handle it.
Optionally, except Exception could be narrowed — MessageQueue.put already swallows a closed queue itself, and the only escape I observed was BrokenPipeError at teardown — but I haven't proven that list exhaustive, so treat it as a suggestion.
Related, not part of this report: pytest-embedded-idf/pytest_embedded_idf/unity_tester.py:181 does a strict .decode('utf8'), and _listen's to_str(msg) uses errors='ignore', so a split character vanishes from the console copy without a trace.
Dev Environment (please complete the following information):
- OS: Linux (Ubuntu 24.04, Docker) and macOS
- Python Version: 3.12.3 and 3.14.5
- Pytest Version: 9.1.1
- pytest-embedded Version: 2.8.1 and 2.2.1 (also 1.18.1, where it exits 1 with a traceback);
git diff v2.8.1..main -- pytest-embedded/pytest_embedded/log.py is empty
I couldn't find an existing issue for this. Happy to send a PR with the 'rb' change and a regression test if that's welcome.
Describe the bug
_PopenRedirectProcess._forward_io(pytest-embedded/pytest_embedded/log.py:155-162, identical on v2.8.1 andmain) reads the redirect log in text mode:The file is being appended to concurrently, so the reader keeps hitting a temporary EOF. When that end falls inside a multi-byte sequence, the decoder raises:
except Exceptionswallows it: the reader exits withexitcode=0, empty stderr, and DUT output simply stops mid-stream. Downstream this surfaces aspexpect.exceptions.TIMEOUTon whateverexpect()was waiting for, with nothing anywhere mentioning decoding.Not specific to
read()without a size —read(n)andreadline()fail the same way; only binary mode avoids it. The text-mode open exists in 1.x too, so the decode issue predates 387b877 (#374); that commit only made it silent (1.18.x prints a traceback and exits 1).Configuration Files
Not relevant — no project config needed, and no ESP-IDF: the defect is in the pure-Python forwarding path.
To Reproduce
The trigger is a multi-byte character split across two writes to the log, which a byte-at-a-time UART/QEMU stream does naturally. Writing whole lines with a single
write()does not reproduce.The decode itself:
Against the real class: start
_PopenRedirectProcess(msg_queue, logfile), append an ASCII chunk (arrives), then append a chunk ending mid-em-dash —is_alive=False,exitcode=0, child stderr empty, and the text before the split byte is lost too (read()raises before returning anything). End to end in my run, one em-dash (U+2014) in device output stopped the stream at line 40 and the run failed withTIMEOUT: Not found "Press ENTER to see the list of tests"after 30 s.Expected behavior
Forwarding should not depend on where chunk boundaries fall.
Suggested fix:
The decode looks unnecessary here:
MessageQueue.putacceptsstr | bytesand callsto_bytes()(identity for bytes),PexpectProcess's buffer is bytes, and the serial transport already puts raw bytes on the same queue (pytest-embedded-serial/pytest_embedded_serial/serial.py:244,self._q.put(s)withs = self._s.read_all(), no decode) — so today's path is a bytes → str → bytes round trip, and'rb'makes the two transports consistent.One honest caveat: text mode does universal-newline translation, so
'rb'stops swallowing CR (CRLF sample: 45 bytes written → 43 received before, 45 after). That is the same raw\r\nthe serial path already delivers, and_listen's\r\n→\nplussplitlines()in the Unity parser already handle it.Optionally,
except Exceptioncould be narrowed —MessageQueue.putalready swallows a closed queue itself, and the only escape I observed wasBrokenPipeErrorat teardown — but I haven't proven that list exhaustive, so treat it as a suggestion.Related, not part of this report:
pytest-embedded-idf/pytest_embedded_idf/unity_tester.py:181does a strict.decode('utf8'), and_listen'sto_str(msg)useserrors='ignore', so a split character vanishes from the console copy without a trace.Dev Environment (please complete the following information):
git diff v2.8.1..main -- pytest-embedded/pytest_embedded/log.pyis emptyI couldn't find an existing issue for this. Happy to send a PR with the
'rb'change and a regression test if that's welcome.