Add redirect_stdout_to_stderr() utility for native libraries

[fly1d]

Summary

Related to #196.

This implements the shared-helper option proposed by @vanshtaneja23 in their issue comment: a file-descriptor-level redirect_stdout_to_stderr() context manager in llm.utils.

The helper flushes buffered Python stdout before file descriptor 1 is redirected and again before it is restored. This prevents model output buffered outside the context from crossing stream boundaries. The docstring also notes that changing file descriptor 1 is process-wide and affects every thread while the context is active.

The regression test runs the helper in a subprocess with captured streams. It verifies that model output written before and after the context stays on stdout, while buffered Python output and a native os.write() inside the context go to stderr. The child process explicitly uses normal buffered output even if the parent test environment sets PYTHONUNBUFFERED.

This PR does not change llm-llama-cpp or llm-gpt4all, so it does not by itself fix #196. Those plugins would still need to wrap their native calls, or use library logging callbacks, before their diagnostics stop leaking to stdout.

Verification

• uv run pytest tests/test_utils.py -q: 92 passed
• PYTHONUNBUFFERED=1 uv run pytest tests/test_utils.py::test_redirect_stdout_to_stderr -q: 1 passed
• uv run pytest -q: 1125 passed
• uv run ruff check llm/utils.py tests/test_utils.py: passed
• uv run black --check llm/utils.py tests/test_utils.py: passed
• git diff --check: passed

Prepared with OpenAI Codex (GPT-5) assistance.

[fly1d]

I’ve corrected the description to clarify that this is only the shared-helper part of #196 and to credit @vanshtaneja23’s fd-level diagnosis and proposed approach. I should have included that credit when I opened the PR. If you were still planning to implement it yourself, I’m happy to step back.

[vanshtaneja23]

Thanks for the credit, and no need to step back — this is your PR, please carry it.

One thing worth fixing before it lands, though: the redirect needs a flush on either side of the dup2, otherwise it can send real model output to stderr — the inverse of what it is for.

sys.stdout is block-buffered whenever stdout is not a tty, which is exactly the llm ... > out.txt and pipe cases. Anything printed before the context manager may still be sitting in that buffer when fd 1 is pointed at stderr, and whatever flushes it inside the block writes it to stderr.

Repro against this branch:

import os, sys
from llm.utils import redirect_stdout_to_stderr

print("MODEL-OUTPUT-LINE-1")          # still in Python's buffer

with redirect_stdout_to_stderr():
    sys.stdout.flush()                 # anything that drains the buffer in here
    os.write(1, b"NATIVE-NOISE\n")

print("MODEL-OUTPUT-LINE-2")

$ python repro.py > out.txt 2> err.txt
$ cat out.txt
MODEL-OUTPUT-LINE-2
$ cat err.txt
MODEL-OUTPUT-LINE-1
NATIVE-NOISE

MODEL-OUTPUT-LINE-1 is model output and it ends up in stderr. An explicit
sys.stdout.flush() is the clearest trigger, but a buffer that fills on its own
during a long generation does the same thing.

Adding a flush in two places fixes it:

    stdout_fd = stdout.fileno()
    stderr_fd = stderr.fileno()
    stdout.flush()                      # drain the buffer while fd 1 is still stdout
    saved_stdout_fd = os.dup(stdout_fd)
    try:
        os.dup2(stderr_fd, stdout_fd)
        yield
    finally:
        stdout.flush()                  # and again before fd 1 points back
        os.dup2(saved_stdout_fd, stdout_fd)
        os.close(saved_stdout_fd)

With that, both model lines land in stdout and only the native noise goes to stderr.
Worth an assertion in the test too — the current one only writes with os.write,
so it passes either way.

One smaller note for the docstring: dup2 on fd 1 is process-global, so while the
block is open the redirect applies to every thread, not just the calling one. Worth
saying out loud, since a plugin loading a model on a background thread would silently
swallow whatever else is printing at the time.

[fly1d]

Thanks for catching this. Fixed in 8095382.

The helper now flushes stdout before redirecting fd 1 and again before restoring it. The restoration and descriptor close use nested finally blocks, and the docstring now calls out that the redirect is process-wide and affects every thread.

I also replaced the fd-only test with a captured subprocess regression. It verifies that model output before and after the context remains on stdout, while buffered Python output and native os.write() output inside the context go to stderr.

tests/test_utils.py: 92 passed. Full suite: 1125 passed. Ruff, Black, and git diff --check also pass.

[vanshtaneja23]

Thanks for the credit, and no need to step back — this is your PR, please carry it.

One thing worth fixing before it lands, though: the redirect needs a flush on either side of the dup2, otherwise it can send real model output to stderr — the inverse of what it is for.

sys.stdout is block-buffered whenever stdout is not a tty, which is exactly the llm ... > out.txt and pipe cases. Anything printed before the context manager may still be sitting in that buffer when fd 1 is pointed at stderr, and whatever flushes it inside the block writes it to stderr.

Repro against this branch:

import os, sys
from llm.utils import redirect_stdout_to_stderr

print("MODEL-OUTPUT-LINE-1")          # still in Python's buffer

with redirect_stdout_to_stderr():
    sys.stdout.flush()                 # anything that drains the buffer in here
    os.write(1, b"NATIVE-NOISE\n")

print("MODEL-OUTPUT-LINE-2")

$ python repro.py > out.txt 2> err.txt
$ cat out.txt
MODEL-OUTPUT-LINE-2
$ cat err.txt
MODEL-OUTPUT-LINE-1
NATIVE-NOISE

MODEL-OUTPUT-LINE-1 is model output and it ends up in stderr. An explicit
sys.stdout.flush() is the clearest trigger, but a buffer that fills on its own
during a long generation does the same thing.

Adding a flush in two places fixes it:

    stdout_fd = stdout.fileno()
    stderr_fd = stderr.fileno()
    stdout.flush()                      # drain the buffer while fd 1 is still stdout
    saved_stdout_fd = os.dup(stdout_fd)
    try:
        os.dup2(stderr_fd, stdout_fd)
        yield
    finally:
        stdout.flush()                  # and again before fd 1 points back
        os.dup2(saved_stdout_fd, stdout_fd)
        os.close(saved_stdout_fd)

With that, both model lines land in stdout and only the native noise goes to stderr.
Worth an assertion in the test too — the current one only writes with os.write,
so it passes either way.

One smaller note for the docstring: dup2 on fd 1 is process-global, so while the
block is open the redirect applies to every thread, not just the calling one. Worth
saying out loud, since a plugin loading a model on a background thread would silently
swallow whatever else is printing at the time.