Some tests fail when ps is not installed

[justeph]

On some container, such as manylinux_2_39_riscv64, based on Rocky Linux, ps utility is not installed by default, making some test to fail:

=================================== FAILURES ===================================
______________________________ test_setproctitle _______________________________
    def test_setproctitle():
        """setproctitle() can set the process title, duh."""
        rv = run_script(
            r"""
    import setproctitle
    setproctitle.setproctitle('Hello, world!')
    
    import os
    print(os.getpid())
    # ps can fail on kfreebsd arch
    # (http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=460331)
    print(os.popen("ps -x -o pid,command 2> /dev/null").read())
    """
        )
        lines = [line for line in rv.splitlines() if line]
        pid = lines.pop(0)
        pids = dict([r.strip().split(None, 1) for r in lines])
>       title = _clean_up_title(pids[pid])
E       KeyError: '150'
/project/tests/setproctitle_test.py:68: KeyError
_________________________________ test_issue_8 _________________________________
tmp_pypath = PosixPath('/tmp/pytest-of-root/pytest-0/test_issue_80')
    def test_issue_8(tmp_pypath):
        """Test that the module works with 'python -m'."""
        module = "spt_issue_8"
        with open(tmp_pypath / f"{module}.py", "w") as f:
            f.write(
                r"""
    import setproctitle
    setproctitle.setproctitle("Hello, module!")
    
    import os
    print(os.getpid())
    print(os.popen("ps -x -o pid,command 2> /dev/null").read())
                """
            )
    
        rv = run_script(args="-m " + module)
        lines = [line for line in rv.splitlines() if line]
        pid = lines.pop(0)
        pids = dict([r.strip().split(None, 1) for r in lines])
    
>       title = _clean_up_title(pids[pid])
E       KeyError: '159'
/project/tests/setproctitle_test.py:192: KeyError
______________________________ test_large_cmdline ______________________________
tmp_pypath = PosixPath('/tmp/pytest-of-root/pytest-0/test_large_cmdline0')
    def test_large_cmdline(tmp_pypath):
        """Test with a 64KB command line."""
        module = "longargs"
        with open(tmp_pypath / f"{module}.py", "w") as f:
            f.write(
                r"""
    import setproctitle
    setproctitle.setproctitle("Hello, long!")
    
    import os
    print(os.getpid())
    print(os.popen("ps -x -o pid,command 2> /dev/null").read())
                """
            )
    
        rv = run_script(args=f"-m {module} {' '.join(['X' * 1024] * 64)}")
        lines = [line for line in rv.splitlines() if line]
        pid = lines.pop(0)
        pids = dict([r.strip().split(None, 1) for r in lines])
    
>       title = _clean_up_title(pids[pid])
E       KeyError: '162'
/project/tests/setproctitle_test.py:216: KeyError
_________________________________ test_unicode _________________________________
    def test_unicode():
        """Title can contain unicode characters."""
        snowman = "\u2603"
        try:
            snowman.encode(sys.getdefaultencoding())
        except UnicodeEncodeError:
            pytest.skip(
                "default encoding '%s' can't deal with snowmen"
                % sys.getdefaultencoding()
            )
    
        try:
            snowman.encode(sys.getfilesystemencoding())
        except UnicodeEncodeError:
            pytest.skip(
                "file system encoding '%s' can't deal with snowmen"
                % sys.getfilesystemencoding()
            )
    
        rv = run_script(
            r"""
    snowman = u'\u2603'
    
    import setproctitle
    setproctitle.setproctitle("Hello, " + snowman + "!")
    
    import os
    import locale
    from subprocess import Popen, PIPE
    print(os.getpid())
    proc = Popen("ps -x -o pid,command 2> /dev/null", shell=True,
        close_fds=True, stdout=PIPE, stderr=PIPE)
    buf = proc.stdout.read()
    print(buf.decode(locale.getpreferredencoding(), 'replace'))
    """
        )
        lines = [line for line in rv.splitlines() if line]
        pid = lines.pop(0)
        pids = dict([r.strip().split(None, 1) for r in lines])
    
        snowmen = [
            "\u2603",  # ps supports unicode
            r"\M-b\M^X\M^C",  # ps output on BSD
            r"M-bM^XM^C",  # ps output on some Darwin < 11.2
            "\ufffdM^XM^C",  # ps output on Darwin 11.2
        ]
>       title = _clean_up_title(pids[pid])
E       KeyError: '165'
/project/tests/setproctitle_test.py:266: KeyError
_______________________________ test_weird_args ________________________________
    def test_weird_args():
        """No problem with encoded arguments."""
        euro = "\u20ac"
        snowman = "\u2603"
        try:
            rv = run_script(
                r"""
    import setproctitle
    setproctitle.setproctitle("Hello, weird args!")
    
    import os
    print(os.getpid())
    print(os.popen("ps -x -o pid,command 2> /dev/null").read())
    """,
                args=" ".join(["-", "hello", euro, snowman]),
            )
        except TypeError:
            pytest.skip("apparently we can't pass unicode args to a program")
    
        lines = [line for line in rv.splitlines() if line]
        pid = lines.pop(0)
        pids = dict([r.strip().split(None, 1) for r in lines])
    
>       title = _clean_up_title(pids[pid])
E       KeyError: '168'
/project/tests/setproctitle_test.py:297: KeyError
_______________________________ test_weird_path ________________________________
tmp_path = PosixPath('/tmp/pytest-of-root/pytest-0/test_weird_path0')
spt_directory = '/tmp/tmp.xz8WNoRJrN/venv/lib/python3.8/site-packages'
    def test_weird_path(tmp_path, spt_directory):
        """No problem with encoded argv[0] path."""
        _check_4388()
        euro = "\u20ac"
        snowman = "\u2603"
        dir = tmp_path / euro / snowman
        try:
            os.makedirs(dir)
        except UnicodeEncodeError:
            pytest.skip("file system doesn't support unicode")
    
        exc = dir / os.path.basename(sys.executable)
        os.symlink(sys.executable, exc)
    
        rv = run_script(
            f"""
    import sys
    sys.path.insert(0, {repr(spt_directory)})
    
    import setproctitle
    setproctitle.setproctitle("Hello, weird path!")
    
    import os
    print(os.getpid())
    print(os.popen("ps -x -o pid,command 2> /dev/null").read())
    """,
            args=" ".join(["-", "foo", "bar", "baz"]),
            executable=exc,
        )
        lines = [line for line in rv.splitlines() if line]
        pid = lines.pop(0)
        pids = dict([r.strip().split(None, 1) for r in lines])
    
>       title = _clean_up_title(pids[pid])
E       KeyError: '173'
/project/tests/setproctitle_test.py:334: KeyError
=========================== short test summary info ============================
FAILED ../../../project/tests/setproctitle_test.py::test_setproctitle - KeyError: '150'
FAILED ../../../project/tests/setproctitle_test.py::test_issue_8 - KeyError: '159'
FAILED ../../../project/tests/setproctitle_test.py::test_large_cmdline - KeyError: '162'
FAILED ../../../project/tests/setproctitle_test.py::test_unicode - KeyError: '165'
FAILED ../../../project/tests/setproctitle_test.py::test_weird_args - KeyError: '168'
FAILED ../../../project/tests/setproctitle_test.py::test_weird_path - KeyError: '173'
============ 6 failed, 17 passed, 3 skipped, 2 deselected in 5.71s =============

This is an important issue, because this is the container image used by cibuildwheel on newer version. We need a recent version because we are building and deploying py-setproctitle for riscv64, and riscv64 is supported only on latest version.

I can help fixing the bug by submitting a patch, just let me know how you would like to handle this: skip test if ps is missing? add a script to check and install system dependencies?

[dvarrazzo]

The entire point of setproctitle is to show the output of ps. If ps is not installed the tests are pretty useless.

ps can be installed in the cibuildwheel image as part of a pre-install script via CIBW_BEFORE_ALL_LINUX script. See psycopg build pipeline and the script it calls, which ends up installing a few packages via apk.

If you would like to propose a MR, you can extend the CI to build the risc images too, and to prepare a CI script to install the test requirements.

[justeph]

Hi @dvarrazzo

The entire point of setproctitle is to show the output of ps. If ps is not installed the tests are pretty useless.

fair enough :)

ps can be installed in the cibuildwheel image as part of a pre-install script via CIBW_BEFORE_ALL_LINUX script. See psycopg build pipeline and the script it calls, which ends up installing a few packages via apk.

This is what we did internally 👍

If you would like to propose a MR, you can extend the CI to build the risc images too, and to prepare a CI script to install the test requirements.

I’d be happy to help on that, but there are a few points to consider:

• Building riscv64 wheels will require a fairly recent version of cibuildwheel (based on an updated manylinux container that support riscv64). While this should not pose a major issue, it may affect the way all wheels are built.

• We also need access to a native riscv64 runner. Many tests fail under QEMU, as process names are often prefixed with QEMU-related strings. To the best of my knowledge, GitHub currently does not provide native riscv64 runners (though my familiarity with GitHub CI is limited). Or we may need to adapt all failing tests under emulation.

What do you think?

[dvarrazzo]

Hello @justeph

I’d be happy to help on that, but there are a few points to consider:

• Building riscv64 wheels will require a fairly recent version of cibuildwheel (based on an updated manylinux container that support riscv64). While this should not pose a major issue, it may affect the way all wheels are built.

No problem: we can run the latest version in Gitlab CI. If we need to adapt building for the other platform we will do that. At worst, we can use different versions for different platforms. I don't see an immediate reason to stop updating cibuildwheel version to whatever the latest is.

• We also need access to a native riscv64 runner. Many tests fail under QEMU, as process names are often prefixed with QEMU-related strings. To the best of my knowledge, GitHub currently does not provide native riscv64 runners (though my familiarity with GitHub CI is limited). Or we may need to adapt all failing tests under emulation.

Probably we will have to use QEMU, yes, and adapting the tests to a different output should be no problem. We do it already for some platforms:

py-setproctitle/tests/setproctitle_test.py

Lines 524 to 531 in 389ed6f

	def _clean_up_title(title):
	"""Clean up a string from the prefix added by the platform."""
	# BSD's setproctitle decorates the title with the process name.
	if "bsd" in sys.platform:
	procname = os.path.basename(sys.executable)
	title = " ".join([t for t in title.split(" ") if procname not in t])
	return title

I don't mind extending it to other cases, as long as we can test some basic functionalities.

[justeph]

Hi @dvarrazzo,

I dug a bit deeper into the test failures by running some of the tests manually inside an emulated container. It turns out the issue is not just a prefix mismatch as I initially suspected:

>>> import os                                                                                  
>>> print(os.getpid())                                                                         
210                                                                                            
>>> print(os.popen("ps -x -o pid,command 2> /dev/null").read())                                
    PID COMMAND                                                                                
      1 /usr/bin/qemu-riscv64-static /bin/bash /bin/bash                                       
     10 /usr/bin/qemu-riscv64-static /venv/bin/python python                                   
    210 /usr/bin/qemu-riscv64-static /venv-3.13/bin/python python                              
    213 /usr/bin/qemu-riscv64-static /bin/sh /bin/sh -c ps -x -o pid,command 2> /dev/null      
    216 ps -x -o pid,command                                                                   
                                                                                               
>>> import setproctitle                                                                        
[SPT]: module init                                                                             
>>> setproctitle.setproctitle("Hello world")                                                   
[SPT]: reading argc/argv from Python main                                                      
[SPT]: found 1 arguments                                                                       
[SPT]: walking from environ to look for the arguments                                          
[SPT]: found environ at 0x7fce8af7dd2e                                                         
[SPT]: argv[0] should be at 0x7fce8af7dd27                                                     
[SPT]: found argv[0]: python                                                                   
[SPT]: environ has been copied                                                                 
>>> print(setproctitle.getproctitle())                                                         
[SPT]: setup was called more than once!                                                        
Hello world                                                                                    
>>> print(os.popen("ps -x -o pid,command 2> /dev/null").read())                                
    PID COMMAND                                                                                
      1 /usr/bin/qemu-riscv64-static /bin/bash /bin/bash                                       
     10 /usr/bin/qemu-riscv64-static /venv/bin/python python                                   
    210 /usr/bin/qemu-riscv64-static /venv-3.13/bin/python python                              
    219 /usr/bin/qemu-riscv64-static /bin/sh /bin/sh -c ps -x -o pid,command 2> /dev/null      
    222 ps -x -o pid,command                                                                   
                                                                                               
>>>                                                                                            

As you can see, getproctitle correctly returns the value set by setproctitle, but the output of ps never reflects the change. That makes me think this is a more fundamental limitation, rather than something we can fix just by stripping prefixes in the tests.

Do you think there’s a proper way to handle this, or is it simply not possible under emulation? Would it even be worth trying, or should we just skip the failing tests under emulation?

FYI:

=========================== short test summary info ============================                                                                   
FAILED ../../../project/tests/setproctitle_test.py::test_setproctitle - AssertionError: assert '/usr/bin/qem...ython /tmp/tm' == 'Hello, world!'   
FAILED ../../../project/tests/setproctitle_test.py::test_issue_8 - AssertionError: assert '/usr/bin/qem...ython /tmp/tm' == 'Hello, module!'       
FAILED ../../../project/tests/setproctitle_test.py::test_large_cmdline - AssertionError: assert '/usr/bin/qem...ython /tmp/tm' == 'Hello, long!'   
FAILED ../../../project/tests/setproctitle_test.py::test_unicode - Failed: unexpected ps output: '/usr/bin/qemu-riscv64-static /tmp/tmp.txmFaZ...  
FAILED ../../../project/tests/setproctitle_test.py::test_weird_args - AssertionError: assert '/usr/bin/qem...ython /tmp/tm' == 'Hello, weird args!'
FAILED ../../../project/tests/setproctitle_test.py::test_weird_path - AssertionError: assert '/usr/bin/qem...est_weird_pat' == 'Hello, weird path!'
FAILED ../../../project/tests/setproctitle_test.py::test_noenv - AssertionError: title not set as expected                                         
FAILED ../../../project/tests/setthreadtitle_test.py::test_thread_title_unchanged - AssertionError: assert 'python\npython\n' == 'python\n'        
FAILED ../../../project/tests/setthreadtitle_test.py::test_set_thread_title - Failed: test script failed                                           
FAILED ../../../project/tests/setthreadtitle_test.py::test_set_threads_title - Failed: test script failed                                          
=========== 10 failed, 13 passed, 3 skipped, 2 deselected in 16.73s ============                                                                   

[gershnik]

This is AFAIK entirely expected. When running under emulation all the changes setproctitle makes happen inside the fake environment - they matter nothing to the normal OS view of the process. As far as ps knows the process being run is /usr/bin/qemu-riscv64-static and its command line hasn't changed. The python being run by the emulator has its own command line and environment probably known only to the emulator.

[justeph]

thank you @gershnik ! So what would be the best way to move on and supporting RISCV wheels? skip failing tests on this architecture?

[justeph]

oh, I see you are already disabling tests on QEMU: https://github.com/dvarrazzo/py-setproctitle/blob/master/.github/workflows/packages.yml#L82
So I guess we don't have anything to do then?

[dvarrazzo]

Do we want to add riscv64 to the images we build officially?

Setpeoctitle, fundamentally, works on a best-effort basis. A module that doesn't crash on import but results in being no-op is already ok, if we can't do anything better. So, ok, even if tests with ps can't work, but we manage to try a module import, that might be better than nothing.

Maybe we could tag the failing tests with a pytest.mark.skip_qemu and run pytest -m "no skip_qemu" in the build-cross-wheel` job?

[justeph]

Hi @dvarrazzo , I opened a merge request to add support for riscv64. I added a pytest.mark.skip_on_qemu mark and added tests command inbuild-cross-wheel job. #156