Weird test-order dependency

[gromgull]

I came across something weird with python3

I had one test that needed the functionality of another, i.e. the first modifies the server-state in a way that the second test also needs. The pattern is probably not recommended, but we did this:

The first test: test_b.py

import pytest

from tornado.gen import sleep

@pytest.mark.gen_test
def test_b():

    yield sleep(0.1) # the real case actually did something 

This passes.

The second test: test_c.py

import pytest
from .test_b import test_b

@pytest.mark.gen_test
def test_c():

    yield test_b()
    # [...] and something else

This also passes.

BUT! only if called test_c.py - if you rename the SAME test to test_a.py (or whatever gets sorted before test_b, it fails with:

test_a.py:8:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
env/lib/python3.6/site-packages/tornado/gen.py:1055: in run
    value = future.result()
env/lib/python3.6/site-packages/tornado/concurrent.py:238: in result
    raise_exc_info(self._exc_info)
<string>:4: in raise_exc_info
    ???
env/lib/python3.6/site-packages/tornado/gen.py:1143: in handle_yield
    self.future = convert_yielded(yielded)
env/lib/python3.6/functools.py:803: in wrapper
    return dispatch(args[0].__class__)(*args, **kw)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

yielded = <generator object test_b at 0x103469780>

    def convert_yielded(yielded):
        """Convert a yielded object into a `.Future`.

        The default implementation accepts lists, dictionaries, and Futures.

        If the `~functools.singledispatch` library is available, this function
        may be extended to support additional types. For example::

            @convert_yielded.register(asyncio.Future)
            def _(asyncio_future):
                return tornado.platform.asyncio.to_tornado_future(asyncio_future)

        .. versionadded:: 4.1
        """
        # Lists and dicts containing YieldPoints were handled earlier.
        if yielded is None:
            return moment
        elif isinstance(yielded, (list, dict)):
            return multi(yielded)
        elif is_future(yielded):
            return yielded
        elif isawaitable(yielded):
            return _wrap_awaitable(yielded)
        else:
>           raise BadYieldError("yielded unknown object %r" % (yielded,))
E           tornado.gen.BadYieldError: yielded unknown object <generator object test_b at 0x103469780>

Tornado doesn't think the function is a coroutine, because the CO_ITERABLE_COROUTINE flag is not set on the code.

This happens because the tests are run in alphabetical order, and if the test_b is run by itself once before this library calls tornado.gen.pytest on it: https://github.com/eugeniy/pytest-tornado/blob/master/pytest_tornado/plugin.py#L106

which in turn calls types.coroutine, which modifies the functions code object in place to set the flag: https://github.com/python/cpython/blob/master/Lib/types.py#L228-L241

I don't know if this can really be "fixed" - in the end our pattern of "called the test function" is probably not recommended, and it's trivial to work around by putting the shared code in it's own coroutine.

However, since we spent hours debugging this, it's maybe worth mentioning for others?