Problem with tornado using python 3.5 async

[felippemr]

Hi!

I am learning tornado and to do so I created a project: https://github.com/felippemr/resistance
I am trying to change my test suite to use pytest-tornado but I keep receiving this error:

platform darwin -- Python 3.5.1, pytest-2.9.1, py-1.4.31, pluggy-0.3.1
rootdir: /Users/XXXXXX/XXXXXX/XXXXXXX, inifile:
plugins: tornado-0.4.5
collected 1 items

tests/test_resources_api.py F

========================================================================== FAILURES ==========================================================================
______________________________________________________________________ test_hello_world ______________________________________________________________________

pyfuncitem = <Function 'test_hello_world'>

    @pytest.mark.tryfirst
    def pytest_pyfunc_call(pyfuncitem):
        gen_test_mark = pyfuncitem.keywords.get('gen_test')
        if gen_test_mark:
            io_loop = pyfuncitem.funcargs.get('io_loop')
            run_sync = gen_test_mark.kwargs.get('run_sync', True)

            funcargs = dict((arg, pyfuncitem.funcargs[arg])
                            for arg in _argnames(pyfuncitem.obj))
            if iscoroutinefunction(pyfuncitem.obj):
                coroutine = pyfuncitem.obj
                future = tornado.gen.convert_yielded(coroutine(**funcargs))
            else:
                coroutine = tornado.gen.coroutine(pyfuncitem.obj)
                future = coroutine(**funcargs)
            if run_sync:
                io_loop.run_sync(lambda: future, timeout=_timeout(pyfuncitem))
            else:
                # Run this test function as a coroutine, until the timeout. When completed, stop the IOLoop
                # and reraise any exceptions

                future_with_timeout = with_timeout(
                        datetime.timedelta(seconds=_timeout(pyfuncitem)),
                        future)
                io_loop.add_future(future_with_timeout, lambda f: io_loop.stop())
                io_loop.start()

                # This will reraise any exceptions that occurred.
>               future_with_timeout.result()

../../.virtualenvs/ressistance/lib/python3.5/site-packages/pytest_tornado/plugin.py:121:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <tornado.concurrent.Future object at 0x106ccbbe0>, timeout = None

    def result(self, timeout=None):
        """If the operation succeeded, return its result.  If it failed,
            re-raise its exception.

            This method takes a ``timeout`` argument for compatibility with
            `concurrent.futures.Future` but it is an error to call it
            before the `Future` is done, so the ``timeout`` is never used.
            """
        self._clear_tb_log()
        if self._result is not None:
            return self._result
        if self._exc_info is not None:
>           raise_exc_info(self._exc_info)
E           tornado.gen.TimeoutError: Timeout

../../.virtualenvs/ressistance/lib/python3.5/site-packages/tornado/concurrent.py:232: TimeoutError
================================================================== 1 failed in 5.13 seconds ==================================================================

I'm doing this test(it will fail):

import pytest
from resistance.app import make_app


@pytest.fixture
def app():
    return make_app()


@pytest.mark.gen_test(run_sync=False)
def test_hello_world(http_client, base_url):
    response = yield http_client.fetch(base_url + '/api/v0.1/resources/bed896005177cc528ddd4375')
    assert response.code == 200

Can you please help me?

[etataurov]

Hi!

You have a Timeout error because your app could not connect to mongo.
Here is why it happens:

• pytest-tornado creates and start its own io_loop for each test method (and make it current io_loop.make_current())
• you initialize motor client in global scope of settings.py and it happens before test start
• motor client expects to be run in default io_loop, but it will never be started, because pytest-tornado creates its own loops and doesn't touch default io_loop

So the best practice to work with pytest-tornado will be initializating stuff in functions/fixtures, but not in a global scope.
So you can solve your problem by changing settings.py like this

DATABASE_CONNECTION = None
def get_app_settings():
    global DATABASE_CONNECTION
    DATABASE_CONNECTION = _setup_db()
    return {"db": DATABASE_CONNECTION, "debug": True}

it will be called only from the app fixture, when the proper io_loop will be already set as current

[beregond]

Hey @etataurov - got similar problems with timeouts when using asyncio (and related aioredis) - maybe note in readme about changing default io_loop to something like this:

@pytest.fixture
def io_loop():
    from tornado.platform.asyncio import AsyncIOMainLoop
    return AsyncIOMainLoop()

could help others, when dealing with async? I spent some time on that, before realizing on the real problem.