Introducing input automation stubs

[wpt-issue-mover]

Originally posted as w3c/testharness.js#250 by @NavidZ on 22 Mar 2017, 16:07 UTC:

@foolip @jgraham @RByers
I created this issue to track adding the automation APIs we talked about here as the first stage. Is that okay to add it to testharnessreport.js for now?

[wpt-issue-mover]

Originally posted as w3c/testharness.js#250 (comment) by @gsnedders on 22 Mar 2017, 22:08 UTC:

Dupe of #142, I'd say?

[wpt-issue-mover]

Originally posted as w3c/testharness.js#250 (comment) by @gsnedders on 22 Mar 2017, 22:17 UTC:

Maybe not a dupe, as pointed out to me:

If we fix #142 literally, "Add a user-gesture helper function", then you can't get an arbitrary UA-initiated event given just a function to enable the user-gesture bit.

Realistically, we're only ever going to fix #142 like this.

[wpt-issue-mover]

Originally posted as w3c/testharness.js#250 (comment) by @NavidZ on 23 Mar 2017, 15:36 UTC:

So @gsnedders what do you think of the proposed API in #251 ? I know it is not complete by any means for example it doesn't have the keyboard events or other stuff. But I just wanted to start landing the changes and iterating over that and using it in different tests to be able to make it more complete with the feedback of different vendors.

[wpt-issue-mover]

Originally posted as w3c/testharness.js#250 (comment) by @foolip on 06 Apr 2017, 17:32 UTC:

To quote myself from public-test-infra:

However, I would like to go a bit further and treat this more like we treat any bit of API that tests [depend on]. Tests should simply assume that the APIs exist, and otherwise fail. Having stubs of the APIs could make test failures more explicit, but it seems like we could do without them.

Would this be workable for input automation, or is this issue entirely separate from the proposed API in https://navidz.github.io/web-input-automation/?

[wpt-issue-mover]

Originally posted as w3c/testharness.js#250 (comment) by @NavidZ on 06 Apr 2017, 17:35 UTC:

I believe this depends on the test to decide how to handle it. I mean the proposed API returns a promise. So I assume a well written test should fail if the returned promise failed and if it passed it should move on to the next step. Right?

[wpt-issue-mover]

Originally posted as w3c/testharness.js#250 (comment) by @foolip on 06 Apr 2017, 17:49 UTC:

Sigh, I didn't see w3c/testharness.js#251 yet. Now I have.

So, I wonder why a stub is needed. A test could look like this:

async_test(t => {
  var driver = new InputDriver();
  driver.sendpointerActions([...]);
  document.addEventListener('something', t.step_func_done());
});

It simply assume the API exists, and if it doesn't it'll fail, in this case by throwing an exception on the first line. Would that work?

(We could still implement the API in our own testharnessreport.js of course, or something that's more like our other LayoutTests-only APIs. It would need to be something where we have a path to making it work for Chrome release builds as well.)

[wpt-issue-mover]

Originally posted as w3c/testharness.js#250 (comment) by @NavidZ on 06 Apr 2017, 18:00 UTC:

I'm not sure failing at a particular line because of non-existence function is the pattern I see anywhere in wpt tests. I'm fine with that certainly if no one else has any objection.
However, putting that point aside, not all the tests use these APIs inside an async_test. I believe you have have an async_test and in different event listeners run the next action and yet keep updating that global async_test object. How about those use cases?

[wpt-issue-mover]

Originally posted as w3c/testharness.js#250 (comment) by @foolip on 06 Apr 2017, 18:20 UTC:

https://github.com/w3c/web-platform-tests/blob/master/battery-status/battery-promise.html is an example of a test that starts by calling navigator.getBattery() and will fail fast if it doesn't exist. Fullscreen tests are similar, and I think for the API under test, this is typical for web-platform-tests.

So, I'm trying to see if we can treat testing APIs just like the APIs under test, just assuming existence.

However, putting that point aside, not all the tests use these APIs inside an async_test. I believe you have have an async_test and in different event listeners run the next action and yet keep updating that global async_test object. How about those use cases?

I'm not sure I understand. In an async_test, all callbacks need to be wrapped in t.step_func, so that any exceptions that are thrown will cause the test to fail. This happens with the fullscreen tests, which call element.requestFullscreen() from a click event handler. However, in #4507 I made some adjustments to make the failure more obvious when the API doesn't exist, which can be useful.

async_test is generally the trickiest to get right, with test any exception thrown is a test failure and with promise_test you usually get the right behavior just by chaining promises in a way that's natural for the test.