Sort out -manual filename convention vs meta element "interact" flag

[annevk]

Both doesn't seem like a good long term solution. The filename convention seems a little clearer.

Inspired by #5370.

cc @zcorpan @mrego @frivoal @fantasai @jgraham

[gsnedders]

The meta element flag should cause a lint failure anywhere outside of css (or rather, a flag that causes them to become manual without the filename also ending in -manual is a lint failure).

The reason why it's in css is to avoid mass-renaming files in there and breaking all existing results in the CSS test harness. Potentially, we could alter the build system such that tests with -manual or -visual got that stripped and the flag added as part of the build system, but that requires someone to do that.

[frivoal]

I agree that one convention would be much better than two. That said, no strong feelings from me in favor of naming vs flags. As long as it is clearly documented, and supported by the infrastructure (including the CSS test harness), I'm happy with either.

[fred-wang]

Are all the manual tests supposed to use either "interact" flag or -manual filename convention?

I had to add -manual suffix for CSS animations tests ( #11215 ) ; these tests use the "animation" flag but not the "interact" one...

cc @youennf

[frivoal]

as of today, there isn't a single flag for non-automatic tests. There are several ones, depending on why it cannot be automated:

• interact (the tester needs to do something)
• animated (the test is animated in final state)
• font (Requires a specific font to be installed at the OS level)
• userstyle (requires the tester to install a particular user stylesheet)

This situation is not good, but that's where we are for now.

We need to do two things:

• decide which convention we want to use going forward (-manual suffix seems nice to me)
• decide whether to convert existing tests, or to deprecate but continue to support old conventions. (convert seems ok with me, but we need to be really sure, as renaming thousands of files isn't nice on systems that have metadata linked to filenames).

[annevk]

Some thoughts:

1. It seems that outside /css we already decided upon -manual.
2. Renaming lots of files has been done quite a bit (at least outside /css). This mostly went fine (updating links can be a pain).
3. If you want to postpone renaming you could safelist all current usage and forbid further flag usage with the lint script.

[fred-wang]

Thanks my confusion is that the doc has general requirements for manual test (including -manual suffix):
https://web-platform-tests.org/writing-tests/manual.html

But additional requirements for CSS tests that I only found after #5293 (meta flags):
https://web-platform-tests.org/writing-tests/css-metadata.html#requirement-flags

[gsnedders]

If you want to postpone renaming you could safelist all current usage and forbid further flag usage with the lint script.

FWIW, we currently have css/* whitelisted for the CONTENT-MANUAL (and the related CONTENT-VISUAL) lint.

as of today, there isn't a single flag for non-automatic tests. There are several ones, depending on why it cannot be automated:

The canonical list for what makes them manual is content_is_css_manual in sourcefile.py: https://github.com/web-platform-tests/wpt/blob/master/tools/manifest/sourcefile.py#L537

Currently, this is:

{"animated", "font", "history", "interact", "paged", "speech", "userstyle"}

[gsnedders]

As for each of those:

• animated: these I don't think anyone is going to have automation testing, especially in a cross-browser way, hence I don't think it's worthwhile distinguishing these
• font: these can be automated with a manual step (installing the font), but it's probably not worthwhile adding links to the fonts in some machine readable form so they can be run automatically; the vast majority can be moved over to using web fonts (see also Tests with a "font" meta flag are marked as manual #9374)
• history: I have some memory of this being here because of there being tests that required "open this other page then reload" but I can't find any that don't also have interact, so maybe it should just go from the set
• interact: obviously
• paged: nothing per-se keeping it here except for No way to test paged media #7760; if we want to do this, it might be worth adding some way to mark these up even if wptrunner doesn't run them
• speech: I don't think there's ever been interest from implementors in running these, and it's unclear how one would test them?
• userstyle: like font these can be automated with a manual step (installing the user stylesheet), but whether it's worthwhile working on depends on whether we expect implementations to keep on shipping support for user stylesheets (several having already dropped support)

[frivoal]

• animated: these I don't think anyone is going to have automation testing, especially in a cross-browser way, hence I don't think it's worthwhile distinguishing these

Agreed. -manual is sufficient.

• font: these can be automated with a manual step (installing the font), but it's probably not worthwhile adding links to the fonts in some machine readable form so they can be run automatically; the vast majority can be moved over to using web fonts (see also Tests with a "font" meta flag are marked as manual #9374)

Agreed. using web fonts or -manual should be sufficient.

• history: I have some memory of this being here because of there being tests that required "open this other page then reload" but I can't find any that don't also have interact, so maybe it should just go from the set

Agreed. -manual is sufficient. If we really want, we could add some bit of metadata that specifically indicates that the page needs to be reloaded, but that seems overkill.

• interact: obviously

• paged: nothing per-se keeping it here except for No way to test paged media #7760; if we want to do this, it might be worth adding some way to mark these up even if wptrunner doesn't run them

paged isn't inherently non-automated. It just assumes a different UA (or a different UA mode).

Within paged tests (which need the right kind of UA anyway), some may be manual, but some may be automated, just the same way any other test is automated. These are orthogonal questions.

I'd keep paged as a flag, and wptrunner / any non paginating UA should skip them.

• speech: I don't think there's ever been interest from implementors in running these, and it's unclear how one would test them?

Like paged this seems more about filtering classes of UAs than about tests being manual.

• userstyle: like font these can be automated with a manual step (installing the user stylesheet), but whether it's worthwhile working on depends on whether we expect implementations to keep on shipping support for user stylesheets (several having already dropped support)

I don't think we need to speculate about implementation intents.

These can just be described as -manual for now. If we ever decide we care more, we can add a piece of machine readable metadata that indicates which user stylesheet to download and install, but I don't think this is high priority.

[zcorpan]

The reason why it's in css is to avoid mass-renaming files in there and breaking all existing results in the CSS test harness.

@plinss has fixed this, so this should unblock the renaming.

I suggest that we keep the metadata when doing the renaming, however when authoring new tests it would be good if it was sufficient to use only the file name convention. I'm not sure if the CSS test harness or something needs further changes to support that.

[zcorpan]

Moreover, we should update the linter/whitelist to complain about any new test that is manual according to flags but doesn't have -manual.

[plinss]

Yes, please keep the rest of the existing metadata for now. I can have the test harness extract the additional flags it needs from the filenames at some point, but it's not doing that yet.