All Errors not reported as Failure

[AFBarstow]

The all test results file [1] has results for 5 UAs and it includes about 40 test files that resulted in 4 Failures plus 1 Timeout. My expectation is that test failures like these are included in the less-than-two table and/or or the complete-fails table yet that didn't happen (see [2] and [3]). That seems like a bug to me.

/cc @bzbarsky @plehegar @siusin

[1] http://w3c.github.io/test-results/websockets/all.html
[2] http://w3c.github.io/test-results/websockets/less-than-2.html
[3] http://w3c.github.io/test-results/websockets/complete-fails.html

[darobin]

Are you talking about the lines that state "ERROR"? These aren't test failures, they're a problem with the test. The problem isn't that they should be reported, it's that they should be fixed so that they don't error when running. Reporting them as failures makes no more (but no less) sense than reporting them as passes: they are bugs and as data points they are meaningless.

[siusin]

@darobin
I agree that "errors" should not be considered as failures. Is it possible to produce another file for the "errors" besides the "less-than-2" or "complete-fails"? I can submit a PR to wptreport if you think it helps.

[darobin]

Yes, I think that would be useful. Also, maybe you could have wptreport list errors on the console when it runs? That might lead people to pay more attention to them.

[siusin]

SGTM :)

[bzbarsky]

These aren't test failures, they're a problem with the test.

But for purposes of determining interop these mean at best that we have less test coverage than we think. It seems to me that progressing to REC should require not having such things in the test suite...

[darobin]

I completely agree @bzbarsky, they shouldn't be glossed over. Reporting them as test failures is not the right option, but they need be reported.

[AFBarstow]

@siusin yes, putting failures^H^H^H^H^H^H^H^Herrors like the ones I reported in a separate file would be fine. Thanks!

[halindrome]

I don't think this ever happened. Prioritizing.