statistics progres artifacts 2016 2017 2018

[timm]

[timm]

nearly everyone got the badge they asked for (exception: one or two authors did not realize the "reproduced" meant reproduced by other people). and some authors even got upgrades (e.g. they asked for available and got "replicated").

[obaysal]

FSE 2018: 20 artifacts were submitted and reviewed out of 55 accepted papers (36% of FSE papers received ACM badges)

[obaysal]

FSE 2016: 17 artifacts were submitted and reviewed out of 74 accepted papers (23% artifacts submission rate).

[timm]

a quarter to a third? that's a large growth! imagine getting to the point where we can say SE research is that field that produces artifacts that let other people check their conclusions?