-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathreqeval.html
More file actions
270 lines (251 loc) · 11 KB
/
Copy pathreqeval.html
File metadata and controls
270 lines (251 loc) · 11 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
<title>NLP4RE'20</title>
<!-- Bootstrap -->
<link href="css/bootstrap.min.css" rel="stylesheet">
<link href="css/custom.css" rel="stylesheet">
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.7.0/css/font-awesome.min.css">
<!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
<!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
<!--[if lt IE 9]>
<script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js"></script>
<script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
<![endif]-->
</head>
<body>
<nav class="navbar navbar-default navbar-fixed-top">
<div class="container">
<div class="navbar-header">
<a class="navbar-brand custom" href="#">
NLP4RE'20
</a>
</div>
<div class="collapse navbar-collapse" id="global-nav">
<div class="navbar-right">
<ul class="nav navbar-nav">
<li>
<a href="index.html">
Home
</a>
</li>
<li>
<a href="cfp.html">Call for Papers</a>
</li>
<li>
<a href="reqeval.html">
<b>
ReqEval Task
</b>
</a>
</li>
<li>
<a href="program.html">
Keynote & Program
</a>
</li>
<li>
<a href="organization.html">Organization</a>
</li>
<li>
<a href="previous.html">Previous editions</a>
</li>
</ul>
</div>
</div>
</div>
</nav>
<div id="start" class="jumbotron">
<div class="container">
<div class="row">
<div class="col-md-6">
<div style="color:floralwhite">
<h2 style="text-align:left;color:#fff;margin-top:20px;margin-bottom:10px;">Third Workshop on
Natural Language Processing for Requirements Engineering
</h2>
<p class="lead">REFSQ2020 Workshop, June 23rd (Online)</p>
</div>
<p>
<a class="btn" href="https://easychair.org/my/conference?conf=nlp4re20" role="button">Submit
here!</a>
</p>
</div>
<div class="col-md-6">
</div>
</div>
</div>
</div>
<div class="container start">
<div class="row">
<div class="col-xs-8">
<h2>ReqEval Task: Ambiguity Detection and Disambiguation in Requrements Specifications</h2>
<h3>Overview</h3>
<p class="text-justify">
Ambiguity is the phenomenon that occurs when a sentence can be interpreted in more than one way.
The notion of nocuous ambiguity in requirements specifications was first introduced by Chantree
<i>et al.</i> (2006).
It is defined as the ambiguity that leads to multiple interpretations by human readers, be it in the
form of their disagreement on the interpretation or acknowledgment of multiple possible readings.
Having multiple interpretations might lead to misunderstanding the requirements, and thus cause
significant implications on the success of the development process.</p>
<p class="text-justify">Detecting nocuous ambiguity in requirements specifications has been applied so
far to several ambiguity types such as coordination (Chantree et al., 2006) and referential (Yang et
al. 2010, 2011). </p>
<p class="text-justify">
Referential ambiguity (also called anaphora ambiguity) occurs when a pronoun is preceded by multiple
antecedents. For example, the pronoun "it" in the following sentence, has three syntactically valid
antecedents (in bold), although only the latter two are semantically valid:
</p>
<blockquote class="blockquote">
<p class="mb-0">The <strong>procedure</strong> shall convert the <strong>24 bit image</strong> to an
<strong>8 bit image</strong>, then display <strong>it</strong> in a dynamic window.</p>
<footer class="blockquote-footer"><cite title="SemEval">SemEval 2015 task 6</cite></footer>
</blockquote>
<p class="text-justify">
Detecting and resolving such an example automatically is challenging but necessary for establishing
a common understanding on what this requirement means.
</p>
<p class="text-justify">
While the ambiguity topic is not new from a Computational Linguistics point of view, there is little
attention given to the domain specificity of requirements specifications, which in turn makes NLP
generic tools less applicable, and requirements-specific solutions less generalizable.
</p>
<p class="text-justify">
ReqEval 2020 is an attempt to create a benchmark for the requirements engineering community on
essential NLP problems that will inspire both researchers and industry practitioners to come up with
innovative solutions and build up future directions.
Task 1 involves referential ambiguity found in requirements specifications from various domains. The
task introduces a dataset of 200 sentences that are manually annotated with being ambiguous or
unambiguous.
</p>
<h3>Dataset</h3>
<p class="text-justify">
The dataset is annotated by five annotators with an expertise in Software Engineering and/or
Computational Linguistics. A sentence is marked as nocuous if it has an ambiguous referential case
acknowledged by at least one of the annotators, or a disagreement on the interpretation of this
ambiguity.
</p>
<p class="text-justify">
The dataset contains 200 sentences, out of which 102 are marked as ambiguous. It consists of 6
domains as described in the figure below. The entire dataset will be split into ⅔ training set and ⅓
test set. The dataset can be downloaded <a href="https://doi.org/10.5281/zenodo.4471411">here.</a>
</p>
<img src="img/dataset.png" height="300" class="center-block">
<h3>The Task</h3>
<p class="text-justify">
The participants should create a system that is able to solve the task in two folds:
<ul>
<li><i>Detection:</i> Detecting the sentences that contain ambiguity.</li>
<li><i>Resolution:</i> For the sentences that are found as unambiguous, they should be
disambiguated with the most likely antecedent.</li>
</ul>
</p>
<p class="text-justify">
Only the training set will be published before the workshop in CSV format. The annotations of the
sentences in the training set will be published in a separate results file where each answer is
associated with a unique ID of the corresponding sentence. In order to get a fair evaluation similar
to the one that we will conduct using the test set, we will publish evaluators. Evaluators will be
implemented in a way that makes them easy to use, ideally taking one results file as an input and
producing a corresponding output file that reports the different evaluation metrics on that
particular results file. Evaluators will be published shortly after the training set.
</p>
<p class="text-justify">
The participating systems should return a results file similar to the one provided for the training
set. Then, the systems will be evaluated using the test set that will be kept unseen until the
workshop date.
</p>
<p class="text-justify">
For information about the task, please <a href="organization.html">contact us.</a>
</p>
<h3>Evaluation</h3>
<p class="text-justify">
For both sub-tasks, the evaluation metrics will be precision and recall, as follows:
<ul>
<li> Detection:
<ul>
<li>Precision = number of sentences correctly detected as ambiguous divided by the total number detected by the system as ambiguous</li>
<li>Recall = number of sentences correctly detected as ambiguous divided by the total number annotated by humans as ambiguous</li>
</ul>
</li>
<li> Resolution:
<ul>
<li>Precision = number of correctly resolved anaphors divided by the total number of anaphors attempted to be resolved</li>
<li>Recall = number of correctly resolved anaphors divided by the total number of unambiguous anaphors </li>
</ul>
</li>
</ul>
</p>
<p class="text-justify">
In addition, the systems will be evaluated across domains; i.e., having the system built and/or trained on a certain domain, how would it perform on another domain.
</p>
<h3>References</h3>
<p class="text-justify">
<ol>
<li> <a href="http://alt.qcri.org/semeval2015/task16/">SemEval 2015 Task 16: Detecting Nocuous Ambiguity</a> </li>
<li>Yang, Hui, Anne De Roeck, Vincenzo Gervasi, Alistair Willis, and Bashar Nuseibeh. <i>"Analysing anaphoric ambiguity in natural language requirements."</i> Requirements engineering 16, no. 3 (2011): 163.
</li>
<li>Yang, Hui, Anne De Roeck, Vincenzo Gervasi, Alistair Willis, and Bashar Nuseibeh. <i> "Extending nocuous ambiguity analysis for anaphora in natural language requirements."</i> In 18th IEEE International Requirements Engineering Conference, pp. 25-34. IEEE, 2010.
</li>
<li>Chantree, Francis, Bashar Nuseibeh, Anne De Roeck, and Alistair Willis. <i>"Identifying nocuous ambiguities in natural language requirements."</i> In 14th IEEE International Requirements Engineering Conference (RE'06), pp. 59-68. IEEE, 2006.
</li>
</ol>
</p>
</div>
<div class="col-xs-4">
<div class="panel panel-default">
<div class="panel-heading">
<b>Important dates</b>
</div>
<div class="panel-body">
<i>Papers submission</i>: <strike>January 17th</strike> <b>January 24th</b>
<br />
<i>Authors notification</i>: February 7th
<br />
<i>Camera Ready</i>: February 28th
<br>
<i>Workshop</i>: June 23rd
<br />
</div>
</div>
</div>
<div class="col-xs-4">
<div class="panel panel-default">
<div class="panel-heading">
<b>Workshop organizers</b>
</div>
<div class="panel-body">
Sallam Abualhaija
<small class="text-muted">University of Luxembourg (Luxembourg)</small>
<br />
Davide Fucci
<small class="text-muted">Blekinge Institute of Technlogy (Sweden)</small>
<br /> Fabiano Dalpiaz
<small class="text-muted">Utrecht University (The Netherlands)</small>
<br /> Xavier Franch
<small class="text-muted">Universitat Politècnica de Barcelona (Spain)</small>
</div>
</div>
<div class="section">
<a class="twitter-timeline" data-height="1000"
href="https://twitter.com/NLP4RE?ref_src=twsrc%5Etfw">Tweets by NLP4RE</a>
<script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
</div>
</div>
</div>
</div>
</div>
<!-- <footer class="footer">
<div class="container">
<p>Made with <3 by Davide Fucci</p>
</div>
</footer> -->
<!-- jQuery (necessary for Bootstrap's JavaScript plugins) -->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.12.4/jquery.min.js"></script>
<!-- Include all compiled plugins (below), or include individual files as needed -->
<script src="js/bootstrap.min.js"></script>
</body>
</html>