-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathefficient_annotations.html
More file actions
531 lines (439 loc) · 30.9 KB
/
Copy pathefficient_annotations.html
File metadata and controls
531 lines (439 loc) · 30.9 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
<!DOCTYPE html>
<html lang="en">
<!-- ======= Head Section - DON'T CHANGE ======= -->
<head>
<meta charset="utf-8">
<meta content="width=device-width, initial-scale=1.0" name="viewport">
<title>Effecient Annotations</title>
<link href="assets\img\Tricuspid.PNG" rel="icon">
<!-- Google Fonts -->
<link href="https://fonts.googleapis.com/css?family=Open+Sans:300,300i,400,400i,600,600i,700,700i|Raleway:300,300i,400,400i,500,500i,600,600i,700,700i|Poppins:300,300i,400,400i,500,500i,600,600i,700,700i" rel="stylesheet">
<!-- Vendor CSS Files -->
<link href="assets/vendor/aos/aos.css" rel="stylesheet">
<link href="assets/vendor/bootstrap/css/bootstrap.min.css" rel="stylesheet">
<link href="assets/vendor/bootstrap-icons/bootstrap-icons.css" rel="stylesheet">
<link href="assets/vendor/boxicons/css/boxicons.min.css" rel="stylesheet">
<link href="assets/vendor/glightbox/css/glightbox.min.css" rel="stylesheet">
<link href="assets/vendor/remixicon/remixicon.css" rel="stylesheet">
<link href="assets/vendor/swiper/swiper-bundle.min.css" rel="stylesheet">
<!-- Main CSS File -->
<link href="assets/css/style.css" rel="stylesheet">
</head>
<!-- ======= Body Section ======= -->
<body>
<!-- ======= Navigation Bar - DON'T CHANGE ======= -->
<header id="header" class="fixed-top">
<div class="container d-flex align-items-center justify-content-between">
<a href="index.html" class="logo"><img src="assets/img/logo.png" alt="" class="img-fluid"></a>
<nav id="navbar" class="navbar">
<ul>
<li><a class="nav-link scrollto" href="index.html">Home</a></li>
<li><a class="nav-link scrollto" href="index.html#projects">Projects</a></li>
<li><a class="nav-link scrollto" href="index.html#team">People</a></li>
<li><a class="nav-link scrollto" href="index.html#resources">Resources</a></li>
<li><a class="nav-link scrollto" href="index.html#funding">Funding</a></li>
<li><a class="nav-link scrollto" href="index.html#partners">Partners</a></li>
<li><a class="nav-link scrollto" href="index.html#workwithus">Vacancies</a></li>
<li><a class="nav-link scrollto" href="index.html#contact">Contact</a></li>
</ul>
<i class="bi bi-list mobile-nav-toggle"></i>
</nav>
</div>
</header><!-- End Navigation Bar -->
<!-- ======= Title Section ======= -->
<section id="project-title" class="d-flex align-items-center">
<div class="container position-relative" data-aos="fade-up" data-aos-delay="100">
<div class="row">
<div class="col-12 text-left">
<h1>Active Learning for creating a Biobank of expert annotations in Echocardiography</h1>
<!-- <h2><i> Eman Alajrami <sup>1</sup>, Elisabeth S Lane <sup>1</sup>, Jevgeni Jevsikov <sup>1</sup>, James P Howard <sup>2</sup>, Matthew J Shun-shin <sup>2</sup>, Graham D Cole <sup>2</sup>, Darrel P Francis <sup>2</sup>, Massoud Zolgharni <sup>1,2</sup><i></h2>
<h3><sup>1</sup> School of Computing and Engineering, University of West London, London, United Kingdom<h3>
<h3><sup>2</sup> National Heart and Lung Institute, Imperial College, London, United Kingdom<h3> -->
</div>
</div>
</div>
</section><!-- End Title -->
<!-- Main Webpage Section -->
<main id="project-main">
<!-- ======= Project Body Section ======= -->
<section id="project-body">
<!-- ======= Project Introduction ======= -->
<div class="project-intro">
<div class="container">
<div class="row justify-content-left">
<div class="col-12">
<div align="justify">
<hr>
<h1><b>Introduction </b> </h1>
<br>
<h3> This project addresses the challenge of scarce annotations; using deep learning and active learning (AL) techniques to reduce annotation costs and efforts while maintaining high segmentation accuracy and improving clinical workflow efficiency.
</h3>
<h3> This work presents several AL methods, contributing to the cost-effective medical image annotation. The first project comprehensively evaluates existing AL approaches, establishing a valuable baseline for comparing different strategies. The second project explores ensemble-based AL techniques, leveraging different loss functions and uncertainty scoring methods. In the third project, a novel semi-supervised active learning method is developed, combining pseudo-labels from model predictions with expert annotations. The final project introduces an optimised clustering method using Fuzzy C-means for diversity sampling. </h3>
<h3> The outcome of the AL study was utilised to create a biobank of echo images for several projects to rank images and videos based on diversity. This method ensures that clinicians focus on labelling the images that will most effectively improve the AI model's performance, reducing the overall number of images that need to be annotated.</h3>
</div>
</div>
</div>
</div>
</div>
<!-- ======= Project Dataset ======= -->
<div class = "project-dataset">
<div class="container project-container">
<div class="row justify-content-left">
<div class="col-12 col-md-12 col-lg-6 col-xl-6">
<figure>
<img class="img-fluid" src="assets\img\projects\Efficient_Annotations\Sample_image.png" alt="Annotation" >
<div class="caption" >Fig. 1 A Sample image from the Unity dataset</div>
<!-- <figcaption>Fig. 1 A Sample image from the Unity dataset</figcaption> -->
</figure>
</div>
<div class="col-6 col-md-6 col-lg-6 col-xl-6">
<div align="justify">
<h2><b>Dataset</b></h2>
<h3>Three different datasets of cardiac imaging are used for our experiments:
the CAMUS dataset, the Unity dataset, and the consensus testing dataset.<br><br>
<b>CAMUS dataset</b> is a public, fully annotated dataset for 2D echocardiographic assessment.
All the details of the CAMUS dataset are available on the official website (https://www.creatis.insa-lyon.fr/Challenge/camus/index.html).
<br><br>
<b>Unity dataset</b>is a private dataset extracted from 1224 videos of the apical four-chamber echocardiographic view retrieved
from Imperial College Healthcare NHS Trust's echocardiogram database.
The images are obtained using ultrasound equipment from GE and Philips manufacturers.
The acquisition of these images is ethically approved by the Health Regulatory Agency
(Integrated Research Application System identifier 243023).<br><br>
It contains 2800 images sampled from different points in the cardiac cycle and labelled by a pool of
experts using our in-house online labelling platform (https://unityimaging.net).
This dataset was used for model developments (i.e., training and validation) and split into 70% for training, 15% for validation and 15% for testing.
<br><br>
<b>Consensus dataset:</b> It was utilised for the testing and was curated from a series of investigations conducted over three working days in 2019, years away from the development dataset. The testing dataset consisted of 100 A4C videos, from which 200 end-diastolic and end-systolic frames were selected automatically using a previously developed model. 11 human experts labelled each image using the same platform, mutually blinded to the labels of the others. The average of these 11 annotations was used for each image to create its GT. This provided high-quality consensus reference annotations, which served as a uniquely robust and representative benchmark for performance evaluations.
</h3>
<br>
</div>
</div>
</div>
</div>
</div>
<!-- ======= Project Architecture ======= -->
<div class = "project-architecture">
<div class="container">
<div class="row justify-content-left">
<div class="col-12">
<h2><b>Network Architecture</b></h2>
</div>
<div class="col-12 col-md-12 col-lg-6 col-xl-6">
<div align="justify">
<h3><br><br>The network architecture utilised in this study is the Monte Carlo dropout (MCD) U-Net with a depth of 5, designed explicitly for Bayesian AL in LV segmentation tasks. The overall architecture is depicted in the above figure. This architecture integrates dropout layers throughout the network to facilitate MCD for uncertainty estimation. The MCD U-Net was used as the baseline model for all of the following studies.
</h3>
</div>
</div>
<div class="col-12 col-md-12 col-lg-6 col-xl-6">
<img class="img-fluid" src="assets\img\projects\Efficient_Annotations\Unet_no_num.jpeg" alt="IMAGE PLACEHOLDER" >
<div class="caption" >Fig. 2 The customised Bayesian U-Net architecture</div>
<!-- <figcaption> Fig. 2 The customised Bayesian U-Net architecture </figcaption> -->
</div>
<br><br>
</div>
</div>
</div>
<!-- ======= Project Study 1 ======= -->
<div class = "project-study-one">
<div class="container">
<hr>
<div class="col-12">
<h2><b>Study 1: Active learning for left ventricle segmentation</b></h2>
</div>
<br>
<div align="justify">
<h3>This study evaluates existing AL approaches ( uncertainty and representativeness methods), establishing a valuable baseline for comparing different strategies.
</h3>
</div>
<div class="row justify-content-left">
<div class="col-12">
<br>
<h2><b>Datasets and Methods</b></h2>
</div>
<div class="col-12 col-md-12 col-lg-6 col-xl-6">
<div align="justify">
<h3><br>
Unity and CAMUS datasets were used in this study for model development and testing.
<br><br> A pool-based AL methodology is applied throughout, encompassing four steps: training the U-Net model on the initial annotated data (L), calculating the model uncertainty scores or representativeness scores on the unlabelled pool of data (U), selecting the top-ranked batch of images (K) to obtain their labels from oracle and add them to L and remove them from U, and finally retrain the model on the updated L. These steps are repeated until the optimal number of AL iterations is reached.
<br><br>
Random sampling and a variety of different selective sampling approaches are used for selecting the next batch of images from the unlabelled pool. The most common AL selection strategy is uncertainty sampling, where the most uncertain unlabelled images are queried for annotation. Such uncertainty methods include: Pixel-wise, Max_Entropy, and MCD ensemble-based methods using MCD_Eentropy, variance (MCD_Var), and Bayesian active learning with disagreement (BALD).
</h3>
</div>
</div>
<div class="col-12 col-md-12 col-lg-6 col-xl-6">
<img class="img-fluid" src="assets/img/projects/Efficient_Annotations/Da.jpg" alt="IMAGE PLACEHOLDER" >
<!-- <div class="caption" >Fig. 2 The customised Bayesian U-Net architecture</div> -->
<!-- <figcaption> Fig. 2 The customised Bayesian U-Net architecture </figcaption> -->
</div>
<br><br>
<div class="col-12">
<div align="justify">
<h2><b>Implementation</b></h2>
<h3>Tensorflow and Keras frameworks are used for the development of DL models, and training was conducted using an Nvidia RTX3090 GPU. U-Net was trained using binary cross-entropy loss and ADAM optimiser with a learning rate of 0.0001 for 200 epochs with early stopping and a patience of 10. Images were resized to 512x512, and a fixed batch size of 8 was applied. For the CAMUS dataset, we selected 10% of the initial training data as L, and U will be the remaining 90%. For the Unity dataset, we chose 4% as the initial L, and the remaining is for the U, which will be used as an oracle.</h3>
</div>
</div>
<div class="col-12">
<h2><b>Evaluation</b></h2>
<div align="justify">
<h3>The model was evaluated using the Dice-Coefficient (DC) metric, and it is computed between the ground truth and the inferred prediction for each image in the testing dataset. Then the mean of Dice scores of all images was calculated to present the model's accuracy. </h3>
</div>
<div class="text-left">
<br>
<!-- <a class="btn btn-primary" href="" role="button">Download the code</a>-->
</div>
</div>
<div class="row justify-content-left">
<div align="justify">
<h2><b>Results: </b></h2>
</div>
<div class="col-12 col-md-12 col-lg-6 col-xl-6">
<img class="img-fluid" src="assets\img\projects\Efficient_Annotations\Merge_unc_5.jpg" alt="IMAGE PLACEHOLDER">
<div class="caption" >Fig. 3 Active learning performance on CAMUS and Unity datasets</div>
<!-- <figcaption> Fig. 3 Active learning performance on CAMUS and Unity datasets</figcaption> -->
</div>
<div class="col-6 col-md-6 col-lg-6 col-xl-6">
<div align="justify">
<h3>Fig. 3 shows the performance of uncertainty sampling techniques compared to random selection on CAMUS and Unity datasets.
Our experiments show that the Max_Entropy is the best uncertainty strategy compared to others on CAMUS and Unity datasets. It achieved 97.7% of the entire dataset performance on CAMUS using 25% of the annotated dataset. At the same time, the other methods required approximately 35% to approach similar performance. Thus, it reduced the annotation cost by 10% to get that performance.
<br><br> For the Unity dataset, the Max_Entropy significantly outperformed all the other methods from the early stages of AL achieving 98.3% and ~ 99% of the entire dataset performance using only 7% and 20% of the labels, respectively.
</h3>
</div>
</div>
</div>
</div><br><br>
<div class="text-center">
<img class="img-fluid" src="assets/img/projects/Efficient_Annotations/Picture2.jpg" alt="model" style="width: 65%; height: auto;">
</div>
<br><br>
<h3> More details on this study can be found at: https://www.sciencedirect.com/science/article/pii/S016926072400107X
</h3>
</div>
</div>
<!-- ======= Project Study 2 ======= -->
<div class = "project-study-two">
<div class="container">
<hr>
<div class="col-12">
<h2><b>Study 2: Ensembles-based active learning for LV Segmentation</b></h2>
</div>
<br>
<div align="justify">
<h3>This study explores the use of established loss functions in ensemble-based AL for LV segmentation compared to the MCD ensemble-based, and different network architectures ensemble-based methods. </h3>
</div>
<div class="row justify-content-left">
<div class="col-12">
<br>
<h2><b>Datasets and Methods</b></h2>
</div>
<div class="col-12">
<div align="justify">
<h3><br>
Unity and CAMUS datasets were used in this study for model development and testing.
<br><br> The MCD U-Net model with dropout layers is used as the baseline model. Additionally, ResUnet and ResUnet++ are used for the ensemble-based method with different DL architectures. We customised their implementation by adding a dropout layer with a dropout probability of 0.3 after each encoder to have stochastic models.
<br><br>
A pool-based AL and different sampling methods were used for selecting the samples, including random and various uncertainty approaches, maximum entropy, variance, and BALD.
<br><br>
Ensemble-based AL approaches:
<br>
<ul>
<li> MCD ensemble-based: MCD U-Net was trained on labelled data and used to predict each image from the unlabelled pool, with 100 predictions to measure its uncertainty. </li>
<li> Multiple architectures ensemble-based: U-Net, ResUnet, and Re sUnet++ were trained on labelled data to predict unlabelled samples, with the predictions used to calculate uncertainty scores for each image. </li>
<li> Different loss functions ensemble-based Three ensembles of the U-Net with three different loss functions (Binary cross entropy (BCE), Dice, and BCE + Dice) were trained.</li>
</ul>
</h3>
</div>
</div>
<br>
<div class="col-12">
<div align="justify">
<h2><b>Implementation and Evaluation</b></h2>
<h3>Implementation, training settings, and evaluation were conducted like the previous study. </h3>
</div>
</div>
<div class="row justify-content-left">
<div align="justify">
<h2><b>Results: </b></h2>
<h3>The U-Net baseline was trained on the whole dataset, achieving a mean DC of 0.946 and 0.914 for the CAMUS and Unity testing datasets, respectively. As shown in the following figure, the results show that the proposed method (using three loss functions) outperformed the random, MCD, and multiple architecture ensemble-based methods for all the uncertainty sampling techniques on both datasets. The figure demonstrates that our proposed ensemble-based AL approach on CA MUS crossed 98% of the maximum performance using less than 30% of annotations for all uncertainty methods. However, 50% and 40% of annotations are needed for entropy and BALD, respectively, to achieve that performance using other ensemble-based methods. </h3>
</div>
<div class="text-center">
<img class="img-fluid" src="assets/img/projects/Efficient_Annotations/Picture3.jpg" alt="model" style="width: 75%; height: auto;">
<div class="caption" >Ensemble-based results for the CAMUS dataset</div>
</div>
<br><br>
<h3><br> The following figure shows that the introduced method outperformed the alternatives on Unity, achieving 98% of the maximum performance utilising 7% of annotations. However, more than 10% of annotations, 208 images, are used to reach that performance using other approaches for all uncertainty strategies. </h3>
<div class="text-center">
<img class="img-fluid" src="assets/img/projects/Efficient_Annotations/Picture4.jpg" alt="model" style="width: 75%; height: auto;">
<div class="caption" >Ensemble-based results for the UNITY dataset</div>
</div>
<br><br>
<h3><br> More details on this study can be found at: https://www.pure.ed.ac.uk/ws/portalfiles/portal/409667993/9782832512319_1_.PDF#page=103</h3>
</div>
</div>
</div>
</div>
<!-- ======= Project Study 3 ======= -->
<div class = "project-study-three">
<div class="container">
<hr>
<div class="col-12">
<h2><b>Study 3: Semi-supervised Active Learning SSAL for LV Segmentation</b></h2>
</div>
<br>
<div align="justify">
<h3>This study investigates semi-supervised active learning for LV segmentation in echocardiography, aiming to reduce the need for extensive manual expert annotations. A novel technique for identifying reliable pseudo-labels is proposed.</h3>
</div>
<div class="row justify-content-left">
<div class="col-12">
<br>
<h2><b>Datasets and Methods</b></h2>
</div>
<div class="col-12">
<div align="justify">
<h3><br>
In this study, the Unity data set was used for model development, while the Consensus dataset was used for model testing.
<br><br> The SSAL method selects reliable pseudo-labels from the mid-range of uncertainty scores in the unlabelled pool, representing the majority of samples, alongside expert labels of uncertain images to optimise annotation. A validation step is applied to choose the most confident pseudo-labels.
<br><br>
The phases of our method are as follows:
<br>
<ul>
<li> The U-Net model is initially trained on the initial labelled data. At each AL iteration, uncertainty scores are computed for every image in the unlabelled pool using the model’s prediction. Then, images are ranked based on these scores, and the most uncertain samples are queried for annotation. </li>
<li> Uncertainty scores for remaining unlabelled samples are normalised; The highest frequency bin in the uncertainty histogram determines the range of images as candidate pseudo labels. </li>
<li> A threshold-shifting is applied to validate candidate pseudo-label quality. We analysed masks created at 0.4, 0.5, and 0.6 thresholds, focusing on variance. Low variance indicates strong model confidence, signifying stable segmentation across threshold adjustments. Reliable pseudo-labels are chosen from the low variance of the predictions, with post-processing applied before training.</li>
<li> The final batch including expert annotations for uncertain images and reliable pseudo-labels is transferred from the unlabelled set to the labelled set, and the model is fine-tuned with the updated labelled data. These steps are iterated until AL iterations are met.</li>
</ul>
</h3>
</div>
</div>
<br>
<div class="col-12">
<div align="justify">
<h2><b>Implementation and Evaluation</b></h2>
<h3>Implementation, training settings, and evaluation were conducted in a similar way to the previous study.</h3>
</div>
</div>
<div class="row justify-content-left">
<div align="justify">
<h2><b>Results: </b></h2>
<h3>Results show a significant reduction in annotation efforts by up to 93%, achieving 99% of the maximum accuracy using only 7% of labelled data. The study contributes to efficient annotation strategies in medical image segmentation.</h3>
</div>
<div class="text-center">
<img class="img-fluid" src="assets/img/projects/Efficient_Annotations/Picture5.png" alt="model" style="width: 75%; height: auto;">
</div>
<br><br>
<h3><br> More details on this study can be found at: https://openreview.net/pdf?id=g9T6yLuMJR</h3>
</div>
</div>
</div>
</div>
<!-- ======= Project Study 4 ======= -->
<div class = "project-study-four">
<div class="container">
<hr>
<div class="col-12">
<h2><b>Study 4: Diversity Sampling using Clustering-based AL in Echocardiography</b></h2>
</div>
<br>
<div align="justify">
<h3>This study introduces an optimised clustering approach using Fuzzy cmeans (Fcmean) for diversity sampling in AL. The proposed method was compared against the benchmark Greedy K-Center and random sampling methods to evaluate its effectiveness. Additionally, different feature extraction methods were applied, and PCA was employed to reduce the dimensionality of the extracted features, facilitating efficient clustering and similarity measurement. This introduced methodology aims to enhance the efficiency and effectiveness of AL in the high-dimensional context of LV segmentation in echocardiography.</h3>
</div>
<div class="row justify-content-left">
<div class="col-12">
<br>
<h2><b>Datasets and Methods</b></h2>
</div>
<div class="col-12">
<div align="justify">
<h3><br>
In this study, the Unity data set was used for model development, while the Consensus dataset was used for model testing.
<br><br> A typical pool-based AL framework is applied. Random, Entropy, two different clustering-based diversity sampling (Greedy K-Center and the optimised Fcmeans clustering), and hybrid approaches (diversity and uncertainty) were used to select the next batch of images from the unlabelled pool for annotation.
<br>
</h3>
</div>
</div>
<br>
<div class="col-12">
<div align="justify">
<h2><b>Implementation and Evaluation</b></h2>
<h3>Implementation, training settings, and evaluation were conducted in a similar way to the previous study. </h3>
</div>
</div>
<div class="col-12">
<div align="justify">
<h2><b>Results</b></h2>
<h3>The results of this study will be published soon and more details and the link will be provided once published.</h3>
</div>
</div>
<div class="col-12">
<div align="justify">
<h2><b>Future Work</b></h2>
<h3>The future work of this study is a promising opportunity for future PhD research.
<br>
Integrating AL with Semi-Supervised and Self-Supervised Learning by developing methodologies to utilise unlabelled data during the AL process under very few label regimes is a promising research direction. Additionally, verification techniques should be developed to ensure the quality of pseudo-labels in SSAL frameworks.
<br>
Handling the noisy labels by applying effective techniques to tackle the noisy labels and integrating these algorithms with AL for a more robust model with fewer and high-quality annotations.
</h3>
</div>
</div>
</div>
</div>
</div>
<div class="container">
<hr>
<div class="row justify-content-left">
<div class="col-12">
<div class="text-left">
<h1> <b> Project Team <b></h1> <br>
<h6> <a href="https://www.uwl.ac.uk/staff/massoud-zolgharni" target="_blank" rel="noopener noreferrer"> Professor Massoud Zolgharni </a></h6>
<h6> <a href="https://www.uwl.ac.uk/staff/nasim-dadashi-serej" target="_blank" rel="noopener noreferrer"> Dr Nasim Dadashi Serej </a></h6>
<h6> <a href="https://www.researchgate.net/profile/Eman-Alajrami" target="_blank" rel="noopener noreferrer"> Dr Eman Alajrami </a> </h6>
<h6> <a href="https://twitter.com/intsav_?lang=en-gb" target="_blank" rel="noopener noreferrer"> Dr Jevgeni Jevsikov</a></h6>
</div>
</div>
</div>
</div>
<div class="container">
<hr>
<div class="row justify-content-left">
<div class="col-12">
<div class="text-left">
<h1> <b> References </b></h1><br>
<h6> <a href="https://www.sciencedirect.com/science/article/pii/S016926072400107X" target="_blank" rel="noopener noreferrer"></a></h6>
<h6> <a href="https://www.pure.ed.ac.uk/ws/portalfiles/portal/409667993/9782832512319_1_.PDF#page=103" target="_blank" rel="noopener noreferrer"></a></h6>
<h6> <a href="https://openreview.net/pdf?id=g9T6yLuMJR" target="_blank" rel="noopener noreferrer"></a></h6>
</div>
</div>
</div>
</div>
</section> <!-- End project-body -->
</main> <!-- End #main -->
<!-- ======= Footer - DONT CHANGE ======= -->
<footer id="footer">
<div class="container d-md-flex py-4">
<div class="me-md-auto text-center text-md-start">
<div class="copyright">
© Copyright <strong><span>IntSaV</span></strong>. All Rights Reserved
</div>
</div>
<div class="social-links text-center text-md-right pt-3 pt-md-0">
<a href="#" class="twitter"><i class="bx bxl-twitter"></i></a>
<a href="#" class="facebook"><i class="bx bxl-facebook"></i></a>
<a href="#" class="instagram"><i class="bx bxl-instagram"></i></a>
<a href="#" class="google-plus"><i class="bx bxl-skype"></i></a>
<a href="#" class="linkedin"><i class="bx bxl-linkedin"></i></a>
</div>
</div>
</footer><!-- End Footer -->
<div id="preloader"></div>
<a href="#" class="back-to-top d-flex align-items-center justify-content-center"><i class="bi bi-arrow-up-short"></i></a>
<!-- Vendor JS Files -->
<script src="assets/vendor/aos/aos.js"></script>
<script src="assets/vendor/bootstrap/js/bootstrap.bundle.min.js"></script>
<script src="assets/vendor/glightbox/js/glightbox.min.js"></script>
<script src="assets/vendor/isotope-layout/isotope.pkgd.min.js"></script>
<script src="assets/vendor/php-email-form/validate.js"></script>
<script src="assets/vendor/purecounter/purecounter.js"></script>
<script src="assets/vendor/swiper/swiper-bundle.min.js"></script>
<!-- Template Main JS File -->
<script src="assets/js/main.js"></script>
</body>
</html>