Reference implementation for: "Beyond the Efficiency-Efficacy Dilemma: A Critical Analysis of Transferability in Adversarial Malware Detection" IEEE RTSI 2026 — Track 4: Digitalisation, ICT, Cybersecurity and Resilient Operation of Energy Infrastructures (https://2026.ieee-rtsi.org/)
AAIE is a modular, automated framework for adversarial robustness testing of machine learning-based security classifiers. It implements a surrogate-based black-box transfer attack pipeline: adversarial examples are crafted against a differentiable surrogate model and transferred to an opaque target classifier, emulating realistic threat scenarios where the attacker has no direct access to the production system.
The core finding this implementation validates: single-step FGSM achieves higher cross-architecture transferability than iterative PGD when attacking a Random Forest target via a neural network surrogate — a counterintuitive result with direct implications for scalable adversarial stress-testing of deployed ML detectors.
AAIE/
├── AAIE_experiment.ipynb # Main experiment notebook (Colab-ready)
├── README.md # This file
├── requirements.txt # Dependencies
├── paper/
│ └── RTSI2026_AAIE.pdf # Conference paper (added after publication)
└── figs/ # Generated result figures
├── attack_success_rate.png
├── transferability_gap_comparison.png
└── der_ri_combined.png
AAIE implements a seven-component pipeline design:
| Module | Acronym | Function |
|---|---|---|
| Input & Context Module | MICo | Collects input data and experiment metadata for reproducibility |
| Attack Specification Module | ASM | Translates a security goal into algorithmic attack specifications |
| Attack Generator | ATGen | Composes gradient-based attacks (FGSM, PGD) and transfer-based variants |
| Transfer Module | TM | Trains surrogate, applies white-box attacks, evaluates transferability against black-box target |
| Orchestrator | — | Initializes, schedules, and terminates experiments |
| Evaluation & Scoring Module | ESM | Computes ASR, Transferability Score, DER, and Resilience Index |
| Provenance Layer | PAL | Logs all parameters, seeds, and results for audit and reproducibility |
The Transfer Module is the focus of this study. It operates as follows:
EMBER2018 features
│
▼
┌─────────────────┐ pseudo-labels ┌──────────────────┐
│ Target Model │ ──────────────────────▶ │ Surrogate Model │
│ (Random Forest)│ │ (Neural Network)│
└─────────────────┘ └────────┬─────────┘
│
FGSM / PGD attack
│
▼
adversarial examples x_adv
│
transfer to target
│
▼
ASR / TS / DER / RI
EMBER2018 — Elastic Malware Benchmark for Empowering Researchers
| Property | Value |
|---|---|
| Source | Anderson & Roth (2018), arXiv:1804.04637 |
| Format | Static Portable Executable (PE) features (JSONL) |
| Original size | ~1M samples |
| This study (stratified subset) | 9,693 train / 1,195 test |
| Class balance | ~50% benign / 50% malicious |
Download: https://github.com/elastic/ember
AAIE extracts a 30-dimensional feature vector from each sample across six static PE categories:
| Category | Features | Count |
|---|---|---|
| Byte histogram statistics | mean, std, max, min, median, fraction-above-mean | 6 |
| Byte entropy statistics | mean, std, max, high-entropy fraction | 4 |
| String characteristics | count, average length, printable ratio | 3 |
| General file metadata | size, virtual size, debug/relocation/resource/signature/TLS flags, export/import counts, symbol count | 10 |
| PE header fields | subsystem type, COFF timestamp | 2 |
| Structural counts | section count, data directory count, DLL import count, export count | 4 |
| Padding | — | 1 |
| Total | 30 |
All features are normalized with sklearn.preprocessing.StandardScaler (fit on training data only).
Emulates production-grade ML malware detectors commonly deployed in endpoint security products.
| Hyperparameter | Value |
|---|---|
| n_estimators | 200 |
| max_depth | 15 |
| class_weight | balanced |
| random_state | 42 |
Trained via pseudo-labeling (target RF predictions as labels) to approximate the target's decision boundary without direct access to its parameters.
| Component | Specification |
|---|---|
| Architecture | Fully-connected: 30 → 64 → 32 → 2 |
| Activations | ReLU (hidden), Softmax (output) |
| Loss | Cross-entropy |
| Optimizer | Adam (lr=1e-3, weight_decay=1e-4) |
| Training | Full-batch, deterministic |
Both attacks are applied in feature space on the surrogate, then transferred to the target RF.
Single-step attack (Goodfellow et al., ICLR 2015):
Iterative attack (Madry et al., ICLR 2018), 10 steps, step size α = ε/4:
Perturbation budgets evaluated:
ε ∈ {0.01, 0.02, 0.03, 0.05, 0.075, 0.10, 0.15, 0.20} (L∞ norm)
| Metric | Symbol | Description |
|---|---|---|
| Attack Success Rate | ASR | Fraction of adversarial examples that cause misclassification |
| Transferability Score | TS = ASR_target / ASR_surrogate | Cross-model attack effectiveness; TS > 1 means target is more vulnerable than surrogate |
| Detection Evasion Ratio | DER = 100 × (DR_adv − DR_clean) | Stealth against secondary OC-SVM anomaly detector; negative = better evasion |
| Resilience Index | RI ∈ [0,100] | Composite metric: 0.4 × accuracy preservation + 0.4 × attack resistance + 0.2 × stealth |
FGSM (single-step) achieves higher transferability than PGD (iterative) when attacking a Random Forest via a neural network surrogate.
This is statistically significant: paired t-test t = 5.234, p = 0.00112, Cohen's d = 0.742.
| ε | FGSM ASR | PGD ASR | FGSM Advantage |
|---|---|---|---|
| 0.050 | 0.321 | 0.278 | +13.5% |
| 0.100 | 0.454 | 0.362 | +20.3% |
| 0.150 | 0.529 | 0.437 | +17.4% |
| 0.200 | 0.527 | 0.452 | +14.3% |
FGSM achieves TS = 0.802 at ε = 0.15, with 19.4% higher average TS than PGD across all budgets.
| Condition | RI |
|---|---|
| Clean (no attack) | 100.0 |
| Under FGSM | 54.99 |
| Under PGD | 49.76 |
Both fall in the moderate resilience range (33–66), indicating that current ML malware detectors remain meaningfully vulnerable to transfer attacks.
PGD iteratively ascends the surrogate's loss landscape, overfitting to the neural network's local curvature and non-linear decision boundary geometry. When transferred to a Random Forest — whose tree-based partitioning creates fundamentally different decision boundaries — these overfit perturbations fail.
FGSM's single gradient step produces a coarser perturbation that aligns with generalizable vulnerability directions shared across model families, consistent with recent theoretical work on feature-level transferability (Li et al., 2025; Zheng et al., 2025).
torch==2.1.0
scikit-learn==1.3.0
numpy
pandas
matplotlib
seaborn
scipyInstall:
pip install -r requirements.txtOr in Colab (GPU recommended):
!pip install torch scikit-learn numpy pandas matplotlib seaborn scipy- Download EMBER2018: https://github.com/elastic/ember
- Place
train_features_*.jsonlandtest_features.jsonlin/content/ - Open
AAIE_experiment.ipynbin Google Colab - Run all cells — full experiment completes in < 15 minutes on T4 GPU
If you use this code, please cite:
@inproceedings{alevcan2026aaie,
author = {Alevcan, Veysel and Ali, Mohammad Furqan and
Sousa, Teresa and Campos, Lu{\'i}s Miguel and
Ntantogian, Christoforos},
title = {Beyond the Efficiency-Efficacy Dilemma: A Critical
Analysis of Transferability in Adversarial Malware
Detection},
booktitle = {Proceedings of the IEEE International Conference on
Environment and Electrical Engineering (RTSI)},
year = {2026},
note = {Track 4: Cybersecurity for Energy Infrastructures}
}📄 Paper PDF will be added to
paper/after publication.
This work was funded by the European Union under Grant Agreements:
- No. 101131292 — AIAS
- No. 101183162 — ANTIDOTE
- No. 101145872 — NITRO
- No. 101168490 — RECITALS
NITRO and RECITALS are supported by the European Cybersecurity Competence Centre (ECCC).
MIT License — see LICENSE for details.
| Name | Affiliation |
|---|---|
| Veysel Alevcan | COPELABS, Lusófona University & PDMFC SA, Lisbon |
| Mohammad Furqan Ali | COPELABS, Lusófona University, Lisbon |
| Teresa Sousa | PDMFC SA, Lisbon |
| Luís Miguel Campos | COPELABS, Lusófona University, Lisbon |
| Christoforos Ntantogian | Dept. of Informatics, Ionian University, Corfu |
Attackers demonstrate the performance of their adversarial attacks based on different datasets and victim models. This brings obstacles to objectively evaluating the adversarial attacks and measuring the robustness of deep learning models. In particular, large and high-quality datasets usually make it hard for adversaries/defenders to attack/defend. In this subsection, we provide an overview of the most prominent datasets employed in building DL-based defense systems and using adversarial machine learning approaches in the cybersecurity domain, organized per specific application/category. Herein, we can find a curated list of cybersecurity datasets.
- Network Intrusion Detection
- Malware Detection and Analysis
- Botnet Detection and Domain Generation Algorithms (DGAs)
- Cyber-Physical System (CPS) Security
- Spam Filtering
- Encrypted Traffic Analysis
- Fraud Detection
- User Identification and Authentication
- KDD Cup 1999: was created based on the DARPA 1998 dataset and inherit the same problems. Nevertheless, it is one of the most employed datasets until now for network intrusion detection. KDDCup99 includes full-packet data, break into subsets for training and testing. The network traffic is labeled in five classes: Normal, Probe, User to Root attacks, Remote to Local attacks, and DoS attacks.
- NSL-KDD: is an improved version of KDD Cup 1999 built to alleviate the redundancy problems, another often used dataset for network intrusion detection. However, it should be regarded that the underlying network traffic of NSL-KDD dates back to the year 1998. The dataset is labeled following the same structure as KDDCup99 and comprises separate sets for training and testing.
- CTU-13: is a dataset of botnet traffic captured in the CTU University, Czech Republic, in 2011. The dataset aimed to have a large capture of real botnet traffic mixed with normal and background traffic. The CTU-13 dataset includes thirteen captures (i.e., scenarios) of different botnet samples. In each scenario, we executed a specific malware, which employed several protocols and performed different actions.
- AWID: focuses on 802.11 networks. A small network environment with ten clients was designed to capture WLAN traffic in a packet-based format. In one hour, around 2 million packets were captured. 155 features were extracted from each packet. For generating malicious traffic, 15 specific attacks against the 802.11 network (e.g., Probe Request, and CTS Flooding) were executed. The dataset is labeled and divided into a training and a test subset.
- CIC-IDS2017: consists of labeled network flows, including full packet payloads in PCAP format, the corresponding profiles and the labeled flows and CSV files.
- CSE-CIC-IDS2018: includes seven different attack scenarios, namely Brute-force, Heartbleed, Botnet, DoS, DDoS, Web attacks, and infiltration of the network from inside. The attacking infrastructure includes 50 machines, and the victim organization has five departments includes 420 PCs and 30 servers. This dataset includes each machine's network traffic and logs files from the victim side, along with 80 network traffic features extracted from captured traffic using CICFlowMeter-V3.
- CIC-DDoS2019: contains many different DDoS attacks that can be carried out through application layer protocols using TCP/UDP. The taxonomy of attacks in the dataset is performed in terms of exploitation-based and reflection-based attacks. The dataset was collected in two separate days for training and testing evaluation. The training set was captured on January 12th, 2019, and contains 12 different kinds of DDoS attacks, each attack type in a separated PCAP file. The attack types in the training day include UDP, SNMP, NetBIOS, LDAP, TFTP, NTP, SYN, WebDDoS, MSSQL, UDP-Lag, DNS, and SSDP DDoS based attacks. The testing data was created on March 11th, 2019, and contains 7 DDoS attacks SYN, MSSQL, UDP-Lag, LDAP, UDP, PortScan, and NetBIOS.
- LITNET-2020: comprises real network attacks in Lithuanian-wide network with servers in four geographic locations within the country. The breakdown of the traffic types is: Smurf (0.13%), ICMP-Flood (0.03%), UDP-Flood (0.21%), TCP SYN-flood (8.22%), HTTP flood (0.05%), LAND Attack (0.12%), Blaster Worm (0.05%), Code Red Worm (2.77%), Spam Bot Detection (0.002%), Reaper Worm (0.003%), Scanning/Spread (0.01%), Packet Fragmentation Attack (0.001%), Normal (88.24%). The entropy across attack classes is lower at 0.333 than between normal and anomalous traffic at 0.522.
- TON_IoT: involves heterogeneous data sources collected from Telemetry datasets of IoT and IIoT sensors, operating systems datasets of Windows 7 and 10 as well as Ubuntu 14 and 18 TLS and Network traffic datasets. The datasets were collected from a realistic and large-scale network designed at the Cyber Range and IoT Labs, the School of Engineering and Information technology (SEIT), UNSW Canberra the Australian Defence Force Academy. A new testbed network was built for the industry 4.0 network, including the Internet of Things (IoT) and Industrial (IIoT) networks. The testbed was deployed employing multiple virtual machines and hosts of windows, Linux, and Kali operating systems to manage the interconnection between the three layers of IoT, Cloud, and Edge/Fog systems. Various attacking techniques, such as DoS, DDoS, and ransomware, against web applications, IoT gateways, and computer systems across the IoT/IIoT network. The datasets were gathered in parallel processing to collect several normal and cyber-attack events from network traffic, Windows audit traces, Linux audit traces, and telemetry data of IoT services.
- IoT-23: consists of twenty-three captures (called scenarios) of different IoT network traffic. These scenarios are divided into twenty network captures (PCAP files) from infected IoT devices (which will have the name of the malware sample executed on each scenario), and three network captures of real IoT devices network traffic (that have the name of the devices where the traffic was captured). A specific malware sample in a Raspberry Pi that used several protocols and performed different actions was executed on each malicious scenario.
- N-BaIoT: contains real traffic obtained from nine commercial IoT devices such as Danmini doorbell, Provision PT-737E security camera, and Ecobee thermostat. For each device, the data was collected under both normal operating conditions and several different attacks performed by BASHLITE (e.g., scan, junk, UDP, TCP, and COMBO) and Mirai botnets (e.g., scan, Ack, Syn, DP, and UDPplain). The dataset has 115 numerical features.
- IoTPOT: is a publicly available dataset that contains IoT threat samples collected by an IoT honeypot. The dataset initially comprises 500 malware samples, where most of them are categorized into four major families, including Linux.Gafgyt, Linux.Gafgyt.1, Trojan.Linux.Fgt and Mirai. The rest of the samples belong to relatively rare families such as Hajime, Tsunami, and LightAidra.
- Genome: has around 1,242 malicious Android applications gathered in 2010 and 2011. The samples are divided into 49 malware families that comprise almost all malware categories, namely Rootkit, Botnet, SMS Trojans, Trojan, Installer, and Spyware. The majority of genome applications come from unofficial Chinese marketplaces.
- Contagio-Mobile: is a part of contagiodump.blogspot.com. It offers an upload dropbox for sharing the malware samples and downloading any samples individually or in one zip.
- AndroZoo: is a growing collection of Android Applications collected from several sources (e.g., Google Play app market). It currently contains 17M different APKs, each of which has been analyzed by tens of different Antivirus products to know which applications are detected as malware.
- VirusShare: is a repository of malware samples to provide security researchers, incident responders, forensic analysts, and access to samples of live malicious code.
- Alexa Internet: Alexa Internet is often referred to simply as Alexa. It is a Web traffic information, metrics, and analytics provider. Alexa can provide to the visitors the following principal utilities:
- See data for Alexa Top Sites.
- Search for traffic data, such as ranking and bounce rate, for websites not included in the top-ranked sites.
- Download the Alexa toolbar, which allows them to see statistics for websites as they browse and records their browsing data for inclusion in Alexa statistics.
- Create a custom toolbar, which can be shared.
- OSINT: is feed from Bambenek Consulting, which includes DGA domains from 50 different DGA families and more than 800K malicious domain names. Moreover, more than 30 reverse-engineered DGAs are provided by DGArchive, which can be employed to create malicious domain names on an internal network.
- DGArchive: allows resolving or calculating domain names that are dynamically created by malware using DGAs.
- 360netlab: suspicious DGA from 360+VT Sandbox and PassiveDNS.cn. It contains 52 DGA families.
- AmritaDGA: is used as part of DMD2018, which comprises 20 DGA fanilies.
- UMUDGA: presents a collection of over 30 million manually-labelled algorithmically generated domain names decorated with a feature set ready-to-use for Machine Learning analysis.
- BATADAL: represents a water distribution network comprised of seven storage tanks with eleven pumps and five valves, controlled via nine Programmable Logic Controllers (PLCs). The network was created using epanetCPA, an open-source, object-oriented Matlab toolbox that allows for the injections of cyber-physical attacks and simulates the response of the network to these attacks. The dataset contains 43 variables, representing the tank water levels (7 variables), the flow and status of all the pumps (24 variables), as well as the inlet and pressure for the pumping stations and valves (12 variables). All variables are continuous, except the status of valve and pumps, represented by binary variables. The training data simulates hourly measurements collected for 356 days, resulting in 8.761 records. The test set comprises 2.089 records, which were gathered for 90 days. There exist seven attacks present in the test data. The attacks involved malicious actuator activation, sensor measurement manipulation, and PLC setpoint changes. Besides, the attacks were concealed from the SCADA system by replacing the PLC-to-SCADA communication data with the data recorded at the same hour during normal operation. BATADAL can be download freely on the website.
- SWaT: was built at the Singapore University of Technology and Design in 2016, which is available upon request. It is a scaled-down, fully operational water treatment plant and comprises a six-stage process. Each stage is equipped with several sensors and actuators. The sensors comprise water pumps, flow meters, valves that control inflow are the actuators, and chemical dosing pumps. The sensors and actuators of each stage are connected to the corresponding PLC, and the PLCs are connected to the SCADA systems. This dataset comprises seven days of recording under normal conditions and four days during which 36 attacks were conducted (e.g., false sensor readings, and false control signals). The entire dataset contains 946,722 records, labeled as either attacks or normal, with 51 attributes corresponding to the sensor and actuator data.
- WADI: was built by the authors of SWaT \cite{goh2016dataset} and contains several large water tanks that supply water to consumer tanks. The dataset contains 15 attacks whose goal is to stop the water supply to the consumer tanks. The attacks were conducted by opening valves and spoofing sensors readings, and partially concealed. WADI is significantly larger than the SWaT and BATADAL dataset, contains 1.221.372 data points (full dataset) and 126 features. WADI is available upon request.
- ICS Cyber Attack Power System Datasets: This dataset is split into three smaller datsets, which include measurements related to electric transmission system normal, disturbance, control, cyber attack behaviors. Measurements in the dataset include synchrophasor measurements and data logs from Snort, a simulated control panel, and relays.
- HAI: was collected from a realistic industiral control system (ICS) testbed augmented with a Hardware-In-the-Loop (HIL) simulator that emulates steam-turbine power generation and pumped-storage hydropower generation.
- Udacity: is mainly composed of video frames taken from urban roads. It provides a total number of 404,916 video frames for training and 5,614 video frames for testing. This dataset is challenging due to severe lighting changes, sharp road curves, and heavy traffic.
- Cityscapes: is a large-scale dataset containing diverse stereo video sequences recorded in street scenes from 50 different cities. It has high-quality pixel-level annotations of 5 000 frames and a more extensive set of 20 000 weakly annotated frames. The dataset is thus an order of magnitude larger than similar previous attempts.
- GTSRB: The German Traffic Sign Benchmark is a multi-class, single-image classification challenge held at the International Joint Conference on Neural Networks (IJCNN) 2011.
- Airsim: is a simulator for drones, cars, and more, built on Unreal Engine (we now also have an experimental Unity release). It is open-source, cross-platform, and supports software-in-the-loop simulation with popular flight controllers such as PX4 & ArduPilot and hardware-in-loop with PX4 for physically and visually realistic simulations. It is developed as an Unreal plugin that can be dropped into any Unreal environment. Similarly, we have an experimental release for a Unity plugin.
- Nuscenes: is a public large-scale dataset for autonomous driving. It enables researchers to study challenging urban driving situations using the full sensor suite of a real self-driving car.
- Apolloscape: is part of the Apollo project for autonomous driving, which is a research-oriented project to foster innovations in all aspects of autonomous driving, from perception and navigation to control. It hosts open access to semantically annotated (pixel-level) street view images and simulation tools that support user-defined policies. It is an evolving project in new datasets, and new capabilities will be added regularly. Various workshops and challenges will be hosted to encourage exchanges of ideas and jointly advance the state of the art in autonomous driving research.
- WEBSPAM-UK2007: is an extensive collection of annotated spam/nonspam hosts labeled by a group of volunteers. The base data is a set of 105,9M pages in 114,529 hosts in the—UK domain. The data was downloaded in May 2007 by the Laboratory of Web Algorithmics, Università Degli Studi di Milano.
- UtkMI: is labeled SMS messages dataset collected for mobile phone message research. It contains 5815 spam samples and 6153 harmful samples.
- Social honeypot: was collected from December 30, 2009 to August 2, 2010 on Twitter. The dataset contains 22,223 content polluters, their number of followings over time, 2,353,473 tweets, and 19,276 legitimate users, their number of followings over time and 3,259,693 tweets.
- SMS Spam Collection v.1: is a public set of SMS labeled messages that have been collected for mobile phone spam research. It has one collection composed of 5,574 English, real and non-encoded messages, tagged according to being legitimate (ham) or spam.
- ISCX VPN-nonVPN: was collected at University of New Brunswick and it contains raw pcap files of several traffic types. The dataset provides fine-grained labels which allows different categorization: application-based (e.g. AIM chat, Gmail, Facebook, etc), traffic-type-based (e.g. chat, streaming, VoIP, etc), and VPN/non-VPN.
- ISCX Tor-nonTor: was created available by the Canadian Institute for Cybersecurity. Three users were used for browser traffic collection and two users for communication (mail, FTP, etc.). There are eight different categories for TOR traffic: Browsing, audio, video, chat, mail, VoIP, FTP, and P2P.
- Open HTTPS: was constructed by crawling HTTPS websites over two weeks (September 2016). It contains full HTTPS raw PCAP files of crawling top 779 accessed HTTPS websites. The scan was made daily based, two times per day using Goolge Chrome and Mozilla Firefox Web browsers.
- QUIC: was captured at the University of California at Davis, which contains QUIC traffic of 5 Google services: Google Doc (1251 flows), Google Drive (1664 flows), Google Music (622 flows), Youtube (1107 flows), Google Search (1945 flows). The dataset contains time-series features: packet length, relative time, and direction. The dataset has already been pre-processed.
- German Credit Data: classifies people into good or bad credit risks.
- European Card: has 284,807 transactions spanning over a few days. Out of these, 492 transactions are fraud.
- IEEE-CIS: is real-world e-commerce transaction data containing 590,540 transactions where 20,663 are fraud.
- Balabit: includes the timing and positioning information of mouse pointers.
- Makeup: is a dataset of unpaired before and after makeup images.
- TWOS: is composed of normal and malicious activities collected from several host-based heterogeneous data sources (such as a mouse, keyboard, processes, and file system).