A Recursive Feature Elimination Random Forest Approach
Code, data and evaluation artifacts for a study on prioritizing Common Vulnerabilities and Exposures (CVEs) for security operations centres (SOCs) using only public vulnerability metadata.
The central methodological claim of this repository is negative and reproducible: once every feature is restricted to information available at the disclosure date, and once the evaluation is moved from a random split to a temporal split, the incremental value of a learned exploitability model over the public exploitation signals it consumes is small, and the differences that remain are mostly not distinguishable from sampling noise.
Scope of the artifact. The model answers one disclosure-time question: how likely is this CVE to enter CISA KEV within 90 days of publication, judged only from what was available at publication. It is an early-warning and prioritization artifact, not a risk register, and it is only defined for records whose exploitation outcome is still unknown. A CVE that KEV has already listed is a known outcome rather than a prediction, so its band must not be read as a statement about its current risk, and a confirmed-exploited record can legitimately carry a low prediction band. That is a scope boundary of an early-warning tool, and the recall figures in this repository quantify it rather than hide it. KEV is the evaluation label and is never an input: a predictor that only raises a record after KEV confirms it would add nothing beyond KEV itself, which is the criticism this study makes of signals that react only after confirmation. A deployment that needs a register rather than a warning should mark KEV-listed records as a separate state at the decision layer and keep that marker out of the model and out of its metrics.
Target. A CVE is labelled positive if it is added to the CISA Known Exploited Vulnerabilities (KEV) catalogue within 90 days of its disclosure date. Three properties motivate this definition over plain catalogue membership:
- the outcome window is identical for every record, so no record is observed longer than another;
- every window is fully observed at the catalogue snapshot used here (version 2026.09.11), because the latest publication date in the corpus is 2025-12-31 — 254 days earlier — so zero records are right-censored;
- CVEs already KEV-listed at disclosure are excluded, since their exploitation is public knowledge and nothing about them is being predicted (49 records).
CVEs absent from the catalogue are described as not listed, never as non-exploited.
Corpus. 44,474 CVE records published during 2025 with a CVSS v3.1 vector, of
which 109 are positive (0.245%). Cutting the KEV catalogue by dateAdded inside
2025 — the natural shortcut — would have missed 30 CVEs published in 2025 but
catalogued in 2026 and silently labelled them as not exploited; the full catalogue
is therefore used.
Features. Thirteen candidates: eight CVSS v3.1 ordinals (attack vector, attack
complexity, privileges required, user interaction, scope, and the confidentiality,
integrity and availability impacts), the CVSS base score, the disclosure-time EPSS
score, the CWE class frequency fitted on the training partition only, the sum of
the three impact ordinals, and the interaction term (AV+1)(AC+1)(PR+1).
Recursive feature elimination with cross-validation retains 7 of the 13.
Vulnerability age is deliberately not a feature: with the prediction moment fixed at disclosure it would be identically zero for every record.
Disclosure-time EPSS. Each CVE's EPSS feature is the score from the first daily
snapshot published on or after its disclosure date, taken from the FIRST archive
(372 snapshots, 2025-01-01 to 2026-01-07; median lag one day). No value is imputed
from a later snapshot, so the feature carries no information from after the
prediction moment. The EPSS model revision is recorded per record (snapshots up to
2025-03-16 carry v2023.03.01, later ones v2025.03.14).
By contrast, NVD does not publish record histories, so the CVSS vector, base score and CWE class are the current values at retrieval. The protocol is therefore retrospective with respect to those fields, and the paper makes no disclosure-time claim for them.
Splits. Temporal, by NVD publication date:
| Partition | Window | Records | Positives |
|---|---|---|---|
| Train | 2025-01-01 – 2025-07-31 | 25,951 | 70 |
| Validation | 2025-08-01 – 2025-09-30 | 7,363 | 18 |
| Test | 2025-10-01 – 2025-12-31 | 11,160 | 21 |
Protocol. Every decision is made without the test partition: feature
selection and hyperparameter tuning use training; the imbalance scheme, the
probability calibration map and the operating point use validation; the test
partition is read once, at the end. Resampling lives inside an imblearn pipeline,
so during cross-validation a sampler only ever sees the training side of a fold,
and the derived features are recomputed after synthesis so that deterministic
relations between features survive it. Models are compared at a matched queue
capacity — the backlog fraction a score may flag is fixed on validation and both
scores are read at their own threshold for that fraction — because a raw score
threshold means different things for differently calibrated scores.
| Metric | Random Forest | EPSS baseline | Paired difference (95% CI) |
|---|---|---|---|
| Recall (own validation threshold) | 0.9048 | 0.4762 | +0.4286 [+0.2222, +0.6364] |
| Precision | 0.0059 | 0.0067 | −0.0009 [−0.0038, +0.0018] |
| MCC | 0.0589 | 0.0439 | +0.0150 [−0.0102, +0.0400] |
| ROC-AUC | 0.8807 | 0.7833 | — |
| PR-AUC (average precision) | 0.0159 | 0.0074 | +0.0085 [−0.0029, +0.0409] |
| Brier (raw scores) | 0.0629 | 0.0019 | +0.0610 [+0.0576, +0.0644] |
| Brier (Platt-calibrated) | 0.00187 | 0.00189 | — |
Intervals are 95% paired-bootstrap intervals over 2,000 resamples of the same test records (the p-value for the average-precision difference is 0.128 and for MCC 0.248; the recall and Brier differences are the only ones that exclude zero).
Interpretation. The forest ranks better and recovers more exploited CVEs at every matched queue capacity, but its advantage on average precision and MCC is not statistically established on 21 test positives, its raw probabilities are far worse calibrated than the baseline until wrapped in a calibration step, and a CVSS-only score remains close behind. The one clearly significant difference — a recall gain of +0.43 at roughly twice the queue size — is an operating-point choice rather than a superiority claim. The repository and the manuscript accordingly withdraw the claim that the forest outperforms EPSS.
Repeated stratified cross-validation (5 folds × 3 repeats) inside training plus validation puts average precision at 0.0410 (standard deviation 0.0229, 95% interval 0.012–0.085), an interval wide enough that smaller differences are not treated as findings.
├── datasets/ input data and provenance log
│ ├── nvd_cve_data_2025.csv 44,523 NVD records (CVSS v3.1, published 2025)
│ ├── kev_full.csv(+.meta.json) CISA KEV catalogue, version 2026.09.11 (1,709 entries)
│ ├── epss_disclosure_2025.csv disclosure-time EPSS score per CVE (+ snapshot date, lag)
│ └── DATA_LOG.md retrieval dates, filters, SHA-256 of every input
├── scripts/
│ ├── fetch_nvd_range.py NVD API v2 retrieval
│ ├── fetch_kev_full.py CISA KEV catalogue retrieval
│ ├── fetch_epss_disclosure.py FIRST daily-archive retrieval (372 snapshots)
│ ├── explore_labels.py target-definition exploration
│ ├── explore_target_and_split.py horizon/right-censoring and split sizing
│ ├── rev_common.py corpus, labels, SMOTENC sampler, metrics, bootstrap
│ ├── rev_run.py full experimental pipeline
│ ├── rev_tables.py generates every table and figure in the manuscript
│ └── rev_smoketest.py building blocks check
├── revision/
│ ├── REVIEW_MAPPING.md reviewer item → fix → evidence
│ ├── outputs/ corpus with split IDs, results, manifest, selection logs
│ └── data/predictions_test.csv raw per-record test predictions
├── IEEEtranBST2/ manuscript source (IEEEtran, conference, A4)
│ ├── draft0.tex / draft0_anonim.tex named and double-blind versions
│ ├── refs.bib, tables_*.tex, figs/ bibliography, generated tables, figures
│ └── draft0.pdf / draft0_anonim.pdf compiled PDFs (6 pages A4)
├── app/ FastAPI serving prototype described in the manuscript
├── tests/ prototype tests
└── docs/proposal-final.pdf original study proposal
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
# 1. retrieve the inputs (only needed if you want to rebuild the datasets)
python scripts/fetch_kev_full.py
python scripts/fetch_epss_disclosure.py # 372 snapshots, ~25 min cold; uses cache if present
python scripts/fetch_nvd_range.py # needs NVD_API_KEY for a fast run
# 2. run the experiment (~16 min on an Apple M3; writes revision/outputs/)
python scripts/rev_run.py --force
# 3. regenerate the manuscript tables and figures
python scripts/rev_tables.py
# 4. compile the manuscript (tectonic, or pdflatex + bibtex)
cd IEEEtranBST2
tectonic draft0.tex
python3 make_anonim.py && tectonic draft0_anonim.texfetch_epss_disclosure.py caches the raw daily snapshots under
datasets/epss_daily_cache/ (about 670 MB); the cache is excluded from version
control and re-downloaded when absent. The experiment itself needs only the three
derived files that ship in datasets/, so step 1 can be skipped.
Every selection step is deterministic under seed 42.
| Source | Version / snapshot | Records used |
|---|---|---|
| NVD API v2 | retrieved for published 2025 CVEs with CVSS v3.1 | 44,523 |
| CISA KEV | catalogue 2026.09.11, released 2026-09-11 | 1,709 |
| FIRST EPSS daily archives | 372 snapshots, 2025-01-01 – 2026-01-07 | 44,522 scored |
SHA-256 digests for every input, the per-snapshot digests, the query parameters,
the missing-value rules, the hyperparameter search space and the threshold
procedure are recorded in revision/outputs/reproducibility.json and
datasets/DATA_LOG.md.
Python 3.11. Model fitting and evaluation: scikit-learn 1.6.1, imbalanced-learn 0.14.2, XGBoost 3.2.0, pandas 2.3.3, NumPy 2.2.6. The serving prototype adds FastAPI, Uvicorn, Pydantic and SQLAlchemy.
app/ contains the inference service described in the manuscript: a thin wrapper
that resolves a single CVE identifier, assembles the selected features, and returns
the raw model score, the calibrated probability, the severity label, the operational
band, the blended prioritization score and an action for each band.
The prototype serves the same artifact the package ships, verified rather than asserted. It loads the revised classifier, applies the Platt calibrator before anything is banded, resolves CWE frequency from the map fitted on the training partition, and reads the operational band thresholds from the artifact metadata instead of carrying its own copy. Scoring all 400 records of the frozen comparison sample through both the prototype and the published package returns identical raw scores, identical calibrated probabilities and identical blended scores, with zero disagreements on either band.
That check was not vacuous. It found one drift, now fixed: records carrying no CWE class were being assigned a frequency of zero, whereas the training corpus contained an empty CWE category and the pipeline had learned its frequency, so 6 of the 17 affected records crossed an operational band boundary. The lookup now resolves to the empty category and a regression test pins it.
Only the source is distributed, and the artifact is not vendored here. Install the package for the classifier, the calibrator, the CWE frequency map and the band thresholds:
pip install CVE-RiskEngine-RF
IEEEtranBST2/draft0.pdf is the named version and draft0_anonim.pdf the
double-blind version; both are 7 pages on A4. The two files are kept in
sync by make_anonim.py, which strips the author block, the photographs, the
biographies and the repository URL from the submission version.
The manuscript is in preparation; venue and citation details will be added once it has been submitted. Until then, cite this repository directly.
@misc{wardana2026exploitability,
author = {Reza Yuzron Wardana and Hari Purnama},
title = {Exploitability-Driven {CVE} Prioritization for Security Operations:
A Recursive Feature Elimination Random Forest Approach},
year = {2026},
note = {Manuscript in preparation},
howpublished = {\url{https://github.com/rezayw/CVE-RiskEngine-RF}},
}No license is attached yet; the author should choose one before reuse. The repository redistributes derived metadata from NVD, CISA and FIRST, and the upstream terms of those sources apply to the data. Third-party reference papers and vendor material gathered during the study are deliberately not included.
Author. Reza Yuzron Wardana, S.Kom. — Master's programme in Cybersecurity,
School of Electrical Engineering and Informatics, Institut Teknologi Bandung.
23525008@mahasiswa.itb.ac.id
Advisor. Dr. Phil. Eng. Hari Purnama, S.Si., M.Si. — STEI, Institut Teknologi Bandung.