An audit of NOTEARS and DAGMA through the lens of varsortability, plus a directed↔undirected (Bayesian-network ↔ Markov-network) bridge on real protein- signaling data.
Continuous-optimization structure learners such as NOTEARS (Zheng et al., 2018) and DAGMA (Bello et al., 2022) post strong numbers on simulated DAG benchmarks. Reisach et al. (2021) showed that much of that success can be an artifact called varsortability: in generically parameterized structural equation models, marginal variance grows along the causal order, so the order leaks through the variances and a trivial variance-sorting baseline can match the sophisticated methods.
This project reproduces that effect in a clean, uniform harness, pushes past the
usual "just standardize the data" takeaway using the scale-invariant
R²-sortability, and then turns the same machinery on the real Sachs
protein-signaling benchmark — where it also compares a directed model against an
undirected Gaussian Markov network learned on the identical data. The
reimplementation of prior methods is deliberately not the contribution; the audit
harness, the residual-structure analysis, and the directed↔undirected bridge are.
See docs/design.md for the full rationale and limitations.
All numbers come from the scripts in src/dag_audit/experiments/ on the
committed configuration. Seed counts are small for fast reproduction, so treat
these as illustrative and expect run-to-run variation.
1. Learner "skill" tracks varsortability. Across data spanning varsortability
0.12 → 1.00, the trivial sortnregress baseline's F1 correlates with
varsortability at Pearson r ≈ 0.96 — it is essentially a varsortability meter —
and at high varsortability it is competitive with, and often ahead of, NOTEARS
and DAGMA. High benchmark scores alone do not demonstrate discovery ability.
2. Standardization removes the signal. On high-varsortability data (≈0.92), standardizing every variable to unit variance drives varsortability to ≈0.50 and collapses every method's F1 — the trivial baseline most of all.
3. But "just standardize" is too glib. Standardization removes varsortability (→0.50) but not the scale-invariant R²-sortability (stays ≈0.68), and the learners keep an above-baseline edge on standardized data. The honest lesson is to report scale-invariant structure, not to assume standardization makes a benchmark safe.
4. Real biology behaves as the thesis predicts. On Sachs, varsortability is only ≈0.36 — the synthetic crutch is absent — and the learners recover the 17-edge consensus network poorly, matching the published literature:
| Method | SHD | F1 | Precision | Recall |
|---|---|---|---|---|
| NOTEARS | 21 | 0.21 | 0.27 | 0.18 |
| DAGMA | 18 | 0.28 | 0.33 | 0.24 |
| sortnregress | 39 | 0.25 | 0.17 | 0.47 |
5. Directed vs. undirected on the same data. Moralizing the learned NOTEARS DAG yields ≈10 undirected edges, while a graphical-lasso Markov network fit to the same data has ≈35, overlapping only modestly (Jaccard ≈0.25). The undirected model asserts many conditional dependencies the sparse DAG does not — a concrete view of the Bayesian-network ↔ Markov-network representation gap on correlated biological data.
- Python >= 3.10
git clone https://github.com/baksho/dag-discovery-audit.git
cd dag-discovery-audit
pip install -e .This installs the dag_audit package and its dependencies (numpy, scipy,
scikit-learn, networkx, matplotlib, and the official dagma package).
No external data is required for experiments 1–3:
python scripts/run_all.pyResults (JSON) are written to results/ and figures to results/figures/.
Experiment 4 needs the Sachs dataset. Fetch the CC BY 4.0 mirror on first use:
SACHS_DOWNLOAD=1 python -m dag_audit.experiments.exp4_sachs_bn_mnThe canonical origin of the data is the bnlearn distribution
(https://www.bnlearn.com/research/sachs05/); see src/dag_audit/data/sachs.py
for provenance, the consensus-graph choice, and the observational-subset caveat.
Everything is a plain function. A minimal end-to-end example:
from dag_audit.data import simulate_linear_sem, standardize
from dag_audit.methods import default_learners
from dag_audit.metrics import varsortability, structural_scores
ds = simulate_linear_sem(d=15, noise_scale_strategy="increasing", seed=0)
print("varsortability:", varsortability(ds.X, ds.W_true))
learn = default_learners()["NOTEARS"]
W_est = learn(ds.X)
print("F1 vs truth:", structural_scores(W_est, ds.B_true).f1)The directed↔undirected bridge:
from dag_audit.methods import graphical_lasso_graph, moralize, compare_moral_vs_ggm
W_dag = default_learners()["NOTEARS"](ds.X)
A_ggm = graphical_lasso_graph(ds.X)
print(compare_moral_vs_ggm(W_dag, A_ggm).as_dict())src/dag_audit/
data/ synthetic linear SEMs, the Sachs loader, and the optional GRN simulator
methods/ from-scratch NOTEARS, a DAGMA wrapper, baselines, and the undirected/bridge tools
metrics/ varsortability, R2-sortability, and structure-recovery metrics
experiments/ the four runnable experiments described above
scripts/ run_all.py
tests/ pytest suite for the metrics, NOTEARS, and moralization
docs/ design.md — rationale, results, and limitations
pip install -e ".[dev]"
pytestThe suite checks varsortability on hand-built and generated cases, that the from-scratch NOTEARS returns an acyclic graph and recovers a known chain, SHD behavior on reversed/perfect graphs, and that moralization marries parents of a v-structure.
- The varsortability effect and its reference code (Reisach et al., 2021) predate this work and are reproduced, not claimed as novel. The audit harness, the R²-sortability analysis, and the directed↔undirected bridge are the contributions.
- Hyperparameters are fixed, not tuned per instance; read the varsortability dependence, not the exact F1 gaps, as the finding.
- The committed Sachs run pools all experimental conditions; the observational subset is the principled choice and a one-line change.
- The Sachs consensus graph is a contested proxy, and its edge count is reported inconsistently (17/18/20) in the literature. This repo fixes the 17-edge version and documents it.
Full detail in docs/design.md.
- Zheng, X., Aragam, B., Ravikumar, P., & Xing, E. P. (2018). DAGs with NO TEARS. NeurIPS. arXiv:1803.01422.
- Bello, K., Aragam, B., & Ravikumar, P. (2022). DAGMA. NeurIPS. arXiv:2209.08037.
- Reisach, A. G., Seiler, C., & Weichwald, S. (2021). Beware of the Simulated DAG! NeurIPS. arXiv:2102.13647.
- Reisach, A. G., Tami, M., Seiler, C., Chambaz, A., & Weichwald, S. (2023). A Scale-Invariant Sorting Criterion. NeurIPS. arXiv:2303.18211.
- Sachs, K., Perez, O., Pe'er, D., Lauffenburger, D. A., & Nolan, G. P. (2005). Science, 308(5721), 523–529.
MIT — see LICENSE.