Pypi packaging and use new DASCore namespaces?

[d-chambers]

Hey @ahmadtourei,

I have been thinking a lot about the DASDAE ecosystem recently, particularly making non-DAScore DASDAE packages easier to use and better integrated together. Recently, I added the concept of namespaces for exactly this reason.

I think das-anomaly is an excellent package that is well positioned to take advantage of this new integration mode. Also, many more people would probably use if it was available on PyPI/conda-forge.

Would you be open to making some modifications to facilitate these two goals? I quickly brainstormed with codex and claude about how best to do this, and it came up with these suggestions. They certainly require a bit more discussion, but are a good starting point. I am happy to open a few PRs over the next couple weeks to make it happen. What do you think?

Integrate with PyPI

Prepare das-anomaly for a conservative first PyPI release by hardening packaging, reducing install friction, adding release automation, and validating the wheel before publishing.

Key Changes

• Rework dependencies in pyproject.toml:
  • Base install: dascore, matplotlib, numpy, scipy, pillow, scikit-learn, tensorflow.
  • mpi=["mpi4py"], test = [...], release = ["build", "twine"].
  • Keep all as the union of optional extras.
• Remove PyPI-hostile package behavior:
  • Rename shipped config_user.py to an example config and stop importing user-local placeholder config automatically.
  • Make import das_anomaly lightweight: no global warning suppression, no environment mutation.
• Add public command-line entry points:
  • das-anomaly-train
  • das-anomaly-detect
  • Optionally das-anomaly-count if the counting workflow is considered user-facing for v1.
• Add release automation:
  • Build source distribution and wheel with python -m build.
  • Publish through GitHub Actions using PyPI Trusted Publishing/OIDC, not long-lived tokens.
  • Trigger real PyPI publish only from GitHub Releases.

Release Workflow

• Add a CI job that runs on PRs:
  • install base package
  • install .[test]
  • run tests
  • build package
  • run twine check dist/*
  • test wheel install in a fresh environment
• Add a release workflow:
  • checkout repo
  • set up Python 3.12
  • install build
  • run python -m build
  • publish with pypa/gh-action-pypi-publish@release/v1
  • set environment: pypi and permissions: id-token: write
• Configure PyPI Trusted Publisher for the GitHub repo and release workflow
  before the first production upload.

Use new DASCore namespaces

The API might become a bit more ergonomic with the namespaces. We can also keep the old namespace for existing users. The goal is to make das-anomaly feel like a natural extension of DASCore. Users would keep using DASCore for loading, selecting, and chunking DAS data, then call anomaly-specific methods through a patch.anomaly or spool.anomaly namespace.

Generate images from a spool:

  import dascore as dc
  import das_anomaly

  spool = (
      dc.spool("/path/to/das/data")
      .update()
      .select(time=("2023-01-01", "2023-01-02"))
      .chunk(time=2, overlap=1)
  )

  spool.anomaly.generate_psds(
      output="psd_images/",  # Output vs output_name so we can plug a spool in here eventually
      data_unit="velocity",
      min_freq=0,
      max_freq=10,
      clip_value_max=7e-6,
  )

Work with single Patch.

  import dascore as dc
  import das_anomaly

  patch = dc.spool("/path/to/file")[0]

  psd = patch.anomaly.to_psd(
      data_unit="velocity",
      min_freq=0,
      max_freq=10,
  )

  psd.viz.waterfall()

Generate Training Images from a Clean Time Range

  import dascore as dc
  import das_anomaly

  clean_spool = (
      dc.spool("/path/to/das/data")
      .update()
      .select(time=("2023-01-01T00:00:00", "2023-01-01T06:00:00"))
      .chunk(time=2, overlap=1)
  )

  clean_spool.anomaly.generate_psds(
      output_path="training/normal/",
      data_unit="velocity",
      min_freq=0,
      max_freq=10,
  )

Run Detection on Generated PSD Images

  import das_anomaly

  results = das_anomaly.detect.run(
      psd_path="psd_images/",
      model_path="models/model_512.keras",
      train_images_path="training/normal/",
      density_threshold=20485.341,
      mse_threshold=0.02,
      output_path="results/",
  )

  print(results.summary)

For Python users, common settings should be passed directly as function arguments or as typed config objects.
These objects would just be Pydantic objects that can be serialized for command line use.

  from das_anomaly.config import PSDConfig

  cfg = PSDConfig(
      output_path="psd_images/",
      data_unit="velocity",
      min_freq=0,
      max_freq=10,
      time_window=2,
      overlap=1,
      clip_value_max=7e-6,
  )

  spool.anomaly.generate_psds(cfg)

For CLI/script users, the same config could live in a YAML, TOML, or JSON file:

  psd:
    output_path: psd_images/
    data_unit: velocity
    min_freq: 0
    max_freq: 10
    time_window: 2
    overlap: 1
    clip_value_max: 7e-6

  detect:
    psd_path: psd_images/
    model_path: models/model_512.keras
    train_images_path: training/normal/
    density_threshold: 20485.341
    mse_threshold: 0.02
    output_path: results/

[ahmadtourei]

Hey @d-chambers,

Thanks for working on this. Great ideas! Pypi installation support has been in my to do list for quite a while…

It would be nice to add namespaces capability for this package. Currently, we 1) generate PSD images from a spool, 2) train the autoencoder on some of them, and 3) detect anomalies within them. The first step can certainly be supported by a spool namespace.

Eventually, I think we can tweak the workflow to 1) generate PSDs only for training (data with background noise only), 2) train the model on them, and 3) for each patch in the chucked spool, use a namespace to i) generate PSD (in memory) and ii) apply the detect function to flag anomaly (and save the PSD and/or raster plot on disk if flagged as anomaly). This will help to better handle storage needa and improve i/o and potentially make the workflow faster as we’ll avoid saving many many PSD images on disk (if not the user is not interested in saving the ones detected as background noise). In that case, namespace can be used in the detection phase as well. But let’s start with the first phase, PSD generation.

Please feel free to start a PR and ping me wherever I can help!