Skip to content

tickets/SP-2852: Update documentation of report generation #190

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ehneilsen merged 4 commits into from
Dec 17, 2025
Merged

tickets/SP-2852: Update documentation of report generation #190

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
81d955d
Describe cron gates
ehneilsen Dec 16, 2025
9ae556d
Remove instructions for prenight dashboard that does not exist anymore
ehneilsen Dec 16, 2025
0e04e30
Add link to prenight sim generation page in rubin_sim
ehneilsen Dec 16, 2025
880f017
Fix typos
ehneilsen Dec 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 2 additions & 6 deletions docs/prenight_sim_generation.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,5 @@
Pre-night simulation generation
===============================

Daily prenight briefing simulations are currently kicked off by a cron job on ``sdfcron001.sdf.slac.stanford.edu`` as user ``neilsen``.

This runs the script at ``/sdf/data/rubin/shared/scheduler/packages/rubin_sim/batch/run_prenight_sims.sh`` as a slurm job at 8:15am Pacific time every morning, and takes about 10 minutes to run.
The ``rubin_sim`` project holds this script in ``batch/run_prenight_sims.sh``.

Additional documentation can be found in the `docs` for `rubin_sim`.
Documentation for pre-night simulation generation can be found in the
`rubin_sim documentation <https://rubin-sim.lsst.io/prenight.html>`__ .
42 changes: 29 additions & 13 deletions docs/reports.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ These scripts are found in the `batch` directory of the `schedview_notebooks` re
Each of these scripts builds an environment in which to convert the notebook, executes the conversion, and updates a corresponding index it include the new reports.
There are currently two indexes of reports.
One index lists publicly accessible, which are served from `https://s3df.slac.stanford.edu/data/rubin/sim-data/schedview/reports/ <https://s3df.slac.stanford.edu/data/rubin/sim-data/schedview/reports/>`__.
Currently, the public reports are limited to a brief nigth summary.
Currently, the public reports are limited to a brief night summary.

Additional reports, currently requiring logging to the the USDF, can be found at `https://usdf-rsp-int.slac.stanford.edu/schedview-static-pages/ <https://usdf-rsp-int.slac.stanford.edu/schedview-static-pages/>`__.

Expand All @@ -38,6 +38,19 @@ The `cron` job runs the reports with the following entries::
15 5 * * * /opt/slurm/slurm-curr/bin/sbatch /sdf/data/rubin/shared/scheduler/packages/schedview_notebooks/batch/scheduler_nightsum.sh 2>&1 >> /sdf/data/rubin/shared/scheduler/schedview/scheduler_nightsum/scheduler_nightsum.out
30 7 * * * /opt/slurm/slurm-curr/bin/sbatch /sdf/data/rubin/shared/scheduler/packages/schedview_notebooks/batch/prenight.sh 2>&1 >> /sdf/data/rubin/shared/scheduler/schedview/prenight/prenight.out

If necessary, these cron jobs can be stopped from doing anything, even a user that does not own the cron job, if they have write access to ``/sdf/data/rubin/shared/scheduler/cron_gates/${SCRIPT}``.
This is accomplished using gate files: early in each script, the script checks for the existence of a file with name ``/sdf/data/rubin/shared/scheduler/cron_gates/${SCRIPT_NAME}/${USER}`` and aborts if it does not exist.
Any user with write access to ``/sdf/data/rubin/shared/scheduler/cron_gates/${SCRIPT_NAME}`` can create or remove files in that directory, so a user can cause these scripts to immediately abort
when started by a cron job owned by a different user by removing ``/sdf/data/rubin/shared/scheduler/cron_gates/${SCRIPT_NAME}/${CRON_JOB_USER}``.

So, to stop the `scheduler_nightsum.sh`` cron job owned by user ``neilsen``, remove the file ``/sdf/data/rubin/shared/scheduler/cron_gates/scheduler_nightsum/neilsen``,
and to stop the ``prenight.sh`` cron job owned by user ``neilsen``, remove the file ``/sdf/data/rubin/shared/scheduler/cron_gates/prenight/neilsen``.

The logs of the cron jobs (and any other executions of these scripts submitted using ``sbatch``) can be found in:
* ``/sdf/data/rubin/shared/scheduler/schedview/sbatch/scheduler_nightsum_%A_%a.out`` and
* ``/sdf/data/rubin/shared/scheduler/schedview/sbatch/prenight_%A_%a.out``
where ``%A`` is the slurm "Job array's master job allocation number" and ``%a`` is the slum "Job array ID (index) number".

The environment
---------------

Expand Down Expand Up @@ -81,7 +94,7 @@ The batch script that generates the prenight briefing reeport requeires a versio
pip install git+https://github.com/lsst/schedview.git@v0.19.0.dev1


Finally, update the bash scripts that need it (batch/prenight.sh and batch/scheduler_nightsum.sh),
Finally, update the bash scripts that need it (``batch/prenight.sh`` and ``batch/scheduler_nightsum.sh``),
for example::

source /sdf/group/rubin/sw/w_latest/loadLSST.sh
Expand All @@ -91,10 +104,10 @@ Even though we aren't using the environment provided by the source of `loadLSST.
it's still needed to get `conda` into our path.


Updating version of `schedview_notebooks` used by the `cron` job
----------------------------------------------------------------
Updating the version of `schedview_notebooks` used by the `cron` job
--------------------------------------------------------------------

The scripts submitted by the cron job supplied above use the version of schedview in `/sdf/data/rubin/shared/scheduler/packages/schedview_notebooks`, which is itsef a link to a directory for a specific version, e.g. `/sdf/data/rubin/shared/scheduler/packages/schedview_notebooks-v0.1.0`.
The scripts submitted by the cron job supplied above use the version of schedview in ``/sdf/data/rubin/shared/scheduler/packages/schedview_notebooks``, which is itsef a link to a directory for a specific version, e.g. ``/sdf/data/rubin/shared/scheduler/packages/schedview_notebooks-v0.1.0``.

To tag and install a new version to be used, start by deciding on a tag. Get sorted existing tags with::

Expand Down Expand Up @@ -128,6 +141,9 @@ Replace the symlink to point to your new one::
Updating other software used by the jobs
----------------------------------------

The jupyter notebooks used to generate the reports import ``schedview`` and related packages from subtirectories of ``/sdf/data/rubin/shared/scheduler/packages``.
New version should be added there to make them available.

Begin by determining the next available tag.

Get sorted existing tags with::
Expand All @@ -148,7 +164,7 @@ Make and push a new tag (with the base of the repository as the current working
git tag ${NEWTAG}
git push origin tag ${NEWTAG}

Then install it in `/sdf/data/rubin/shared/scheduler/packages`::
Then install it in ``/sdf/data/rubin/shared/scheduler/packages``::

PACKAGEDIR="/sdf/data/rubin/shared/scheduler/packages"
TARGETDIR="${PACKAGEDIR}/${MODULENAME}-${NEWVERSION}"
Expand Down Expand Up @@ -182,10 +198,10 @@ The general pattern followed by these instructions is:
#. Call `nbconvert` with a command that looks similar to this::

jupyter nbconvert \
--to html \
--execute \
--no-input \
--ExecutePreprocessor.kernel_name=python3 \
--ExecutePreprocessor.startup_timeout=3600 \
--ExecutePreprocessor.timeout=3600 \
whatever_notebook.ipynb
--to html \
--execute \
--no-input \
--ExecutePreprocessor.kernel_name=python3 \
--ExecutePreprocessor.startup_timeout=3600 \
--ExecutePreprocessor.timeout=3600 \
whatever_notebook.ipynb
92 changes: 0 additions & 92 deletions docs/usage.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,95 +30,3 @@ To start the dashbaord in LFA mode::
$ scheduler_dashboard --lfa

In each case, the app will then give you the URL at which you can find the app.

Running ``prenight``
--------------------

Activate the conda environment and start the app:

::

$ conda activate schedview
$ prenight

The app will then give you the URL at which you can find the app.

By default, the app will allow the user to select ``opsim`` database, pickles of
scheduler instances, and rewards data from ``/sdf/group/rubin/web_data/sim-data/schedview``
(if it is being run at the USDF) or the samples directory (elsewhere).
The data directory from which a user can select files can be set on startup:

::

$ prenight --data_dir /path/to/data/files

Alternately, ``prenight`` can be set to look at an archive of simulation
output in an S3 bucket:

::

$ export S3_ENDPOINT_URL='https://s3dfrgw.slac.stanford.edu/'
$ export AWS_PROFILE=prenight_aws_profile
$ prenight --resource_uri='s3://rubin-scheduler-prenight/opsim/' --data_from_archive

where ``prenight_aws_profile`` should be replaced by whatever section of
the ``~/.lsst/aws-credentials.ini`` file has the credentials needed for
access to the ``rubin-scheduler-prenight`` bucket.

The ``resources-uri`` can also be set to a local directory tree with the same
layout as the above S3 bucket, in which case filesystem access is needed to
that directory tree, but the environment variables above are not. For example:

::

$ prenight --resource-uri='file:///where/my/data/is/' --data_from_archive

Note that the trailing ``/`` in the ``resource-uri`` value is required.

Finally, the user can be allowed to enter arbitrary URLs for these files.
(Note that this is not secure, because it will allow the user to upload
malicious pickles. So, it should only be done when public access to the
dashboard is not possible.) Such a dashboard can be started thus:

::

$ prenight --data_from_urls

You can also supply an initial set of data files to show on startup:

::

$ conda activate schedview
$ prenight --night 2023-10-01 \
> --opsim_db /sdf/data/rubin/user/neilsen/devel/schedview/schedview/data/sample_opsim.db \
> --scheduler /sdf/data/rubin/user/neilsen/devel/schedview/schedview/data/sample_scheduler.pickle.xz \
> --rewards /sdf/data/rubin/user/neilsen/devel/schedview/schedview/data/sample_rewards.h5 \
> --port 8080

The (optional) rewards data, used in the "Rewards plot" tab, can be generated
by adding an extra option to ``sim_runner`` when running the simulation that
creates the opsim database being examined.
For example, to return the data when running ``sim_runner``:

::

>>> from rubin_sim.scheduler import sim_runner
>>> observatory, scheduler, observations, reward_df, obs_rewards = sim_runner(
... observatory,
... scheduler,
... sim_start_mjd=mjd_start,
... sim_duration=night_duration,
... record_rewards=True,
... )

The returned ``reward_df`` and ``obs_rewards`` data can then be saved to an `h5`
file that can then be loaded by ``prenight``:

::

>>> rewards_fname = "my_rewards.h5"
>>> reward_df.to_hdf(rewards_fname, "reward_df")
>>> obs_rewards.to_hdf(rewards_fname, "obs_rewards")

To be valid, the rewards data *must* be generated by the same execution of
``sim_runner`` that generates the opsim database being examined.
Loading