From 81d955ded87907059f064a2e04b05094a52b0b0a Mon Sep 17 00:00:00 2001 From: Eric Neilsen Date: Tue, 16 Dec 2025 13:24:59 -0800 Subject: [PATCH 1/4] Describe cron gates --- docs/reports.rst | 36 ++++++++++++++++++++++++++---------- 1 file changed, 26 insertions(+), 10 deletions(-) diff --git a/docs/reports.rst b/docs/reports.rst index b45440f1..35a5d8aa 100644 --- a/docs/reports.rst +++ b/docs/reports.rst @@ -26,7 +26,7 @@ These scripts are found in the `batch` directory of the `schedview_notebooks` re Each of these scripts builds an environment in which to convert the notebook, executes the conversion, and updates a corresponding index it include the new reports. There are currently two indexes of reports. One index lists publicly accessible, which are served from `https://s3df.slac.stanford.edu/data/rubin/sim-data/schedview/reports/ `__. -Currently, the public reports are limited to a brief nigth summary. +Currently, the public reports are limited to a brief night summary. Additional reports, currently requiring logging to the the USDF, can be found at `https://usdf-rsp-int.slac.stanford.edu/schedview-static-pages/ `__. @@ -38,6 +38,19 @@ The `cron` job runs the reports with the following entries:: 15 5 * * * /opt/slurm/slurm-curr/bin/sbatch /sdf/data/rubin/shared/scheduler/packages/schedview_notebooks/batch/scheduler_nightsum.sh 2>&1 >> /sdf/data/rubin/shared/scheduler/schedview/scheduler_nightsum/scheduler_nightsum.out 30 7 * * * /opt/slurm/slurm-curr/bin/sbatch /sdf/data/rubin/shared/scheduler/packages/schedview_notebooks/batch/prenight.sh 2>&1 >> /sdf/data/rubin/shared/scheduler/schedview/prenight/prenight.out +If necessary, these cron jobs can be stopped from doing anything, even a user that does not own the cron job, if they have write access to ``/sdf/data/rubin/shared/scheduler/cron_gates/${SCRIPT}``. +This is accomplished using gate files: early in each script, the script checks for the existence of a file with name ``/sdf/data/rubin/shared/scheduler/cron_gates/${SCRIPT_NAME}/${USER}`` and aborts if it does not exist. +Any user with write access to ``/sdf/data/rubin/shared/scheduler/cron_gates/${SCRIPT_NAME}`` can create or remove files in that directory, so a user can cause these scripts to immediately abort +when started by a cron job owned by a different user by removing ``/sdf/data/rubin/shared/scheduler/cron_gates/${SCRIPT_NAME}/${CRON_JOB_USER}``. + +So, to stop the `scheduler_nightsum.sh`` cron job owned by user ``neilsen``, remove the file ``/sdf/data/rubin/shared/scheduler/cron_gates/scheduler_nightsum/neilsen``, +and to stop the ``prenight.sh`` cron job owned by user ``neilsen``, remove the file ``/sdf/data/rubin/shared/scheduler/cron_gates/prenight/neilsen``. + +The logs of the cron jobs (and any other executions of these scripts submitted using ``sbatch``) can be found in: +* ``/sdf/data/rubin/shared/scheduler/schedview/sbatch/scheduler_nightsum_%A_%a.out`` and +* ``/sdf/data/rubin/shared/scheduler/schedview/sbatch/prenight_%A_%a.out`` +where ``%A`` is the slurm "Job array's master job allocation number" and ``%a`` is the slum "Job array ID (index) number". + The environment --------------- @@ -81,7 +94,7 @@ The batch script that generates the prenight briefing reeport requeires a versio pip install git+https://github.com/lsst/schedview.git@v0.19.0.dev1 -Finally, update the bash scripts that need it (batch/prenight.sh and batch/scheduler_nightsum.sh), +Finally, update the bash scripts that need it (``batch/prenight.sh`` and ``batch/scheduler_nightsum.sh``), for example:: source /sdf/group/rubin/sw/w_latest/loadLSST.sh @@ -128,6 +141,9 @@ Replace the symlink to point to your new one:: Updating other software used by the jobs ---------------------------------------- +The jupyter notebooks used to generate the reports import ``schedview`` and related packages from subtirectories of ``/sdf/data/rubin/shared/scheduler/packages``. +New version should be added there to make them available. + Begin by determining the next available tag. Get sorted existing tags with:: @@ -148,7 +164,7 @@ Make and push a new tag (with the base of the repository as the current working git tag ${NEWTAG} git push origin tag ${NEWTAG} -Then install it in `/sdf/data/rubin/shared/scheduler/packages`:: +Then install it in ``/sdf/data/rubin/shared/scheduler/packages``:: PACKAGEDIR="/sdf/data/rubin/shared/scheduler/packages" TARGETDIR="${PACKAGEDIR}/${MODULENAME}-${NEWVERSION}" @@ -182,10 +198,10 @@ The general pattern followed by these instructions is: #. Call `nbconvert` with a command that looks similar to this:: jupyter nbconvert \ - --to html \ - --execute \ - --no-input \ - --ExecutePreprocessor.kernel_name=python3 \ - --ExecutePreprocessor.startup_timeout=3600 \ - --ExecutePreprocessor.timeout=3600 \ - whatever_notebook.ipynb + --to html \ + --execute \ + --no-input \ + --ExecutePreprocessor.kernel_name=python3 \ + --ExecutePreprocessor.startup_timeout=3600 \ + --ExecutePreprocessor.timeout=3600 \ + whatever_notebook.ipynb From 9ae556dd37caa26b40de674aa57ec0ccff1fec61 Mon Sep 17 00:00:00 2001 From: Eric Neilsen Date: Tue, 16 Dec 2025 13:31:32 -0800 Subject: [PATCH 2/4] Remove instructions for prenight dashboard that does not exist anymore --- docs/usage.rst | 92 -------------------------------------------------- 1 file changed, 92 deletions(-) diff --git a/docs/usage.rst b/docs/usage.rst index 5199877f..6add63f4 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -30,95 +30,3 @@ To start the dashbaord in LFA mode:: $ scheduler_dashboard --lfa In each case, the app will then give you the URL at which you can find the app. - -Running ``prenight`` --------------------- - -Activate the conda environment and start the app: - -:: - - $ conda activate schedview - $ prenight - -The app will then give you the URL at which you can find the app. - -By default, the app will allow the user to select ``opsim`` database, pickles of -scheduler instances, and rewards data from ``/sdf/group/rubin/web_data/sim-data/schedview`` -(if it is being run at the USDF) or the samples directory (elsewhere). -The data directory from which a user can select files can be set on startup: - -:: - - $ prenight --data_dir /path/to/data/files - -Alternately, ``prenight`` can be set to look at an archive of simulation -output in an S3 bucket: - -:: - - $ export S3_ENDPOINT_URL='https://s3dfrgw.slac.stanford.edu/' - $ export AWS_PROFILE=prenight_aws_profile - $ prenight --resource_uri='s3://rubin-scheduler-prenight/opsim/' --data_from_archive - -where ``prenight_aws_profile`` should be replaced by whatever section of -the ``~/.lsst/aws-credentials.ini`` file has the credentials needed for -access to the ``rubin-scheduler-prenight`` bucket. - -The ``resources-uri`` can also be set to a local directory tree with the same -layout as the above S3 bucket, in which case filesystem access is needed to -that directory tree, but the environment variables above are not. For example: - -:: - - $ prenight --resource-uri='file:///where/my/data/is/' --data_from_archive - -Note that the trailing ``/`` in the ``resource-uri`` value is required. - -Finally, the user can be allowed to enter arbitrary URLs for these files. -(Note that this is not secure, because it will allow the user to upload -malicious pickles. So, it should only be done when public access to the -dashboard is not possible.) Such a dashboard can be started thus: - -:: - - $ prenight --data_from_urls - -You can also supply an initial set of data files to show on startup: - -:: - - $ conda activate schedview - $ prenight --night 2023-10-01 \ - > --opsim_db /sdf/data/rubin/user/neilsen/devel/schedview/schedview/data/sample_opsim.db \ - > --scheduler /sdf/data/rubin/user/neilsen/devel/schedview/schedview/data/sample_scheduler.pickle.xz \ - > --rewards /sdf/data/rubin/user/neilsen/devel/schedview/schedview/data/sample_rewards.h5 \ - > --port 8080 - -The (optional) rewards data, used in the "Rewards plot" tab, can be generated -by adding an extra option to ``sim_runner`` when running the simulation that -creates the opsim database being examined. -For example, to return the data when running ``sim_runner``: - -:: - - >>> from rubin_sim.scheduler import sim_runner - >>> observatory, scheduler, observations, reward_df, obs_rewards = sim_runner( - ... observatory, - ... scheduler, - ... sim_start_mjd=mjd_start, - ... sim_duration=night_duration, - ... record_rewards=True, - ... ) - -The returned ``reward_df`` and ``obs_rewards`` data can then be saved to an `h5` -file that can then be loaded by ``prenight``: - -:: - - >>> rewards_fname = "my_rewards.h5" - >>> reward_df.to_hdf(rewards_fname, "reward_df") - >>> obs_rewards.to_hdf(rewards_fname, "obs_rewards") - -To be valid, the rewards data *must* be generated by the same execution of -``sim_runner`` that generates the opsim database being examined. From 0e04e304d5dd470cd40eac12b9ad4342d0da83a9 Mon Sep 17 00:00:00 2001 From: Eric Neilsen Date: Tue, 16 Dec 2025 13:38:40 -0800 Subject: [PATCH 3/4] Add link to prenight sim generation page in rubin_sim --- docs/prenight_sim_generation.rst | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/docs/prenight_sim_generation.rst b/docs/prenight_sim_generation.rst index 45a66159..18741e74 100644 --- a/docs/prenight_sim_generation.rst +++ b/docs/prenight_sim_generation.rst @@ -1,9 +1,5 @@ Pre-night simulation generation =============================== -Daily prenight briefing simulations are currently kicked off by a cron job on ``sdfcron001.sdf.slac.stanford.edu`` as user ``neilsen``. - -This runs the script at ``/sdf/data/rubin/shared/scheduler/packages/rubin_sim/batch/run_prenight_sims.sh`` as a slurm job at 8:15am Pacific time every morning, and takes about 10 minutes to run. -The ``rubin_sim`` project holds this script in ``batch/run_prenight_sims.sh``. - -Additional documentation can be found in the `docs` for `rubin_sim`. +Documentation for pre-night simulation generation can be found in the +`rubin_sim documentation `__ . From 880f017f23aaa266010401af36336e3d955fbdaa Mon Sep 17 00:00:00 2001 From: Eric Neilsen Date: Tue, 16 Dec 2025 13:41:09 -0800 Subject: [PATCH 4/4] Fix typos --- docs/reports.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/reports.rst b/docs/reports.rst index 35a5d8aa..655a1ce6 100644 --- a/docs/reports.rst +++ b/docs/reports.rst @@ -104,10 +104,10 @@ Even though we aren't using the environment provided by the source of `loadLSST. it's still needed to get `conda` into our path. -Updating version of `schedview_notebooks` used by the `cron` job ----------------------------------------------------------------- +Updating the version of `schedview_notebooks` used by the `cron` job +-------------------------------------------------------------------- -The scripts submitted by the cron job supplied above use the version of schedview in `/sdf/data/rubin/shared/scheduler/packages/schedview_notebooks`, which is itsef a link to a directory for a specific version, e.g. `/sdf/data/rubin/shared/scheduler/packages/schedview_notebooks-v0.1.0`. +The scripts submitted by the cron job supplied above use the version of schedview in ``/sdf/data/rubin/shared/scheduler/packages/schedview_notebooks``, which is itsef a link to a directory for a specific version, e.g. ``/sdf/data/rubin/shared/scheduler/packages/schedview_notebooks-v0.1.0``. To tag and install a new version to be used, start by deciding on a tag. Get sorted existing tags with::