Skip to content

Update armer.rst#11

Open
JiayueLiuBMI wants to merge 9 commits into
developfrom
JiayueLiuBMI-patch-1
Open

Update armer.rst#11
JiayueLiuBMI wants to merge 9 commits into
developfrom
JiayueLiuBMI-patch-1

Conversation

@JiayueLiuBMI
Copy link
Copy Markdown

No description provided.

@shaoqx
Copy link
Copy Markdown
Contributor

shaoqx commented May 28, 2024

Please rename your branch following the format "name"/"task" (see branch from others for examples)

@shaoqx
Copy link
Copy Markdown
Contributor

shaoqx commented May 28, 2024

  1. The idea of ARMer
  • briefly talks about what is ARMer
  • cite the paper
  1. The 3 main functions of ARMer
  • config job (i.e.: submission script, etc.), submit, monitor (i.e.: wait_to_end) --- we will not go into details but just explain the concepts
  • How ARMer is used by APIs in EnzyHTP to wrap commands into jobs and submit to HPCs
  1. The cluster job config dictionary (This is the VIP of this page)
  • What is the cluster job config dictionary (mention it is the only thing user needs to know about ARMer)
  • The format of the cluster job config dictionary
  • The default values; the preset values
  1. Examples
  • Give examples about how the cluster job config dictionary is used in single_point on ACCRE
  1. How to make mods to support your own HPC
  • just link it to the how-to page

Copy link
Copy Markdown
Contributor

@shaoqx shaoqx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job reformatting the page! Please address the comments and adapt the new outline.

Comment thread docs/source/sci_api_tutorial/armer.rst Outdated
Comment thread docs/source/sci_api_tutorial/armer.rst Outdated
Comment thread docs/source/sci_api_tutorial/armer.rst Outdated
Comment thread docs/source/sci_api_tutorial/armer.rst Outdated
Comment thread docs/source/sci_api_tutorial/armer.rst Outdated
Comment thread docs/source/sci_api_tutorial/armer.rst Outdated
Comment thread docs/source/sci_api_tutorial/armer.rst
Comment thread docs/source/sci_api_tutorial/armer.rst Outdated
Comment thread docs/source/sci_api_tutorial/armer.rst Outdated
Comment thread docs/source/sci_api_tutorial/armer.rst Outdated
* you can set the self value during config_job to make each job different


Example Code
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use examples from https://enzyhtp-doc.readthedocs.io/en/latest/sci_api_tutorial/single_point.html
But here we focus on the cluster_job_config

@JiayueLiuBMI JiayueLiuBMI deleted the JiayueLiuBMI-patch-1 branch June 11, 2024 09:12
@shaoqx shaoqx reopened this Jun 12, 2024
Comment thread docs/source/sci_api_tutorial/armer.rst Outdated
Comment thread docs/source/sci_api_tutorial/armer.rst Outdated
Comment on lines +136 to +137
For most users, only the `Job configuration` requires specific input from different users.
The first step involves configuring a job using the config_job method. This method prepares the submission script by specifying the commands to be executed, environmental settings, and resource requirements.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change this Bold and change it to say

none of the function will be directly access by user in normal EnzyHTP workflows. They are used by developers in APIs of EnzyHTP. However, all of the APIs that uses ARMer will expose an input that allow users to config the resource ...(elaborate)...

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And in the next section, you can smoothly introduce the cluster_job_config dict

Comment thread docs/source/sci_api_tutorial/armer.rst Outdated
For most users, only the `Job configuration` requires specific input from different users.
The first step involves configuring a job using the config_job method. This method prepares the submission script by specifying the commands to be executed, environmental settings, and resource requirements.

.. admonition:: Here is the `cluster_job_config` dictionary
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make this a seperate section in parallel with the 3 main functions

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's introduce in full the format of the dictionary and the meaning of each keys

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are missing the "cluster" key here in parallel with the "res_keywords" key

Comment thread docs/source/sci_api_tutorial/armer.rst Outdated
Comment thread docs/source/sci_api_tutorial/armer.rst Outdated
Comment thread docs/source/sci_api_tutorial/armer.rst
Copy link
Copy Markdown
Contributor

@shaoqx shaoqx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job addressing the comment!
There are stilll some organization issue to address. Please read carefully the outline I sent and address the comments

Thanks!

Comment thread docs/source/sci_api_tutorial/armer.rst
Comment thread docs/source/sci_api_tutorial/armer.rst Outdated
Comment on lines +187 to +202
These parameters are specified under the argument ``res_keywords``.

``core_type``: This specifies that the job should be run on GPU/CPU cores.

``nodes``: How many nodes needed to request for the job.

``node_cores``: How many cores needed for each node. If GPU is used, usually only one core per node will be requested.

``job_name``: This sets the name of the job to "job_name". You can change this to a more descriptive name for your job.

``partition``: This specifies that the job should be submitted to a specific partition, which is likely a partition dedicated to GPU resources.

``mem_per_core``: This requests a number of gigabytes of memory per core.

``walltime``: This sets the maximum walltime (execution time) for the job. '24:00:00' means 24 hours.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rearrange this section to align with the hierarchy of the actual dictionary.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also be aware of the confusion Yinjie made. emphasize these keywords are build-in keywords in EnzyHTP. it is not the same as the keywords of SLURM or other job scheduling system

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By the "hierarchy of the actual dictionary" I mean like this:

There are 2 key-value pairs accepted in this dictionary, ``cluster`` and ``res_keywords``.
``cluster``
         The value under this key defines ... as a ClusterInterface() object
         Example: ...
``res_keywords``
         The value under this key defines ... as a dictionary
         <more to say about what k-v pairs of this dictionary>
         Example: ...

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also we need to explain the "default" behavior of the dictionary under res_keywords when a key is not specified.

Comment thread docs/source/sci_api_tutorial/armer.rst
Comment thread docs/source/sci_api_tutorial/armer.rst Outdated
Copy link
Copy Markdown
Contributor

@shaoqx shaoqx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job addressing all the comments! Please see these new ones!

Comment thread docs/source/sci_api_tutorial/armer.rst Outdated

* you can set the self value during config_job to make each job different

**Specifically, only the ``Job configuration`` requires specific input from different users.**
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need exactly the same name "cluster_job_config" so that user can link it to the the next section

Comment thread docs/source/sci_api_tutorial/armer.rst Outdated

**These keywords are build-in keywords in EnzyHTP, which are not the same as the keywords of SLURM or other job scheduling system.**

``res_keywords``:
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change this ":" to a English ":" lol




.. admonition:: Here is the `cluster_job_config` dictionary
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See an unresolved comment above

Copy link
Copy Markdown
Contributor

@shaoqx shaoqx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job! We can now focus on the core component about the cluster_job_config

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants