Script without Slurm

[maruf-anu]

I am trying to reproduce the results with the trained huggingface model.
But, all scripts/configuration files are coded with Slurm.
Can you provide some scripts that can run on a single GPUs without Slurm?

[iSach]

Hello @maruf-anu,

You can run every script on a single GPU by setting the backend to async in the hardware section of the associated configuration YAML file of the script.

Pay attention that some scripts take advantage of Slurm for parallelization, and schedule multiple jobs. These scripts might try to summon all the tasks at the same time with async scheduling. For example, for inference such as reanalysis, I recommend you use only one date and one sample per date. One way to fix that would be put pools=1 in the schedule(...) calls of the concerned scripts.

Feel free to tell me what scripts you are trying to use and if you're still having issues.

Cheers,
Sacha.

[maruf-anu]

Hi Sacha,

Thanks very much for your reply.
I am interested in inference with the ERA5 data.

So, in the download script, I changed it to async like this

    schedule(
        *jobs,
        name="ERA5",
        export="ALL",
        #backend=hardware.backend,
        #account=hardware.account,
        backend='async',
    )

However, I am getting the following error

Traceback (most recent call last):
  File "/download_era5.py", line 80, in <module>
    **hardware.job,
      ^^^^^^^^^^^^
...

omegaconf.errors.ConfigAttributeError: Key 'job' is not in struct
    full_key: hardware.job
    object_type=dict      

We run jobs using PBS Pro, and many others do as well.
So, it would benefit if you make those scripts scheduler independent.

Best,
Maruf

[iSach]

Hi Sacha,

Thanks very much for your reply. I am interested in inference with the ERA5 data.

So, in the download script, I changed it to async like this

schedule(
    *jobs,
    name="ERA5",
    export="ALL",
    #backend=hardware.backend,
    #account=hardware.account,
    backend='async',
)

However, I am getting the following error

Traceback (most recent call last):
File "/download_era5.py", line 80, in
**hardware.job,
^^^^^^^^^^^^
...

omegaconf.errors.ConfigAttributeError: Key 'job' is not in struct
full_key: hardware.job
object_type=dict
We run jobs using PBS Pro, and many others do as well. So, it would benefit if you make those scripts scheduler independent.

Best, Maruf

Did you delete the job section in the hardware section of the config?

Please try like this:

    schedule(
        *jobs,
        name="ERA5",
        export="ALL",
        backend='async',
        pools=1,
    )

hardware:
  backend: async
  account: users
  job:
    cpus: 8
    ram: 16GB
    time: "07-00:00:00"
    partition: shared

Leave the job as above, it will be ignored but it needs to be there. Technically, you could simply edit the backend in the YAML to async and it should work. Just avoid using split_vars=True then. The only additional thing to change is actually pools=1 if you use split_vars so it does things separately.

Best,
Sacha.

[iSach]

We run jobs using PBS Pro, and many others do as well. So, it would benefit if you make those scripts scheduler independent.

Also, please note that the scheduler is independent, and managed by Dawgz. One could try to add support for PBS (Pro) in that package.