Use a subcommand-based command-line interface

[vkarak]

Currently, ReFrame has lots of options that makes it hard to use. We should move an action-based or verb-based CLI, where each subcommand will have its own options and help. Here are some examples:

• Listing tests

reframe list --detailed -n Foo -E 'num_nodes==16'

• Running tests

reframe run -n Foo -E 'num_nodes==16'

• Querying past results

reframe analyze --list-testcases=now-10d:now/mean:/

Etc.

[vkarak]

This is a table I compiled that maps the existing options to the existing actions (future subcommands):

Options and subcommands

Option	Subcommands	Comments
--compress-report	run
--dont-restage	run
--keep-stage-files	run
--output	list, run	Rename to --outputdir
--perflogdir	list, run
--prefix	ci-generate, list, run
--report-file	run
--report-junit	run
--stage	list, run	Rename to --stagedir
--save-log-files	run
--timestamp	list, run
--checkpath	ci-generate, list, run
--recursive	ci-generate, list, run
--cpu-only	ci-generate, list, run	Consider dropping this option
--failed	ci-generate, list, run
--gpu-only	ci-generate, list, run	Consider dropping this option
--maintainer	ci-generate, list, run
--name	analyze, ci-generate, list, run
--prgenv	ci-generate, list, run	Consider renaming to --environ
--exclude-tag	ci-generate, list, run
--tag	ci-generate, list, run
--exclude	ci-generate, list, run	Maybe use this option also for analyze
--filter-expr	ci-generate, analyze, list, run
--ci-generate	None	Convert to subcommand ci-generate
--delete-stored-sessions	analyze	Convert to --show-sessions
--describe-stored-sessions	analyze	Convert to --show-sessions
--describe-stored-testcases	analyze	Convert to --show-testcases
--detect-host-topology	None	Convert to subcommand detect-host-topology
--dry-run	run
--list-detailed	list	Convert to --detailed
--list-stored-sessions	analyze	Convert to --list-sessions
--list	None	Convert to subcommand list
--list-tags	list	Convert to --tags-only
--performance-compare	analyze	Maybe convert to --compare-performance
--run	None	Convert to subcomand run
--show-config	None	Convert to subcommand config --show
--version	None	Remains global option
--disable-hook	ci-generate, list, run	NOTE: post-init hooks can affect listing
--duration	run
--exec-order	list, run	NOTE: exec order is also preserved in listing
--exec-policy	run
--flex-alloc-nodes	run
--flex-alloc-strict	run
--job-option	run
--max-retries	run
--maxfail	run
--mode	ci-generate, list, run
--reruns	run
--restore-session	list, run	Need to verify
--retries-threshold	run
--setvar	ci-generate, list, run
--skip-performance-check	run
--skip-prgenv-check	list, run	Convert to --skip-environ-check
--skip-sanity-check	run
--skip-system-check	list, run
--map-module	run	Need to verify
--module-mappings	run	Need to verify
--module-path	run	Need to verify
--non-default-craype	run	Need to verify; maybe drop the option
--purge-env	run	Need to verify
--unload-module	run	Need to verify
--distribute	ci-generate, list, run
--parameterize	ci-generate, list, run
--param-values-delim	list, run
--repeat	list, run	Rename to --clone
--performance-report	run
--session-extras	run
--failure-stats	run
--config-file	None	Global option
--nocolor	None	Global option
--table-format	analyze, run	Needed in run due to --performance-report
--table-format-delim	analyze, run
--verbose	None	Global option
--quiet	None	Global option

[vkarak]

For ci-generate we should make sure that the options included do have an effect.