Task management

Difficulty level: easy
Time need to lean: 10 minutes or less
Key points:
- sos status lists status of tasks
- sos kill terminates running tasks
- sos queue remove tasks from the system
- sos execute executes tasks on a specified host

`sos status`

Specification of tasks

Command

sos status [tasks] -q queue

checkes the status of tasks. You can specify any number of first characeters of a task to specify a task, for example,

sos task 7
sos task 77e
sos task 7736e

would all work for a task with ID 77e36e7404cf6c2ef7079a09e84a4d6d, but multiple tasks could be identifies if they share the same leading digits. Actually,

sos task

would match all tasks and list the status of all local tasks.

Option -q specifies the task queue to monitor. For example,

sos status -q docker

would check the status of all tasks on a remote host docker. You can monitor the tasks on docker on any machine with defined host docker.

Verbosity of output

Option -v controls the details of the output of command sos status. For example,

sos status e7404cf6c2 -v0

would print just the status of the task (e.g. running).

sos task 77e -v1

would print the task id and their status

77e36e7404cf6c2ef7079a09e84a4d6d    running
77e3c2ef7079a236e7404cf6c2f343d3    completed

Option -v0 and -v1 could check the status of multiple tasks, as realized by SoS. Some tasks queues have their own task status command and option -v2 (and upper) will use these commands (if specified) to check the status of the jobs. That is to say

sos task 77e36 -v2

If you would like to know more about the tasks,

sos task 77e36 -v3

would list the script the task is running and all variables in abbreviated format, and

sos task 77e36 -v4

would list details of tasks.

`sos kill`

Command

sos kill [tasks] [-q queue]

kills specified or all tasks on specified job queue queue. Because the same job could be executed on different queues (you have have done so), you will have to specify the correct queue name to kill the job on different queues.

`sos execute`

Command

sos execute [tasks] [-q queue]

is the command that is used internally by sos run to execute tasks but you could use this command to execute tasks externally. For example, if a task failed on a server, you could use command

sos execute task_id -q server

to execute the command on another server. Note that task_id specifies a local task with local paths. The task will be converted to a remote task (with path names converted for that host) if server specifies a remote host. This makes it easy for you to re-submit tasks to the same server after changing server configuration, or submit the same task to a different server.

`sos purge`

Command sos purge is equivalent to magic %task purge from SoS Notebook.

usage: %task purge [-h] [-q QUEUE] [-c CONFIG] [-v {0,1,2,3,4}]
                   [tasks [tasks ...]]

positional arguments:
  tasks                 ID of the tasks to be removed. There is no need to
                        specify compelete task IDs because SoS will match
                        specified name with tasks starting with these names.
                        If no task ID is specified, all tasks related to
                        specified workflows (option -w) will be removed.

optional arguments:
  -h, --help            show this help message and exit
  -q QUEUE, --queue QUEUE
                        Remove tasks on specified tasks queue or remote host
                        if the tasks . The queue can be defined in global or
                        local sos configuration file, or a file specified by
                        option --config. A host is assumed to be a remote
                        machine with process type if no configuration is
                        found.
  -c CONFIG, --config CONFIG
                        A configuration file with host definitions, in case
                        the definitions are not defined in global sos
                        config.yml files.
  -v {0,1,2,3,4}        Output error (0), warning (1), info (2), and debug (3)
                        information to standard output (default to 2).

Command sos purge removes specified tasks, tasks matching specified criteria, or all tasks from local or remote host (option -q).

For example, command

sos purge --all -q cluster

removes all tasks from a remote cluster,

sos purge --all --age 2d

removes all tasks that are created more than 2 days ago, and

sos purge -all -q cluster -s completed

removes all completed tasks from the remote cluster.

You can also specify the IDs of tasks to be removed, e.g.

sos purge 38ef

removes all tasks with ID starting with 38ef.

Finally, if you would like to remove all tasks related information, including tasks that are not generated by workflows from the current project (directory), you can use command

sos purge --all

to clear all information.