Dryrun mode

Difficulty level: easy
Time need to lean: 10 minutes or less
Key points:
- Option -n turns workflow into dryrun mode
- sos dryrun is equivalent to sos run -n

`dryrun` mode

The dryrun mode is used to check for errors of SoS workflows and running environments without actually executing any of the actions and tasks. It can be specified with option -n to command run (sos run -n, sos-runner -n, or %run -n) or with subcommand dryrun (sos dryrun).

Checking syntax error

The first use of dryrun mode is to check syntax error of scripts. For example, the following script yields an error because the header of the step is wrong (should use : instead of ,).

ERROR: File contains parsing errors: /Users/bpeng1/sos/sos-docs/src/user_guide/.tmp_script_3ijm49e9.sos
	[line  2]: [10, skip=False]
sh:
   echo "I am command echo"

Invalid statements: SyntaxError('invalid syntax', ('<string>', 1, 10, '[10, skip=False]\n'))

RuntimeError: Workflow exited with code 1

Print instead of execute scripts

Script-executing actions such as run, sh, and python does not execute the scripts in dryrun mode. In stead, they will print the script with the command to execute them. This usage allows you to check if the scripts are interpolated correctly, or even executing them outside of SoS.

For example, the following script will print the expanded version of the scripts to be executed:

HINT: /bin/sh SCRIPT
sleep 100
echo "Generating test_1.txt"
touch test_1.txt


HINT: /bin/sh SCRIPT
sleep 200
echo "Generating test_2.txt"
touch test_2.txt

It is worth noting, however, that sos executes Python statements in dryrun mode as usual so your workflow will appear to be executed normally if you do not use actions.

For example, if you implement your steps in Python and include them directly, the step will be executed

Generating {_output}
Generating {_output}

and the output files will be generated:

Generation of placeholder files

Another consequence of running workflows in dryrun mode is that SoS generates empty placeholder files during dryrun and remove them afterwards. This allows the workflows to be executed "normally" because the execution of the next steps might depend on the existence of the output files.

For example, when running in dryrun mode, the following script will be executed "normally". The _input files (which are the output of step 10) could be opened at step 20, but the content would be empty:

HINT: /bin/sh SCRIPT
echo "Generating test_1.txt" > test_1.txt


HINT: /bin/sh SCRIPT
echo "Generating test_2.txt" > test_2.txt


Content of test_1.txt:

HINT: /bin/sh SCRIPT
cp test_1.txt test_1.txt.bak


Content of test_2.txt:

HINT: /bin/sh SCRIPT
cp test_2.txt test_2.txt.bak

test_1.txt and test_2.txt would be generated normally in run mode:

External tasks

External tasks are executed in dryrun mode as follows:

Only task from the first substep will be generated and submitted.
The task will be submitted to remote hosts as usual, with input files synchronized to remote host if needed. This allows you to test if the remote hosts are accessible.
The task will be executed in dryrun mode, and will print instead of execute the scripts defined in actions.

HINT: /bin/sh SCRIPT
echo "Generating test_0.txt" > test_0.txt


HINT: /bin/sh SCRIPT
echo "Generating test_1.txt" > test_1.txt


HINT: /bin/sh SCRIPT
echo "Generating test_2.txt" > test_2.txt


HINT: /bin/sh SCRIPT
echo "Generating test_3.txt" > test_3.txt


HINT: /bin/sh SCRIPT
echo "Generating test_4.txt" > test_4.txt

Docker actions

When an action is executed in docker in dryrun mode, SoS would download the docker image if needed but will not execute the script. Instead, SoS will print the interpolated script and the docker command needs to execute the script.

For example, running an docker_build action in dryrun mode yields the following output:

HINT: docker build -f /Users/bpeng1/sos/sos-docs/src/user_guide/tmpudz3l0mu/Dockerfile . --tag test/docker_build1 --label "label with space" --compress --memory 2G
#
# Super simple example of a Dockerfile
#
FROM ubuntu:latest
MAINTAINER Andrew Odewahn "odewahn@oreilly.com"

WORKDIR /home

and running a script in docker in dryrun mode:

HINT: Pulling docker image ubuntu
HINT: docker run --rm   -v /Users/bpeng1/sos/sos-docs/src/user_guide:/Users/bpeng1/sos/sos-docs/src/user_guide -v /Users/bpeng1/sos/sos-docs/src/user_guide/tmpernk6bj_/docker_run_37468.sh:/var/lib/sos/docker_run_37468.sh    -t  -w=/Users/bpeng1/sos/sos-docs/src/user_guide -u 1985961928:895809667    ubuntu /bin/bash -ev /var/lib/sos/docker_run_37468.sh
echo 'Echo'