- Difficulty level: intermediate
- Time need to lean: 30 minutes or less
- Key points:
- Using
container='library:xxx'etc will enableengine='singularity' engine='singularity'is explicitly needed for docker images.
- Using
Before you run any script using singularity, please check if singularity is installed by checking the availability of command singularity. Also, it would be helpful for you to read the sigularity user's guide or some online tutorial to understand how singularity works before you try to use singularity in SoS.
Although running singularity images does not need root privilege, building singularity images often requires sudo access. SoS provides an option sudo=True to the singularity_build action (an equivalence to command singularity build) to execute the command in sudo mode, but it does not accept interactive input of password. So before running any singularity_build action with option sudo=True, please run sudo -i to enter sudo mode, or add your username as a sudo user without password (google "sudo without password" for instructions). The latter is not safe but can be very convenient for you to prepare singularity images on a personal linux workstation.
SoS executes scripts with a singularity container by calling command singularity exec with appropriate parameters. For example, if you specify a container with a library: schema, sos will first pull the image, save it as a local image, and use singularity exec to run it:
The actual singularity exec command executed by SoS can be shown when you execute the script in dryrun mode (with -n option). In this mode, SoS would print the interpolated script (if option expand=True is set) and the docker command to execute it:
As you can see, the docker command looks similar to
singularity exec alpine-latest.sif /bin/sh /path/to/a/temp/script
Basically, SoS pulls the image and runs command singularity exec to execte the script with /bin/sh with the singularity image.
You can also use a docker image with singularity. However, because docker:// images are by default executed by docker, you will need to specify the use of singularity using parameter engine='singularity':
In summary, as listed here, you can invoke singularity with the following combinations of parameters container and engine:
container |
engine |
execute by | example | comment |
|---|---|---|---|---|
filename.simg |
|
singularity | container='ubuntu.simg' |
|
shub://tag |
|
singularity | container='shub://GodloveD/lolcow' |
Image will be pulled to a local image |
library://tag |
|
singularity | container='library://GodloveD/lolcow' |
Image will be pulled to a local image |
name |
singularity |
singularity | container='a_dir', engine='singularity' |
treat name as singularity image file or directory |
docker://tag |
singularity |
singularity | container='docker://godlovdc/lolcow', engine='singularity' |
|
file://filename |
|
singularity | container='file://ubuntu.simg' |
Simply put, singularity will be used by default with container shub://, file://, and filename.simg, but you will have to specify engine='singularity' if you would like to use a docker image or a directory.
Although singularity accepts docker://, shub://, and 'library://' container types, SoS always pull the image and build a local .sif file before executing it. If the container is used again, the local .sif file will be used directly.
For example, because an image has already been downloaded from the above example, SoS will not pull docker://ubuntu when using the container again
You can use the name of the image directly (e.g. container='ubuntu.sif') if you know the image has been downloaded:
Customize singularity image path
By default, downloaded singularity images are saved to the SoS Singularity Library located at ~/.sos/singularity/library. Environment variable SOS_SINGULARITY_LIBRARY can be used to set customized directory, e.g.,
export SOS_SINGULARITY_LIBRARY=/path/to/sif_folder
SoS will use *.sif files in $SOS_SINGULARITY_LIBRARY if they exist. As a result you can manually place existing image files to $SOS_SINGULARITY_LIBRARY to avoid pull images online attempts. This is useful when for example a workflow runs on a compute node without internet access.
A very useful feature about singularity is that you can use the container almost as a local command with access to local file system,
However, singularity only binds current working directory, /tmp, and your home directory. Other directories would be from within the image, or appear to be missing even if they exist on the host file system. For example, the following command lists /usr/local inside the image:
and /usr/local/var would appear to be missing
To allow singularity to see more directories, you can add one or more parameters to the bind parameter. For example, with bind='/usr/local', the singularity exec command lists directory /usr/local from the host filesystem:
and we can see /usr/local/var actually exists
Parameter bind accepts host_dir:img_dir pairs to mount host_dir from host as img_dir seen by the image
You can use parameter entrypoint to change the environment in which the interpreter will be executed. An entrypoint is usually the name of a shell script that defines a number of environment variables and run the command given by the command line parameters in such a way that the current process is replaced by it,
#!/bin/bash
set -o errexit
set -o pipefail
set -o nounset
export MY_ENV=MY_VALUE
exec "$@"
but it can also be other commands such as micromamba run -n ENVIRONMENT.
For example, the following statement will execute interpreter Rscript
It's equivalent to the following statement
If you need to pass other options to command singularity exec, you can specify them as additional keyword arguments for the run (or other) actions. For example,
Note that you need to use format OPT=True to specify flag options such as --nv, and need to use underscore to specify options with - in their names.
Action singularity_build calls command singularity build with appropriate command line options. The SoS equivalence of the example in the singularity user's guide
singularity build lolcow.sif library://sylabsed/examples/lolcow
is the singularity_build action with options src and dest:
You can also use action singularity_build to build an image from a singularity definition file. Using an example from the user's guide:
Options such as notest=True could be add to the action. Note that the content of the definition file is indented for clarify, but you can include the file as it is (no indentation).