Introductory Videos

Getting help

SoS Notebook comes with an extensive set of documentation but it can be daunting to dive into it to get the information you need.

• If you have a casual question like if certain feature exists or how certain things are handled, it is better to ask it in our gitter channel. You can also use the button at the top right corner of the homepage.

• If you have a more serious question or suggestion, please feel free to post to our issue tracker. SoS consists of tens of modules and related issue trackers but free feel to post to any of them because it is easy for us to move the issue to an appropriate tracker if needed.

Documentation

Our documentation is hosted on our live server and can be edited directly. If you spot an error or feel like contributing to the document, please

1. Click the     button to the top right corner of each tutorial
2. Execute the notebook and edit it on our live server
3. Enter !create-pr <Shift-Enter> in the console panel to send us a PR

The basics

• SoS workflow: a 20 minute overview
• Variables and parameters
• Step input, output, and substeps
• Basic SoS workflows
• Using SoS Workflow for daily data analysis

Syntax and data types

• Syntax and file formats (global and other sections)
• Script format of function calls
• The parameter statement
• Configuration files (option -c)
• SoS targets (file_target, sos_variable, R_Library, and more)
• The sos_targets data type
• SoS functions (get_output and expand_pattern)
• Extending SoS

SoS Steps

• The input statement
  • Named input (keyword and dictionary input)
  • Input option group_by
  • Input option for_each
  • Input options paired_with and group_with
  • Input option pattern and function expand_pattern
  • Input option concurrent
  • Dynamic input (dynamic target)
• The output statement
  • Output option group_by
  • Output option paired_with and group_with
  • Dynamic output (dynamic target)
• The depends statement
• SoS actions and common options (action options active, allow_error, args, container, engine, default_env, env, input, output, stdout, stderr, template, template_name, tracked, and workdir)
  • Command execution actions (actions run, sh, and bash)
  • Script execution actions (actions Python, R, JavaScript etc)
  • Report generation actions (actions report, pandoc, and RMarkdown)
  • Conditional actions (actions warn_if, skip_if, done_if and fail_if)
  • Action download
  • Running scripts in docker containers (Action option container and action docker_build)
  • Running scripts in singularity (Action options container and engine, action singularity_build)
• The task statement and option -q, task options queue, shared, env, prepend_path)
  • Host configuration (~/.sos/hosts.yml and sos remote command)
  • Task templates (options such as mem, cores, walltime, and nodes)
  • Task tags (option tags)
  • Combining tasks (options trunk_size and trunk_workers)
  • Path translation and file synchronization (options workdir, to_host and from_host)
  • Working with remote files (remote target)

SoS Workflows

• Summary of step dependencies
  • Named output (function named_output)
  • Output from another step (function output_from)
  • Explicit step and workflow dependency (target sos_step)
  • Makefile-style pattern-matching rules (section option provides)
  • Global and local variables
  • Sharing variables across steps (step and task option shared and target sos_variable)
• Introduction to SoS Workflows
  • Process-oriented workflows
  • Outcome-oriented workflows
  • Mixed-style workflows
  • Data-flow style workflows
  • Nested workflow (function sos_run)

Command line interface

• List of all sos commands and options
• Execution of workflows
  • Verbosity and number of workers (options -v and -j)
  • Runtime signature (option -s)
  • Error handling (option -e)
  • Dependency tracing (option -T)
  • Dryrun mode (option -n)
  • Customized environments and remote execution of workflows (option -r)
  • Output DAG of workflows (option -d)
  • Report generation (option -p)
• Utilities
  • Task management (commands status, kill, and purge)
  • Setup and test remote hosts (command sos remote)
  • Format conversion (command sos convert and %convert)

Publication

Please cite the following publication if you use SoS workflow engine for your work:

Script of Scripts: a pragmatic workflow system for daily computational research Gao Wang and Bo Peng (2019) PLoS Computational Biology. doi: 10.1371/journal.pcbi.1006843

Here are a list of examples from the publication:

• Process_Oriented.sos (script)
  Demonstration of a process_oriented forward-style workflow that executes steps in pre-sepecified order.
• Process_Oriented_Modular.sos (script)
  Same example as above with codes for core computations stored as separate scripts.
• Outcome_Oriented.sos (script)
  Demonstration of a outcome-oriented Makefile-style workflow that executes steps that are needed to generate specified outcome.
• Outcome_Oriented_Step_Targets.sos (script)
  Demonstration of a outcome-oriented Makefile-style workflow that determines and executes dependencies by step targets instead of files.
• Mixed_Style.sos (script)
  Demonstration of a mixed-style workflow that executes steps of a forward-step workflow with dependencies produced by auxiliary steps.
• Mixed_Style_Data_Flow.sos (script)
  Demonstration of a mixed-style workflow that executes steps of a forward-step workflow with dependencies specified by named outputs.
• Next-generation sequencing genotyping pipeline (notebook)
  This is the SoS version of the samtools workflow: WGS/WES Mapping to Variant Calls - Version 1.0
• RNA Seq Differential Expression Analysis Workflow (notebook and generated report)
  This is the SoS version of the RNA-seq workflow: gene-level exploratory analysis and differential expression from bioconductor.
• RNA Seq Normalization and Expression Residuals Analysis Workflow (notebook and generated report)
  This is the SoS version of the RNA-seq preprocessing step of eQTL discovery pipeline from the GTEx Consortium , updated July 31, 2017.

Please follow instructions here to reproduce all but the last (due to data accessibility restrictions) examples.

Example Scripts and workflow notebooks

SoS scripts for the maintenance of SoS

• update_toc.sos (script)
  A sos script that was used by SoS to update TOC of the wiki pages
• release (script)
  Script to manage SoS website and release SoS to pip, docker hub, etc

Pedagogical examples

• DEG Annotation with external task (notebook)
  The same example but submit the R part of the workflow as an external task
• Simple External Tasks (notebook)
  An example to generate and execute a number of external tasks in Jupyter notebook. You can try to use option trunk_size to group the tasks. This example also demonstrates the use of option shared to collect results from tasks as variables
• Remote Target (notebook)
  An example on the analysis of targets that reside on remote servers using target remote() and step option remote

Real-world applications

• Data preprocessing for TADA-A R analysis, for Liu et al 2018, American Journal of Human Genetics.
• Data preprocessing for MASH R analysis and Multivariate eQTL analysis pipeline, for Urbut et al 2018, Nature Genetics.