Script of Scripts (SoS) is a computational environment for the development and execution of scripts in multiple languages for daily computational research. It can be used to develop scripts to analyze data interactively in a Jupyter environment, and, with minimal effort, convert the scripts to a workflow that analyzes a large amount of data in batch mode.
SoS consists of a ployglot notebook that allows the use of multiple kernels in one Jupyter notebook, and a workflow system that is designed for daily computational research. Basically,
The figure is linked to a youtube video for a presentation on SoS during the 2018 JupyterCon, which introduces both SoS Notebook and SoS Workflow System and can be a good starting point for you to learn SoS. The SoS Workflow part starts at 20min.
As an interactive environment and notebook tool that promotes literate programming, SoS allows you to perform and record your analysis in different languages in a single Jupyter notebook, with seamless integration of multiple Jupyter kernels (e.g. python, and R). The ability to exchange data between live jupyter kernels allows you to use the most appropriate languages for different parts of your analyses.
SoS can work with all Jupyter kernels and currently provides native support (with data exchange) for the following languages:
SoS follows your definition of workflow that allows you to define workflows in both foward and makefile style, even a mixture of both. More specifically, you can define workflows as ordered steps that would be executed sequentially step by step to process specified input files; or as a set of makefile-style rules that would be executed to generated specified output files; or as a sequence of steps with help from makefile-style steps. SoS automatically analyzes the input and output of steps and execute workflows using a dynamic DAG (Directed Acyclic Graph) that would be expanded and shrinked during the execution.
SoS is very easy to learn so that you can start using it in minutes without having to learn any of the advanced features. As demonstrated in our Quick Start Guide, you can easily consolidate your scripts into an executable workflow using SoS and add more features to the script step by step if necessary. The multi-kernel feature of the Jupyter notebook is also easy to use with only two magics to remember.