SoS
Notebook environment for both interactive data analysis and batch data processing

SoS (Script of Scripts) is the umbrella title for SoS Polyglot Notebook and SoS Workflow Engine. SoS Notebook is a customized Jupyter kernel with associated frontend extensions that allows the use of multiple kernels in one Jupyter notebook. SoS Workflow engine is a Python 3 based workflow engine designed to be readable, sharable, and suitable for daily data analysis.

SoS Notebook SoS Workflow System

SoS Notebook and SoS workflow engine share the same website because they are designed to work with each other to facilitate both interactive data analysis and batch data processing.

A Multi-Language Notebook ...

... that Transforms ...

... to a Workflow


NOTE: Notebooks for these three animations are available under directory examples in our live SoS server.

Multi-language Jupyter Notebook

As an interactive environment and notebook tool that promotes literate programming, SoS allows you to perform and record your analysis in different languages in a single Jupyter notebook, with seamless integration of multiple Jupyter kernels (e.g. python, and R). The ability to exchange data between live jupyter kernels allows you to use the most appropriate languages for different parts of your analyses.

SoS can work with all Jupyter kernels and currently provides native support (with data exchange) for the following languages:

Bash
Bash
JavaScript
JavaScript
Julia
Julia
MATLAB
MATLAB
Ruby
Ruby
Octave
Octave
Python
Python 2
Python3
Python 3
R
R
SAS
SAS
Stata
Stata
TypeScript
TypeScript

Mixed Style Workflows

SoS follows your definition of workflow that allows you to define workflows in both foward and makefile style, even a mixture of both. More specifically, you can define workflows as ordered steps that would be executed sequentially step by step to process specified input files; or as a set of makefile-style rules that would be executed to generated specified output files; or as a sequence of steps with help from makefile-style steps. SoS automatically analyzes the input and output of steps and execute workflows using a dynamic DAG (Directed Acyclic Graph) that would be expanded and shrinked during the execution.

Power without Steep Learning Curve

SoS is very easy to learn so that you can start using it in minutes without having to learn any of the advanced features. As demonstrated in our Quick Start Guide, you can easily consolidate your scripts into an executable workflow using SoS and add more features to the script step by step if necessary. The multi-kernel feature of the Jupyter notebook is also easy to use with only two magics to remember.