- Difficulty level: hard
- Time need to lean: refer to this document only when needed
- Key points:
- Use
%<tab>to get a list of magics - Use
%MAGIC -hto get the help messages
- Use
In addition to SoS statements, you can use a few SoS magics in Jupyter notebooks with a SoS kernel. Note that
- SoS magics have to be specified at the beginning of a cell, although they can be specified after empty lines and comments.
- Lines ending with
"\"will be joined so you can break long magics into multiple lines - Multiple magics can be used in a single cell.
In any SoS cell, you can type % and use the <TAB> key to get a list of SoS magics. Then, for any magic you need to use, you can type %MAGIC -h to get the help information. This reference manual is simply a composition of these help messages.
In [1]:
usage: %capture [-h] [--as [{text,json,csv,tsv}]] [-t VAR | -a VAR]
[{stdout,stderr,text,markdown,html,raw,error}]
Capture output from a subkernel as variable in SoS
positional arguments:
{stdout,stderr,text,markdown,html,raw,error}
Message type to capture. In terms of Jupyter message
types, "stdout" refers to "stream" message with
"stdout" type, "stderr" refers to "stream" message
with "stderr" type, "text", "markdown" and "html"
refers to "display_data" or "execute_result" messages
with "text/plain", "text/markdown" and "text/html"
type respectively, and 'error' refers to "evalue" of
"error" messages. If no value or "raw" is specified,
all returned messages will be returned in alist
format, and will be displayed in the console panel.
This will help you determine the right type to
capture.
optional arguments:
-h, --help show this help message and exit
--as [{text,json,csv,tsv}]
How to interpret the captured text. This only
applicable to stdout, stderr and text message type
where the text from cell output will be collected. If
this option is given, SoS will try to parse the text
as json, csv (comma separated text), tsv (tab
separated text), and store text (from text), Pandas
DataFrame (from csv or tsv), dict or other types (from
json) to the variable.
-t VAR, --to VAR Name of variable to which the captured content will be
saved. If no varialbe is specified, the return value
will be saved to variable "__captured" and be
displayed at the side panel.
-a VAR, --append VAR Name of variable to which the captured content will be
appended. This option is equivalent to --to if VAR
does not exist. If VAR exists and is of the same type
of new content (str or dict or DataFrame), the new
content will be appended to VAR if VAR is of str (str
concatenation), dict (dict update), or DataFrame
(DataFrame.append) types. If VAR is of list type, the
new content will be appended to the end of the list.
In [2]:
usage: %cd [-h] dir change directory of SoS and all subkernels. positional arguments: dir destination directory optional arguments: -h, --help show this help message and exit
In [3]:
usage: %convert [-h] [-t {sos,html}] [-f] [--template TEMPLATE] [filename]
Convert the current notebook to another format.
positional arguments:
filename Filename of saved report or script. Default to
notebookname with file extension determined by option
--to.
optional arguments:
-h, --help show this help message and exit
-t {sos,html}, --to {sos,html}
Destination format, default to html.
-f, --force If destination file already exists, overwrite it.
--template TEMPLATE Template to generate HTML output. The default template
is a template defined by configuration key default-
sos-template, or sos-report-toc if such a key does not
exist.
In [4]:
usage: %env [-h] [--new] [--set SET [SET ...]]
[--prepend-path PREPEND_PATH [PREPEND_PATH ...]] [--tempdir]
[--expect-error] [--allow-error]
Adjust the running environment for the cell, such as running with a new dict,
under a different directory, and expect an error from the execution of the
cell.
optional arguments:
-h, --help show this help message and exit
--new Execute workflow with a fresh SoS environment
--set SET [SET ...] Set one more more environment variables. Parameters of
this option can be 'KEY=VALUE' or just 'KEY'. An empty
evnironment variable will be set in the latter case.
Note that the environments will be reset out of the
cell.
--prepend-path PREPEND_PATH [PREPEND_PATH ...]
Prepend one or more paths before "$PATH" so that
commands in those paths will take priority. Note that
`$PATH` will be reset after the completion of the
cell.
--tempdir Execute workflow in temporary directory, which will be
removed after the completion of the cell. Therefore
you cannot use this option when running the cell
content in background mode.
--expect-error If set, expect error from the excution and report
success if an error occurs, and report error if an
error does not occur.
--allow-error If set, return success even if the underlying cell
reports error.
In [5]:
usage: %expand [-h] [-i KERNEL] [sigil] [right_sigil]
Expand the script in the current cell with default ({}) or specified sigil.
positional arguments:
sigil Sigil to be used to interpolated the texts. It can be
quoted, or be specified as two options.
right_sigil Right sigil if the sigil is specified as two pieces.
optional arguments:
-h, --help show this help message and exit
-i KERNEL, --in KERNEL
Expand the cell content in specific kernel, default to
"SoS". This requires that the language module supports
the "expand" featire.
In [6]:
usage: %get [-h] [--from __FROM__] [vars [vars ...]]
Get specified variables from another kernel, which is by default the SoS
kernel.
positional arguments:
vars Names of SoS variables
optional arguments:
-h, --help show this help message and exit
--from __FROM__ Name of kernel from which the variables will be obtained.
Default to the SoS kernel.
In [7]:
usage: %matplotlib [-h] [-l]
[{agg,gtk,gtk3,inline,ipympl,nbagg,notebook,osx,pdf,ps,qt,qt4,qt5,svg,tk,widget,wx}]
Set matplotlib parser type
positional arguments:
{agg,gtk,gtk3,inline,ipympl,nbagg,notebook,osx,pdf,ps,qt,qt4,qt5,svg,tk,widget,wx}
Name of the matplotlib backend to use (‘agg’, ‘gtk’,
‘gtk3’,
optional arguments:
-h, --help show this help message and exit
-l, --list Show available matplotlib backends
In [8]:
usage: %put [-h] [--to __TO__] [vars [vars ...]]
Put specified variables in the subkernel to another kernel, which is by
default the SoS kernel.
positional arguments:
vars Names of SoS variables
optional arguments:
-h, --help show this help message and exit
--to __TO__ Name of kernel from which the variables will be obtained.
Default to the SoS kernel.
In [9]:
usage: %preview [-h] [-k KERNEL] [-w] [-s {table,scatterplot,png}] [-r HOST]
[-p | -n] [-c CONFIG]
[items [items ...]]
Preview files, sos variables, or expressions in the side panel, or notebook if
side panel is not opened, unless options --panel or --notebook is specified.
positional arguments:
items Filename, variable name, or expression. Wildcard
characters such as '*' and '?' are allowed for
filenames.
optional arguments:
-h, --help show this help message and exit
-k KERNEL, --kernel KERNEL
kernel in which variables will be previewed. By
default the variable will be previewed in the current
kernel of the cell.
-w, --workflow Preview notebook workflow
-s {table,scatterplot,png}, --style {table,scatterplot,png}
Option for preview file or variable, which by default
is "table" for Pandas DataFrame. The %preview magic
also accepts arbitrary additional keyword arguments,
which would be interpreted by individual style.
Passing '-h' with '--style' would display the usage
information of particular style.
-r HOST, --host HOST Preview files on specified remote host, which should
be one of the hosts defined in sos configuration
files.
-p, --panel Preview in side panel even if the panel is currently
closed
-n, --notebook Preview in the main notebook.
-c CONFIG, --config CONFIG
A configuration file with host definitions, in case
the definitions are not defined in global or local sos
config.yml files.
In [10]:
usage: %render [-h] [--as [AS_TYPE]] [{stdout,text}]
Treat the output of a SoS cell as another format, default to markdown.
positional arguments:
{stdout,text} Message type to capture, default to standard output. In
terms of Jupyter message types, "stdout" refers to "stream"
message with "stdout" type, and "text" refers to
"display_data" message with "text/plain" type.
optional arguments:
-h, --help show this help message and exit
--as [AS_TYPE] Format to render output of cell, default to Markdown, but
can be any format that is supported by the IPython.display
module such as HTML, Math, JSON, JavaScript and SVG.
In [11]:
usage: %revision [-h] [-s [SOURCE]] [-l LINKS [LINKS ...]]
Revision history of the document, parsed from the log message of the notebook
if it is kept in a git repository. Additional parameters to "git log" command
(e.g. -n 5 --since --after) could be specified to limit the revisions to
display.
optional arguments:
-h, --help show this help message and exit
-s [SOURCE], --source [SOURCE]
Source URL to to create links for revisions. SoS
automatically parse source URL of the origin and
provides variables "repo" for complete origin URL
without trailing ".git" (e.g.
https://github.com/vatlab/sos-notebook), "path" for
complete path name (e.g. src/document/doc.ipynb),
"filename" for only the name of the "path", and
"revision" for revisions. Because sos interpolates
command line by default, variables in URL template
should be included with double braceses (e.g. --source
{{repo}}/blob/{{revision}}/{{path}})). If this option
is provided without value and the document is hosted
on github, a default template will be provided.
-l LINKS [LINKS ...], --links LINKS [LINKS ...]
Name and URL or additional links for related files
(e.g. --links report URL_to_repo ) with URL
interpolated as option --source.
In [12]:
usage: %run [-h] Execute the current cell with specified command line arguments. If the magic ends with "&", it will be sent to a queue to be executed sequentially. optional arguments: -h, --help show this help message and exit
In [13]:
usage: %runfile [-h] script Execute an external SoS script, which is identical to run !sos run script but allows the display of task and workflow status in notebook. It also accepts default parameters of %set magic. positional arguments: script Script to be executed. optional arguments: -h, --help show this help message and exit
Magic %save saves the content of the current cell (after the magic itself) to specified file. It accepts the following options:
In [14]:
usage: %save [-h] [-r] [-f] [-a] [-x] filename Save the content of the cell to specified file. It ignores magic lines after the %save magic unless a blank line is used to separate the magics and the content to be saved. positional arguments: filename Filename of saved report or script. optional arguments: -h, --help show this help message and exit -r, --run Continue to execute the cell once content is saved. -f, --force If destination file already exists, overwrite it. -a, --append If destination file already exists, append to it. -x, --set-executable Set `executable` permission to saved script.
In [15]:
usage: %sosrun [-h] Execute the entire notebook with steps consisting of SoS cells (cells with SoS kernel) with section header, with specified command line arguments. Arguments set by magic %set will be appended at the end of command line. If the magic ends with "&", it will be sent to a queue to be executed sequentially. optional arguments: -h, --help show this help message and exit
In [16]:
usage: %sessioninfo [-h] [-w __WITH__]
List the session info of all subkernels, and information stored in variable
sessioninfo
optional arguments:
-h, --help show this help message and exit
-w __WITH__, --with __WITH__
Name of variable that contains extra information to be
appended. This variable should be a dictionary, with
keys being the section headers and items being the
session information, which can be a string, a list of
strings, a dictionary, or a list of `(key, value)`
pairs. Encoded strings (bytes) are acceptable in
places of strings.
In [17]:
usage: %shutdown [-h] [-r] [kernel]
Shutdown or restart specified subkernel
positional arguments:
kernel Name of the kernel to be restarted, default to the current
running kernel.
optional arguments:
-h, --help show this help message and exit
-r, --restart Restart the kernel
In [18]:
usage: %task [-h] [-a | --age AGE | -s STATUS [STATUS ...] | -t
[TAGS [TAGS ...]]]
{status,execute,kill,purge} ...
Get information on specified task. By default sos would query against all
running task queues but it would start a task queue and query status if option
-q is specified.
positional arguments:
{status,execute,kill,purge}
actions
status task status
execute execute task
kill kill single task or tasks with the same tags
purge kill and purge task
optional arguments:
-h, --help show this help message and exit
-a, --all Clear all task information on local or specified
remote task queue, including tasks created by other
workflows.
--age AGE Remove all tasks that are created more than (default)
or within specified age. Value of this parameter can
be in units s (second), m (minute), h (hour), or d
(day, default), or in the foramt of HH:MM:SS, with
optional prefix + for older (default) and - for newer
than specified age.
-s STATUS [STATUS ...], --status STATUS [STATUS ...]
Remove all tasks with specified status, which can be
pending, submitted, running, completed, failed, and
aborted. One of more status can be specified.
-t [TAGS [TAGS ...]], --tags [TAGS [TAGS ...]]
Remove all tsks with one of the specified tags.
In [19]:
usage: %task status [-h] [-q QUEUE] [-c CONFIG] [-v {0,1,2,3,4}]
[-t [TAGS [TAGS ...]]] [-s [STATUS [STATUS ...]]]
[--age AGE]
[tasks [tasks ...]]
positional arguments:
tasks ID of the task. All tasks that are releted to the
workflow executed under the current directory will be
checked if unspecified. There is no need to specify
compelete task IDs because SoS will match specified
name with tasks starting with these names.
optional arguments:
-h, --help show this help message and exit
-q QUEUE, --queue QUEUE
Check the status of job on specified tasks queue or
remote host if the tasks . The queue can be defined in
global or local sos configuration file, or a file
specified by option --config. A host is assumed to be
a remote machine with process type if no configuration
is found.
-c CONFIG, --config CONFIG
A configuration file with host definitions, in case
the definitions are not defined in global sos
config.yml files.
-v {0,1,2,3,4} Output error (0), warning (1), info (2), and debug (3)
information to standard output (default to 2).
-t [TAGS [TAGS ...]], --tags [TAGS [TAGS ...]]
Only list tasks with one of the specified tags.
-s [STATUS [STATUS ...]], --status [STATUS [STATUS ...]]
Display tasks with one of the specified status.
--age AGE Limit to tasks that are created more than (default) or
within specified age. Value of this parameter can be
in units s (second), m (minute), h (hour), or d (day,
default), or in the foramt of HH:MM:SS, with optional
prefix + for older (default) and - for newer than
specified age.
In [20]:
usage: %task execute [-h] [-q QUEUE] [-c CONFIG] [-v {0,1,2,3,4}]
[tasks [tasks ...]]
positional arguments:
tasks ID of the tasks to be removed. There is no need to
specify compelete task IDs because SoS will match
specified name with tasks starting with these names.
If no task ID is specified, all tasks related to
specified workflows (option -w) will be removed.
optional arguments:
-h, --help show this help message and exit
-q QUEUE, --queue QUEUE
Remove tasks on specified tasks queue or remote host
if the tasks . The queue can be defined in global or
local sos configuration file, or a file specified by
option --config. A host is assumed to be a remote
machine with process type if no configuration is
found.
-c CONFIG, --config CONFIG
A configuration file with host definitions, in case
the definitions are not defined in global sos
config.yml files.
-v {0,1,2,3,4} Output error (0), warning (1), info (2), and debug (3)
information to standard output (default to 2).
In [21]:
usage: %task kill [-h] [-a] [-q QUEUE] [-t [TAGS [TAGS ...]]] [-c CONFIG]
[-v {0,1,2,3,4}]
[tasks [tasks ...]]
positional arguments:
tasks IDs of the tasks that will be killed. There is no need
to specify compelete task IDs because SoS will match
specified name with tasks starting with these names.
optional arguments:
-h, --help show this help message and exit
-a, --all Kill all tasks in local or specified remote task queue
-q QUEUE, --queue QUEUE
Kill jobs on specified tasks queue or remote host if
the tasks . The queue can be defined in global or
local sos configuration file, or a file specified by
option --config. A host is assumed to be a remote
machine with process type if no configuration is
found.
-t [TAGS [TAGS ...]], --tags [TAGS [TAGS ...]]
Only kill tasks with one of the specified tags.
-c CONFIG, --config CONFIG
A configuration file with host definitions, in case
the definitions are not defined in global sos
config.yml files.
-v {0,1,2,3,4}, --verbosity {0,1,2,3,4}
Output error (0), warning (1), info (2), and debug (3)
information to standard output (default to 2).
In [22]:
usage: %task purge [-h] [-q QUEUE] [-c CONFIG] [-v {0,1,2,3,4}]
[tasks [tasks ...]]
positional arguments:
tasks ID of the tasks to be removed. There is no need to
specify compelete task IDs because SoS will match
specified name with tasks starting with these names.
If no task ID is specified, all tasks related to
specified workflows (option -w) will be removed.
optional arguments:
-h, --help show this help message and exit
-q QUEUE, --queue QUEUE
Remove tasks on specified tasks queue or remote host
if the tasks . The queue can be defined in global or
local sos configuration file, or a file specified by
option --config. A host is assumed to be a remote
machine with process type if no configuration is
found.
-c CONFIG, --config CONFIG
A configuration file with host definitions, in case
the definitions are not defined in global sos
config.yml files.
-v {0,1,2,3,4} Output error (0), warning (1), info (2), and debug (3)
information to standard output (default to 2).
In [23]:
usage: %use [-h] [-k KERNEL] [-l LANGUAGE] [-c COLOR] [-r] [name]
Switch to an existing subkernel or start a new subkernel.
positional arguments:
name Displayed name of kernel to start (if no kernel with
name is specified) or switch to (if a kernel with this
name is already started). The name is usually a kernel
name (e.g. %use ir) or a language name (e.g. %use R)
in which case the language name will be used. One or
more parameters --language or --kernel will need to be
specified if a new name is used to start a separate
instance of a kernel.
optional arguments:
-h, --help show this help message and exit
-k KERNEL, --kernel KERNEL
kernel name as displayed in the output of jupyter
kernelspec list. Default to the default kernel of
selected language (e.g. ir for language R.
-l LANGUAGE, --language LANGUAGE
Language extension that enables magics such as %get
and %put for the kernel, which should be in the name
of a registered language (e.g. R), or a specific
language module in the format of package.module:class.
SoS maitains a list of languages and kernels so this
option is only needed for starting a new instance of a
kernel.
-c COLOR, --color COLOR
Background color of new or existing kernel, which
overrides the default color of the language. A special
value "default" can be used to reset color to default.
-r, --restart Restart the kernel if it is running.
In [24]:
usage: %with [-h] [-i [IN_VARS [IN_VARS ...]]] [-o [OUT_VARS [OUT_VARS ...]]]
[name]
Use specified subkernel to evaluate current cell, with optional input and
output variables
positional arguments:
name Name of an existing kernel.
optional arguments:
-h, --help show this help message and exit
-i [IN_VARS [IN_VARS ...]], --in [IN_VARS [IN_VARS ...]]
Input variables (variables to get from SoS kernel)
-o [OUT_VARS [OUT_VARS ...]], --out [OUT_VARS [OUT_VARS ...]]
Output variables (variables to put back to SoS kernel
before switching back to the SoS kernel
!shell-command
If any other command is entered after !, sos will treat the rest of the line as a shell command and execute it. Only single-line commands are supported. String interpolation is supported. Note that !cd does not change the current working directory because the command is executed in a separate process. Use magic %cd for that purpose.