Edit this page on our live server and create a PR by running command !create-pr in the console panel

Conversion of SoS files

The sos convert command allows you to convert from .sos to some other formats including ipynb (Jupyter notebook), HTML and MD (markdown), and from .ipynb to .sos. You can get a list of converters using command

In [1]:
usage: sos convert [-h] [-v {0,1,2,3,4}]
                   {sos-html,sos-term,sos-md,sos-ipynb,ipynb-sos,ipynb-html,ipynb-pdf,ipynb-md,ipynb-ipynb}
                   ...

Converts .sos to various formats including .html for web display, to jupyter
notebook (.ipynb), and to terminal for syntax highlighted viewing on terminal.
It also allows converting from jupyter notebook (.ipynb) to sos script (.sos).

optional arguments:
  -h, --help            show this help message and exit
  -v {0,1,2,3,4}, --verbosity {0,1,2,3,4}
                        Output error (0), warning (1), info (2), debug (3) and
                        trace (4) information to standard output (default to
                        2).

converters (name of converter is not needed from command line):
  {sos-html,sos-term,sos-md,sos-ipynb,ipynb-sos,ipynb-html,ipynb-pdf,ipynb-md,ipynb-ipynb}
    sos-html            Convert sos file to html format with syntax
                        highlighting, and save the output either to a HTML
                        file or view it in a broaser.
    sos-term            Write script to terminal with syntax highlighting.
    sos-md              Convert SOS scriot to a markdown format with scripts
                        quoted in markdown syntax.
    sos-ipynb           Convert a sos script to Jupyter notebook (.ipynb) so
                        that it can be opened by Jupyter notebook.
    ipynb-sos           Export Jupyter notebook with a SoS kernel to a .sos
                        file. The cells are presented in the .sos file as cell
                        structure lines, which will be ignored if executed in
                        batch mode
    ipynb-html          Export Jupyter notebook with a SoS kernel to a .html
                        file. Additional command line arguments are passed
                        directly to command "jupyter nbconvert --to html" so
                        please refer to nbconvert manual for available
                        options.
    ipynb-pdf           Export Jupyter notebook with a SoS kernel to a .pdf
                        file. Additional command line arguments are passed
                        directly to command "jupyter nbconvert --to pdf" so
                        please refer to nbconvert manual for available
                        options.
    ipynb-md            Export Jupyter notebook with a SoS kernel to a
                        markdown file. Additional command line arguments are
                        passed directly to command "jupyter nbconvert --to
                        markdown" so please refer to nbconvert manual for
                        available options.
    ipynb-ipynb         Export a Jupyter notebook with a non-SoS kernel to a
                        SoS notebook with SoS kernel. A SoS notebook will
                        simply be copied to the destination file.

Extra command line argument could be specified to customize the style of html,
markdown, and terminal output.

Command line interface

The convert command uses file extension and an option --to to determine the converter to use, and allows you to convert to a file or print the output to standard output. For example, you can use command

% sos convert myscript.sos myscript.html

to convert a sos script to a HTML file, or

% sos convert myscript.sos --to html

to write the HTML file to standard output.

If you would like to know available parameters for a particular converter, you can use option -h

In [2]:
usage: sos convert FILE.sos FILE.html (or --to html) [-h] [--raw RAW]
                                                     [--style {colorful,bw,tango,algol_nu,igor,algol,manni,vs,perldoc,borland,paraiso-light,default,paraiso-dark,trac,native,emacs,murphy,rrt,monokai,xcode,fruity,autumn,lovelace,pastie,vim,friendly}]
                                                     [--linenos] [-v]

Convert sos file to html format with syntax highlighting, and save the output
either to a HTML file or view it in a broaser.

optional arguments:
  -h, --help            show this help message and exit
  --raw RAW             URL to the raw sos file, which will be linked to
                        filenames in the HTML output
  --style {colorful,bw,tango,algol_nu,igor,algol,manni,vs,perldoc,borland,paraiso-light,default,paraiso-dark,trac,native,emacs,murphy,rrt,monokai,xcode,fruity,autumn,lovelace,pastie,vim,friendly}
                        Pygments style for the HTML output.
  --linenos             Display lineno to the left of the source code
  -v, --view            Open the output file in a broswer. In case no html
                        file is specified, this option will display the HTML
                        file in a browser, instead of writing its content to
                        standard output.

SoS -> HTML

The sos to html converter converts .sos script to HTML format. It can be either written to a HTML file, or to standard output if option --to html is specified without a destination filename.

In [3]:
INFO: SoS script saved to update_toc.html

The converter also accepts a number of parameters (as shown above). The raw parameter adds a URL to filename in the HTML file so that you can link to the raw .sos file from the .html output. The linenos adds line numbers, and style allows you to choose from a number of pre-specified styles. Finally, the view option would open the resulting HTML file in a browser.

For example,

sos convert ../examples/update_toc.sos --to html --view --style xcode

would show a HTML file as

HTML output of update_toc.sos

SoS -> ipynb

You can convert an existing SoS script to the .ipynb format using command

$ sos convert myscript.sos myscript.ipynb

and open the resulting notebook from the web interface.

Depending on how the sos file was written (or converted from .ipynb file), sos takes different approach in splitting content of the script into cells of ipython notebook. More specifically,

  1. If the script contains cell spliting magic %cell, sos would split the sos script according to %cell, and restore proper metadata (e.g. kernel used for each cell) and execution index. Such a .sos file is usually exported from command sos convert ipynb sos --all.
  2. If the script does not contain cell spliting magic, sos will split each step into a separate cell.

The resulting .ipynb files have only input code and markdown cells. However, you can execute the notebook from command line if you add an -e (or --execute) option to the converter. That is to say, if you execute

$ sos convert myscript.sos myscript.ipynb --execute

You would get a notebook with both input and output cells, as if you have opened the non-executed ipynb file and selected Cell -> Run all.

ipynb -> SoS

A Jupyter notebook can contain markdown cell and code cell with different kernels, and a sos cell might or might not contain a real sos step (with section header).

You can save a Jupyter notebook with SoS kernel to a SoS script using File -> Download As -> SoS from the browser, or using command

$ sos convert myscript.ipynb myscript.sos

The conversion process will export the workflow defined in the jupyter notebook in the resulting .sos file, ignoring all cell magics, non-sos cells, and sos cells that do not start with section headers. The resulting .sos file is a proper SoS workflow and can be executed by the sos command in batch mode.

If you would like to keep most of the information of the Jupyter notebook, you can convert the notebook using option --all (or -a),

$ sos convert myscript.ipynb myscript.sos --all

This option will export each cell with a %cell magic and record meta data in the .sos file. The resulting file can be converted back to ipynb format with minimal lose of information (without result), but might not be able to be executed in batch mode.

ipynb -> HTML

Command sos convert my.ipynb my.html --template essentially calls jupyter nbconvert --to html to convert notebook to HTML format, with additional templates provided by SoS Notebook.

SoS provides the following templates

template code highlighting TOC Hide Cell Suit for
sos-full jupyter no no static short report
sos-cm SoS CodeMirror no no output similar to notebook interface
sos-report jupyter no yes static report with hidden details
sos-full-toc jupyter yes no static long report
sos-cm-toc SoS CodeMirror yes none output of long notebook with notebook interface
sos-report-toc jupyter yes yes long report with hidden details

Where:

  • code highlighting: Jupyter can hilight sos source code using a static syntax hilighter. The output is lightweight but not as nice as the codemirror highlighter.
  • TOC: Automatically generate a table of content to the left of the page.
  • Hide Cell: HTML page by default only displays only markdown and selected ouptput cells (cells with report_cell tag, output of cells with report_output tag). A control panel to the left top corner of the page can be used to show all content

ipynb -> pdf

This command essentially calls command jupyter nbconvert --to pdf to convert notebook to PDF format.

ipynb -> md

This command essentially calls command jupyter nbconvert --to markdown to convert notebook to Markdown format.

ipynb -> ipynb

This command converts a Jupyter notebook in another kernel to a SoS notebook, with the original kernel language as the language of each code cell.

If the original notebook has kernel python3, an option --python3-to-sos can be used to convert code cells to SoS.

This converter will copy the input notebook to output if the notebook is already a SoS notebook. However, if an option --inplace is specified, it will overwrite the original notebook with the converted one.

Note that if you already have a non-SoS notebook opened in Jupyter, you can simply use

Kernel -> Change kernel -> sos

to convert the kernel to SoS. You can then use the global language selector to select the appropriate default langauge for the notebook and re-execute the notebook to set the language to each cell.

Rmd -> ipynb

RMarkdown is a Markdown format with inline and block code executed by RStudio. SoS can convert RMarkdown files to a SoS notebook but the conversion is limited. In particular, SoS converts the block code to SoS code cells and the rest into Markdown cells, and leave inline expressions untouched. It also ignores all code options.

Note that SoS does not support string interpolation in markdown cells so inline expressions in RMarkdown are not converted. If you need to insert R expression in markdown, you can convert the paragraph into a code cell and use R to print the evaluated string to standard output (use cat function). Then you can add a %render magic to the beginning of the cell so to render the outputed markdown text.