On the Generation of HTML Report from Jupyter Notebook

This example demonstrates how to use the %preview magic to preview results and how to generate a report in HTML format from a Jupyter notebook with SoS kernel. The report has the following features:

The report contains all details of the analysis but only displays output or both input and output of selected cells by default.
A control panel (hidden at the top left corner of the screen) can be used to display hidden contents.
The report can contain interactive tables and plots generated from the %preview magic of SoS. The tables are sortable and filterable, and is displayed in a scrollable window so you can include complete tables with thousand of records in a compact format. You can hover over data points of scatter plots to get details of each data point.

Description of data

We have got an excel file with a list of differentially expressed genes for certain experiment. The analysis was done with ensembl genes so it is difficult to figure out what they are. The goal of the analysis is to annotate the gene IDs with HGNC names.

Let us first read the data from the excel file and have a look:

%preview -n data -l 2000
depends: Py_Module('xlrd')
import pandas as pd
data = pd.read_excel('DEG_list.xlsx')

Data analysis

The biomaRt bioconductor package provides required datasets to annotate the ensembl IDs so we transfer the gene list from Python to R, and use the biomaRt package to annotate the IDs.

As we can see from the following list, the genes are correctly annotated

%get data

%preview -n hgnc[1:5,]
library(biomaRt)
ensembl <- useEnsembl(biomart='ensembl')
#listDatasets(ensembl)
ensembl <- useEnsembl(biomart="ensembl", dataset="mmusculus_gene_ensembl")
                     
hgnc <- getBM(attributes=c('ensembl_gene_id', 'external_gene_name'),
        filters = 'ensembl_gene_id', values = data['ensembl_gene_id'], mart = ensembl)

annotated <- merge(data, hgnc, by='ensembl_gene_id', all.x=TRUE)

Result

We transfer the annotated result back to Python and write it to another excel file. The content of the file is previewed and list here:

%get annotated --from R
%preview -n annotated_DEG_list.xlsx -l 2000
annotated = annotated.set_index('external_gene_name')
annotated.sort_values(by='padj', inplace=True)
annotated.to_excel('annotated_DEG_list.xlsx')

Just to demonstrate the scatterplot style of %preview magic and the report_cell tag, here is the command to display a scatter plot of log2FoldChange vs baseMean. With the tooltip, you can easily figure out which gene has the highest baseMean.

%preview annotated -n -s scatterplot log2FoldChange baseMean \
    --tooltip ensembl_gene_id pvalue padj --log y

Report Generation

Now that we have completed our analysis, you can mark the cells with tags to control how they appear in the report. If you are not familiar with Jupyter, a tag is any string that can be attached to a cell. You can view, add, and remove tags from the tag toolbar, which can be turned on using View -> Cell Toolbar -> Tags.

You can

Mark code cells that will be outputed with shortcut Ctrl-Shift-O. This will add a report_output tag to the cell and mark the cell with a gray bar to the right of the output area.
Using the same shortcut, mark markdown cell that will be hidden with tag hide_output.
Exclude cells from the report by tagging them with tag scratch. There is no shortcut so you have to do this manually.

Now you have a notebook with the following types of cells, they will appear in the generated report as follows:

|Cell type | Tag | Default status in report | |---|---|---|---| |Markdown cell| None | Displayed| |code cell| None | Hidden | |Markdown cell|hide_output | Hidden| |Code cell|report_cell| Displayed with hidden input and messages| |Code cell|report_output| Displayed with hidden messages | |Code cell|scratch| Excluded, not in report | |Markdown cell|scratch| Excluded, not in report |

To generate a report, you can execute cell magic

%sossave --to html --force

to convert file.ipynb to file.html under the same directory. The --force option allows SoS to overwrite an existing report. This command is available from the dropdown list of the panel cell input box so you do not have to enter it manually.

If you would like to save the report in another name or to another directory, you can use magic

%sossave myreport.html --force

or use convert the notebook from the command line using command

$ sos convert myanalysis.ipynb myanalysis.html --template sos-report

And this paragraph will be hidden from the report because of the hide_output tag.

ensembl_gene_id	external_gene_name
ENSMUSG00000000214	Th
ENSMUSG00000000278	Scpep1
ENSMUSG00000000308	Ckmt1
ENSMUSG00000000320	Alox12
ENSMUSG00000000416	Cttnbp2

	ensembl_gene_id	baseMean	log2FoldChange	stat	pvalue	padj
0	ENSMUSG00000042816	1686.121821	3.390100	-28.589868	8.979679e-180	7.696034e-176
1	ENSMUSG00000031635	358.401785	3.614133	-21.471980	2.846325e-102	8.131476e-99
2	ENSMUSG00000020838	551.448621	2.532469	-18.878942	1.699551e-79	2.913201e-76
3	ENSMUSG00000021765	597.716318	2.265537	-17.571072	4.103412e-69	5.024042e-66
4	ENSMUSG00000027907	3236.448003	1.306965	-15.152852	7.254873e-52	6.908654e-49
5	ENSMUSG00000026247	744.567819	1.999957	-14.202403	8.852428e-46	6.897249e-43
6	ENSMUSG00000030272	3322.013499	1.029098	-13.056674	5.821582e-39	3.837990e-36
7	ENSMUSG00000069793	317.952685	1.645894	-12.727226	4.174068e-37	2.384924e-34
8	ENSMUSG00000028494	4659.305199	1.051199	-12.102120	1.029183e-33	5.188595e-31
9	ENSMUSG00000059645	107.043454	1.777720	-11.629145	2.930187e-31	1.287855e-28
10	ENSMUSG00000026834	184.146231	1.576185	-10.924823	8.771122e-28	3.496414e-25
11	ENSMUSG00000069355	78.423674	1.896248	-10.639715	1.947240e-26	7.101625e-24
12	ENSMUSG00000024066	1207.689222	1.287185	-10.412868	2.165954e-25	7.279729e-23
13	ENSMUSG00000059173	2047.297734	1.290401	-10.363868	3.620191e-25	1.128249e-22
14	ENSMUSG00000044626	107.545685	1.479431	-9.920675	3.384591e-23	9.833097e-21
15	ENSMUSG00000029552	990.570372	0.942947	-9.652480	4.798095e-22	1.305463e-19
16	ENSMUSG00000000278	2246.095680	0.820046	-9.477766	2.597862e-21	6.646262e-19
17	ENSMUSG00000023067	3740.729262	0.935633	-9.180521	4.290090e-20	1.035725e-17
18	ENSMUSG00000021843	8362.210457	0.774680	-8.942443	3.806602e-19	8.699862e-17
19	ENSMUSG00000044258	622.236008	1.062715	-8.723723	2.692001e-18	5.840961e-16
20	ENSMUSG00000020205	1313.876243	1.029466	-8.587091	8.919931e-18	1.842127e-15
21	ENSMUSG00000037280	336.544531	1.119365	-8.342550	7.270520e-17	1.432460e-14
22	ENSMUSG00000027556	365.075577	1.161261	-8.021451	1.045036e-15	1.968457e-13
23	ENSMUSG00000031799	7421.027362	0.624473	-7.913186	2.508835e-15	4.526730e-13
24	ENSMUSG00000036905	1706.843435	0.709253	-7.846027	4.294230e-15	7.435091e-13
25	ENSMUSG00000044667	545.471721	0.867126	-7.784684	6.988793e-15	1.163057e-12
26	ENSMUSG00000021125	389.752252	0.884616	-7.713982	1.219520e-14	1.953626e-12
27	ENSMUSG00000025351	5138.213127	0.601866	-7.593747	3.107838e-14	4.799230e-12
28	ENSMUSG00000032666	5242.276743	0.592958	-7.454353	9.030998e-14	1.346090e-11
29	ENSMUSG00000019866	554.456799	0.850782	-7.439136	1.013460e-13	1.459809e-11
30	ENSMUSG00000025608	1575.642204	0.686405	-7.364919	1.772542e-13	2.470174e-11
31	ENSMUSG00000043336	572.398895	0.802930	-7.234809	4.661851e-13	6.292031e-11
32	ENSMUSG00000037940	265.079339	0.911853	-7.210731	5.565229e-13	7.312611e-11
33	ENSMUSG00000039364	194.167058	0.998772	-6.998846	2.580798e-12	3.276849e-10
34	ENSMUSG00000056749	345.920854	0.914688	-6.969709	3.175962e-12	3.916486e-10
35	ENSMUSG00000032802	17749.447535	0.598266	-6.892196	5.493748e-12	6.585198e-10
36	ENSMUSG00000024672	211.819493	0.887656	-6.825385	8.769012e-12	1.022515e-09
37	ENSMUSG00000010086	4098.481132	-0.572777	6.720270	1.813879e-11	2.059053e-09
38	ENSMUSG00000058665	29.247691	0.967621	-6.683662	2.330438e-11	2.577164e-09
39	ENSMUSG00000009596	58.610232	0.944848	-6.642619	3.081587e-11	3.301343e-09

	external_gene_name	ensembl_gene_id	baseMean	log2FoldChange	stat	pvalue	padj
0	Gpr151	ENSMUSG00000042816	1686.121821	3.390100	-28.589868	8.979679e-180	7.696034e-176
1	Anxa10	ENSMUSG00000031635	358.401785	3.614133	-21.471980	2.846325e-102	8.131476e-99
2	Slc6a4	ENSMUSG00000020838	551.448621	2.532469	-18.878942	1.699551e-79	2.913201e-76
3	Fst	ENSMUSG00000021765	597.716318	2.265537	-17.571072	4.103412e-69	5.024042e-66
4	S100a11	ENSMUSG00000027907	3236.448003	1.306965	-15.152852	7.254873e-52	6.908654e-49
5	Ecel1	ENSMUSG00000026247	744.567819	1.999957	-14.202403	8.852428e-46	6.897249e-43
6	Camk1	ENSMUSG00000030272	3322.013499	1.029098	-13.056674	5.821582e-39	3.837990e-36
7	Slfn9	ENSMUSG00000069793	317.952685	1.645894	-12.727226	4.174068e-37	2.384924e-34
8	Plin2	ENSMUSG00000028494	4659.305199	1.051199	-12.102120	1.029183e-33	5.188595e-31
9	Gm7361	ENSMUSG00000059645	107.043454	1.777720	-11.629145	2.930187e-31	1.287855e-28
10	Acvr1c	ENSMUSG00000026834	184.146231	1.576185	-10.924823	8.771122e-28	3.496414e-25
11	Gm5152	ENSMUSG00000069355	78.423674	1.896248	-10.639715	1.947240e-26	7.101625e-24
12	Xdh	ENSMUSG00000024066	1207.689222	1.287185	-10.412868	2.165954e-25	7.279729e-23
13	Pde1a	ENSMUSG00000059173	2047.297734	1.290401	-10.363868	3.620191e-25	1.128249e-22
14	Liph	ENSMUSG00000044626	107.545685	1.479431	-9.920675	3.384591e-23	9.833097e-21
15	Tes	ENSMUSG00000029552	990.570372	0.942947	-9.652480	4.798095e-22	1.305463e-19
16	Scpep1	ENSMUSG00000000278	2246.095680	0.820046	-9.477766	2.597862e-21	6.646262e-19
17	Cdkn1a	ENSMUSG00000023067	3740.729262	0.935633	-9.180521	4.290090e-20	1.035725e-17
18	Ktn1	ENSMUSG00000021843	8362.210457	0.774680	-8.942443	3.806602e-19	8.699862e-17
19	Ctla2a	ENSMUSG00000044258	622.236008	1.062715	-8.723723	2.692001e-18	5.840961e-16
20	Phlda1	ENSMUSG00000020205	1313.876243	1.029466	-8.587091	8.919931e-18	1.842127e-15
21	Galnt6	ENSMUSG00000037280	336.544531	1.119365	-8.342550	7.270520e-17	1.432460e-14
22	Car1	ENSMUSG00000027556	365.075577	1.161261	-8.021451	1.045036e-15	1.968457e-13
23	Tpm4	ENSMUSG00000031799	7421.027362	0.624473	-7.913186	2.508835e-15	4.526730e-13
24	C1qb	ENSMUSG00000036905	1706.843435	0.709253	-7.846027	4.294230e-15	7.435091e-13
25	Plppr4	ENSMUSG00000044667	545.471721	0.867126	-7.784684	6.988793e-15	1.163057e-12
26	Arg2	ENSMUSG00000021125	389.752252	0.884616	-7.713982	1.219520e-14	1.953626e-12
27	Cd63	ENSMUSG00000025351	5138.213127	0.601866	-7.593747	3.107838e-14	4.799230e-12
28	1700025G04Rik	ENSMUSG00000032666	5242.276743	0.592958	-7.454353	9.030998e-14	1.346090e-11
29	Aim1	ENSMUSG00000019866	554.456799	0.850782	-7.439136	1.013460e-13	1.459809e-11
30	Podxl	ENSMUSG00000025608	1575.642204	0.686405	-7.364919	1.772542e-13	2.470174e-11
31	Filip1l	ENSMUSG00000043336	572.398895	0.802930	-7.234809	4.661851e-13	6.292031e-11
32	Inpp4b	ENSMUSG00000037940	265.079339	0.911853	-7.210731	5.565229e-13	7.312611e-11
33	Sectm1b	ENSMUSG00000039364	194.167058	0.998772	-6.998846	2.580798e-12	3.276849e-10
34	Nfil3	ENSMUSG00000056749	345.920854	0.914688	-6.969709	3.175962e-12	3.916486e-10
35	Srxn1	ENSMUSG00000032802	17749.447535	0.598266	-6.892196	5.493748e-12	6.585198e-10
36	Ms4a7	ENSMUSG00000024672	211.819493	0.887656	-6.825385	8.769012e-12	1.022515e-09
37	Rnf112	ENSMUSG00000010086	4098.481132	-0.572777	6.720270	1.813879e-11	2.059053e-09
38	En1	ENSMUSG00000058665	29.247691	0.967621	-6.683662	2.330438e-11	2.577164e-09
39	Taf7l	ENSMUSG00000009596	58.610232	0.944848	-6.642619	3.081587e-11	3.301343e-09