Edit this page on our live server and create a PR by running command !create-pr in the console panel

Supported Languages

There are more than 60 Jupyter kernels that supports most, if not all, scripting languages in use today. SoS Notebook supports all these kernels in the sense that

  1. You can use magic %expand to pass information from SoS to any subkernel.
  2. You can use magic %capture to capture output from these kernels.

Actually, most SoS features (e.g. line by line execution) and magics (e.g. %clear) work for all subkernels, but magics %get, %put, and some options for %use and %with only work with kernels for supported languages. This is because these magics communicate with the subkernels and exchange variables between them, and need to understand the syntax of underlying languages.

SoS defines language modules for an increasing number of languages. This chapter lists the supported languages and demonstrates how variables are exchanged among them. It is worth mentioning that some kernels do not have a concept of variable (e.g. most database kernels that only submit queries and display results), they can be used with SoS Notebook but will never support magic %get.

Bash

The Bash kernel is not rich in data types and everything is essentially a string. You can pass primitive datatypes from SoS to Bash but expect to get string representation of variables out of it.

JavaScript

SoS exchange data with a JavaScript kernel using JSON format. It passes all JSON serializable datatypes using the json module, and convert numpy arrays to list before translation (array.tolist()), and use DataFrame.to_jason to translate Pandas DataFrame to a list of records in JS.

Python condition JavaScript
None null
boolean, int, str, etc corresponding JS type
numpy.ndarray array
numpy.matrix nested array
pandas.DataFrame table with scheme and array of records (dictionary)

Translation from JavaScript to Python is easier because Python supports all JSON serializable JS datatypes.

The DataFrame translation is particularly interesting because it allows you to pass complex datatypes in Python and R for visualization. For example, for the following data.frame in R,

In [25]:
mpgcyldisphpdratwtqsecvsamgearcarb
Mazda RX421.0 6 160.0110 3.90 2.62016.460 1 4 4
Mazda RX4 Wag21.0 6 160.0110 3.90 2.87517.020 1 4 4
Datsun 71022.8 4 108.0 93 3.85 2.32018.611 1 4 1
Hornet 4 Drive21.4 6 258.0110 3.08 3.21519.441 0 3 1
Hornet Sportabout18.7 8 360.0175 3.15 3.44017.020 0 3 2
Valiant18.1 6 225.0105 2.76 3.46020.221 0 3 1
Duster 36014.3 8 360.0245 3.21 3.57015.840 0 3 4
Merc 240D24.4 4 146.7 62 3.69 3.19020.001 0 4 2
Merc 23022.8 4 140.8 95 3.92 3.15022.901 0 4 2
Merc 28019.2 6 167.6123 3.92 3.44018.301 0 4 4
Merc 280C17.8 6 167.6123 3.92 3.44018.901 0 4 4
Merc 450SE16.4 8 275.8180 3.07 4.07017.400 0 3 3
Merc 450SL17.3 8 275.8180 3.07 3.73017.600 0 3 3
Merc 450SLC15.2 8 275.8180 3.07 3.78018.000 0 3 3
Cadillac Fleetwood10.4 8 472.0205 2.93 5.25017.980 0 3 4
Lincoln Continental10.4 8 460.0215 3.00 5.42417.820 0 3 4
Chrysler Imperial14.7 8 440.0230 3.23 5.34517.420 0 3 4
Fiat 12832.4 4 78.7 66 4.08 2.20019.471 1 4 1
Honda Civic30.4 4 75.7 52 4.93 1.61518.521 1 4 2
Toyota Corolla33.9 4 71.1 65 4.22 1.83519.901 1 4 1
Toyota Corona21.5 4 120.1 97 3.70 2.46520.011 0 3 1
Dodge Challenger15.5 8 318.0150 2.76 3.52016.870 0 3 2
AMC Javelin15.2 8 304.0150 3.15 3.43517.300 0 3 2
Camaro Z2813.3 8 350.0245 3.73 3.84015.410 0 3 4
Pontiac Firebird19.2 8 400.0175 3.08 3.84517.050 0 3 2
Fiat X1-927.3 4 79.0 66 4.08 1.93518.901 1 4 1
Porsche 914-226.0 4 120.3 91 4.43 2.14016.700 1 5 2
Lotus Europa30.4 4 95.1113 3.77 1.51316.901 1 5 2
Ford Pantera L15.8 8 351.0264 4.22 3.17014.500 1 5 4
Ferrari Dino19.7 6 145.0175 3.62 2.77015.500 1 5 6
Maserati Bora15.0 8 301.0335 3.54 3.57014.600 1 5 8
Volvo 142E21.4 4 121.0109 4.11 2.78018.601 1 4 2

It appears in SoS as a pandas DataFrame

In [26]:
Out[26]:
mpg cyl disp hp drat wt qsec vs am gear carb
Mazda RX4 21.0 6.0 160.0 110.0 3.90 2.620 16.46 0.0 1.0 4.0 4.0
Mazda RX4 Wag 21.0 6.0 160.0 110.0 3.90 2.875 17.02 0.0 1.0 4.0 4.0
Datsun 710 22.8 4.0 108.0 93.0 3.85 2.320 18.61 1.0 1.0 4.0 1.0
Hornet 4 Drive 21.4 6.0 258.0 110.0 3.08 3.215 19.44 1.0 0.0 3.0 1.0
Hornet Sportabout 18.7 8.0 360.0 175.0 3.15 3.440 17.02 0.0 0.0 3.0 2.0
Valiant 18.1 6.0 225.0 105.0 2.76 3.460 20.22 1.0 0.0 3.0 1.0
Duster 360 14.3 8.0 360.0 245.0 3.21 3.570 15.84 0.0 0.0 3.0 4.0
Merc 240D 24.4 4.0 146.7 62.0 3.69 3.190 20.00 1.0 0.0 4.0 2.0
Merc 230 22.8 4.0 140.8 95.0 3.92 3.150 22.90 1.0 0.0 4.0 2.0
Merc 280 19.2 6.0 167.6 123.0 3.92 3.440 18.30 1.0 0.0 4.0 4.0
Merc 280C 17.8 6.0 167.6 123.0 3.92 3.440 18.90 1.0 0.0 4.0 4.0
Merc 450SE 16.4 8.0 275.8 180.0 3.07 4.070 17.40 0.0 0.0 3.0 3.0
Merc 450SL 17.3 8.0 275.8 180.0 3.07 3.730 17.60 0.0 0.0 3.0 3.0
Merc 450SLC 15.2 8.0 275.8 180.0 3.07 3.780 18.00 0.0 0.0 3.0 3.0
Cadillac Fleetwood 10.4 8.0 472.0 205.0 2.93 5.250 17.98 0.0 0.0 3.0 4.0
Lincoln Continental 10.4 8.0 460.0 215.0 3.00 5.424 17.82 0.0 0.0 3.0 4.0
Chrysler Imperial 14.7 8.0 440.0 230.0 3.23 5.345 17.42 0.0 0.0 3.0 4.0
Fiat 128 32.4 4.0 78.7 66.0 4.08 2.200 19.47 1.0 1.0 4.0 1.0
Honda Civic 30.4 4.0 75.7 52.0 4.93 1.615 18.52 1.0 1.0 4.0 2.0
Toyota Corolla 33.9 4.0 71.1 65.0 4.22 1.835 19.90 1.0 1.0 4.0 1.0
Toyota Corona 21.5 4.0 120.1 97.0 3.70 2.465 20.01 1.0 0.0 3.0 1.0
Dodge Challenger 15.5 8.0 318.0 150.0 2.76 3.520 16.87 0.0 0.0 3.0 2.0
AMC Javelin 15.2 8.0 304.0 150.0 3.15 3.435 17.30 0.0 0.0 3.0 2.0
Camaro Z28 13.3 8.0 350.0 245.0 3.73 3.840 15.41 0.0 0.0 3.0 4.0
Pontiac Firebird 19.2 8.0 400.0 175.0 3.08 3.845 17.05 0.0 0.0 3.0 2.0
Fiat X1-9 27.3 4.0 79.0 66.0 4.08 1.935 18.90 1.0 1.0 4.0 1.0
Porsche 914-2 26.0 4.0 120.3 91.0 4.43 2.140 16.70 0.0 1.0 5.0 2.0
Lotus Europa 30.4 4.0 95.1 113.0 3.77 1.513 16.90 1.0 1.0 5.0 2.0
Ford Pantera L 15.8 8.0 351.0 264.0 4.22 3.170 14.50 0.0 1.0 5.0 4.0
Ferrari Dino 19.7 6.0 145.0 175.0 3.62 2.770 15.50 0.0 1.0 5.0 6.0
Maserati Bora 15.0 8.0 301.0 335.0 3.54 3.570 14.60 0.0 1.0 5.0 8.0
Volvo 142E 21.4 4.0 121.0 109.0 4.11 2.780 18.60 1.0 1.0 4.0 2.0

we can get the data from the JS script kernel as follows:

In [27]:
Out[27]:
{ mpg: 21,
  cyl: 6,
  disp: 160,
  hp: 110,
  drat: 3.9,
  wt: 2.62,
  qsec: 16.46,
  vs: 0,
  am: 1,
  gear: 4,
  carb: 4 }

Julia

Julia's data types are pretty close to Python and there is almost a one-to-one correspondence between Julia and Python types.

When transferring data from Python (SoS) to Julia (e.g. %get from Julia), the resulting Julia types are list as follows:

Python condition Julia
None NaN
boolean Bool
integer Int64
numpy.float64 Float64
float Float64
complex Complex{}
str String
Sequence (list, tuple, ...) homogenous type, all integer Array{Int64,1}
Sequence (list, tuple, ...) homogenous type, all numeric with any float Array{Float64,1}
Sequence (list, tuple, ...) homogenous type, all str Array{String,1}
Sequence (list, tuple, ...) multiple types Array{Any,1}
set homogenous type, all integer Set{Int64}
set homogenous type, all numeric with any float Set{Float64}
set homogenous type, all str Set{String}
set multiple types Set{Any}
dict Dict{}
numpy.ndarray Array{}
numpy.matrix Array{}
pandas.Series NamedArrays.NamedArray{}
pandas.DataFrame Dataframes.Dataframe{}

Python objects in other datatypes are transferred as string "Unsupported datatype".

It is worth noting that although Julia supports Char (single character) and String types, SoS always transfer Python str to Julia String, so if you have a Python string X,

In [1]:
In [2]:
Out[2]:
"X"
In [3]:
Out[3]:
String

If you need to access the first character as a Char, just use

In [4]:
Out[4]:
88

Conversion of datatypes from Julia to SoS (%get var --from Julia from SoS) follows the following rules:

Julia condition Python
NaN None
Bool boolean
Int64 integer
Char str
Complex{} complex
Float64 numpy.float64
String str
Array{,1} Sequence (list,tuple, ...)
Set{} set
Dict{} dict
Array{Int64/Float64,2} 2 dimensions numpy.matrix
Array{Int64/Float64,} > 2 dimensions numpy.ndarray
NamedArrays.NamedArray 1 dimension pandas.Series
Dataframes.Dataframe pandas.DataFrame

Julia objects in other datatypes are transferred as string "Unsupported datatype".

Julia supports DataFrame from its DataFrames package so you will need to install this package before using the Julia kernel. For example, a R dataframe is transfered as Dataframes.Dataframe to Julia.

In [5]:
Loading required package: feather
Raw index is ignored because Julia DataFrame does not support raw index.
Out[5]:
mpgcyldisphpdratwtqsecvsamgearcarb
121.06.0160.0110.03.92.6216.460.01.04.04.0
221.06.0160.0110.03.92.87517.020.01.04.04.0
322.84.0108.093.03.852.3218.611.01.04.01.0
421.46.0258.0110.03.083.21519.441.00.03.01.0
518.78.0360.0175.03.153.4417.020.00.03.02.0
618.16.0225.0105.02.763.4620.221.00.03.01.0
714.38.0360.0245.03.213.5715.840.00.03.04.0
824.44.0146.762.03.693.1920.01.00.04.02.0
922.84.0140.895.03.923.1522.91.00.04.02.0
1019.26.0167.6123.03.923.4418.31.00.04.04.0
1117.86.0167.6123.03.923.4418.91.00.04.04.0
1216.48.0275.8180.03.074.0717.40.00.03.03.0
1317.38.0275.8180.03.073.7317.60.00.03.03.0
1415.28.0275.8180.03.073.7818.00.00.03.03.0
1510.48.0472.0205.02.935.2517.980.00.03.04.0
1610.48.0460.0215.03.05.42417.820.00.03.04.0
1714.78.0440.0230.03.235.34517.420.00.03.04.0
1832.44.078.766.04.082.219.471.01.04.01.0
1930.44.075.752.04.931.61518.521.01.04.02.0
2033.94.071.165.04.221.83519.91.01.04.01.0
2121.54.0120.197.03.72.46520.011.00.03.01.0
2215.58.0318.0150.02.763.5216.870.00.03.02.0
2315.28.0304.0150.03.153.43517.30.00.03.02.0
2413.38.0350.0245.03.733.8415.410.00.03.04.0
2519.28.0400.0175.03.083.84517.050.00.03.02.0
2627.34.079.066.04.081.93518.91.01.04.01.0
2726.04.0120.391.04.432.1416.70.01.05.02.0
2830.44.095.1113.03.771.51316.91.01.05.02.0
2915.88.0351.0264.04.223.1714.50.01.05.04.0
3019.76.0145.0175.03.622.7715.50.01.05.06.0

As you can see from the warning, row labels of the dataframe is not transferred because Julia's dataframe does not support row label. If you really need the row labels, you would have to assign row labels to a separate variable and transfer.

In [6]:
Out[6]:
32-element Array{Float64,1}:
 21.0
 21.0
 22.8
 21.4
 18.7
 18.1
 14.3
 24.4
 22.8
 19.2
 17.8
 16.4
 17.3
  ⋮  
 21.5
 15.5
 15.2
 13.3
 19.2
 27.3
 26.0
 30.4
 15.8
 19.7
 15.0
 21.4
In [7]:

When an Julia Array is transferred to SoS (Python3), it becomes a numpy.array.

In [9]:
Out[9]:
array([ 21. ,  21. ,  22.8,  21.4,  18.7,  18.1,  14.3,  24.4,  22.8,
        19.2,  17.8,  16.4,  17.3,  15.2,  10.4,  10.4,  14.7,  32.4,
        30.4,  33.9,  21.5,  15.5,  15.2,  13.3,  19.2,  27.3,  26. ,
        30.4,  15.8,  19.7,  15. ,  21.4])

MATLAB (Octave)

SoS tries to exchange variables with MATLAB/Octave in the most natural way. For example it converts numpy arrays to MATLAB matrix, and dictionaries to MATLAB struct. It converts Python DataFrame (or R data.frame) to table in MATLAB but to dataframe in Octave because the latter is closer to Python DataFrame.

The convertion of datatype from SoS to MATLAB (Octave) is as followings:

Python condition MATLAB/Octave
None NaN
boolean logical
integer integer
float float
complex complex
str str
char char
Sequence (list, tuple, ...) homogenous type, all numeric vector
Sequence (list, tuple, ...) homogenous type, all char char
Sequence (list, tuple, ...) multiple types cell
set cell
dict struct
numpy.ndarray cell
numpy.matrix matrix
pandas.DataFrame table (MATLAB)
dataframe (Octave)

Python objects in other datatypes are transferred as string "Unsupported datatype".

The convertion of datatype from MATLAB (Octave) to SoS is as followings:

MATLAB length (n) Python
NaN None
logical 1 boolean
integer 1 integer
numeric 1 double
character 1 string
complex 1 complex
logical n > 1 list
integer n > 1 list
complex n > 1 list
numeric n > 1 list
character n > 1 list
struct n > 0 numpy.ndarray
matrix n > 0 numpy.array
cell n > 0 numpy.ndarray
table n > 0 pandas.DataFrame

For example, a R dataframe is transfered as table to MATLAB.

In [3]:
mtcars =

  32�11 table

    mpg     cyl    disp     hp     drat     wt      qsec     vs    am    gear    carb
    ____    ___    _____    ___    ____    _____    _____    __    __    ____    ____

      21    6        160    110     3.9     2.62    16.46    0     1     4       4   
      21    6        160    110     3.9    2.875    17.02    0     1     4       4   
    22.8    4        108     93    3.85     2.32    18.61    1     1     4       1   
    21.4    6        258    110    3.08    3.215    19.44    1     0     3       1   
    18.7    8        360    175    3.15     3.44    17.02    0     0     3       2   
    18.1    6        225    105    2.76     3.46    20.22    1     0     3       1   
    14.3    8        360    245    3.21     3.57    15.84    0     0     3       4   
    24.4    4      146.7     62    3.69     3.19       20    1     0     4       2   
    22.8    4      140.8     95    3.92     3.15     22.9    1     0     4       2   
    19.2    6      167.6    123    3.92     3.44     18.3    1     0     4       4   
    17.8    6      167.6    123    3.92     3.44     18.9    1     0     4       4   
    16.4    8      275.8    180    3.07     4.07     17.4    0     0     3       3   
    17.3    8      275.8    180    3.07     3.73     17.6    0     0     3       3   
    15.2    8      275.8    180    3.07     3.78       18    0     0     3       3   
    10.4    8        472    205    2.93     5.25    17.98    0     0     3       4   
    10.4    8        460    215       3    5.424    17.82    0     0     3       4   
    14.7    8        440    230    3.23    5.345    17.42    0     0     3       4   
    32.4    4       78.7     66    4.08      2.2    19.47    1     1     4       1   
    30.4    4       75.7     52    4.93    1.615    18.52    1     1     4       2   
    33.9    4       71.1     65    4.22    1.835     19.9    1     1     4       1   
    21.5    4      120.1     97     3.7    2.465    20.01    1     0     3       1   
    15.5    8        318    150    2.76     3.52    16.87    0     0     3       2   
    15.2    8        304    150    3.15    3.435     17.3    0     0     3       2   
    13.3    8        350    245    3.73     3.84    15.41    0     0     3       4   
    19.2    8        400    175    3.08    3.845    17.05    0     0     3       2   
    27.3    4         79     66    4.08    1.935     18.9    1     1     4       1   
      26    4      120.3     91    4.43     2.14     16.7    0     1     5       2   
    30.4    4       95.1    113    3.77    1.513     16.9    1     1     5       2   
    15.8    8        351    264    4.22     3.17     14.5    0     1     5       4   
    19.7    6        145    175    3.62     2.77     15.5    0     1     5       6   
      15    8        301    335    3.54     3.57     14.6    0     1     5       8   
    21.4    4        121    109    4.11     2.78     18.6    1     1     4       2   

Python 2 and Python 3

SoS uses pickle to exchange data between SoS (Python3) and Python2 and Python3 kernels so any pickleable data types can be exchanged.

R

Because there is no one to one correspondence of data types between Python and R datatypes, SoS tries to translate variables in the most natural way. For example, although 3 and [1, 4] are both numeric type in R (the former have length 1), they are translated to Python variables 3 (an integer) and [1, 4] (a list).

SoS %put variables in SoS to R as follows:

Python condition R
None NULL
boolean logical
integer integer
float numeric
complex complex
str character
Sequence (list, tuple, ...) homogenous type c()
Sequence (list, tuple, ...) multiple types list
set list
dict list with names
numpy.ndarray array
numpy.matrix matrix
pandas.DataFrame R data.frame

Python objects in other datatypes are transferred as string "Unsupported datatype".

SoS %get data from R as follows:

R length (n) Python
NULL None
logical 1 boolean
integer 1 integer
numeric 1 double
character 1 string
complex 1 complex
logical n > 1 list
integer n > 1 list
complex n > 1 list
numeric n > 1 list
character n > 1 list
list without names n > 0 list
list with names n > 0 dict (with ordered keys)
matrix n > 0 numpy.array
data.frame n > 0 DataFrame
array n >= 0 numpy.array

For example, the scalar data is converted from SoS to R as follows:

In [1]:
In [2]:
> null_var:
NULL
> num_var:
123
> logic_var:
TRUE
> char_var:
'1"23'
> comp_var:
1+2i

The one-dimension (vector) data is converted from SoS to R as follows:

In [3]:
In [4]:
> char_arr_var:
  1. '1'
  2. '2'
  3. '3'
> list_var:
  1. 1
  2. 2
  3. '3'
> dict_var:
$a
1
$b
2
$c
'3'
> set_var:
  1. 1
  2. 2
  3. '3'
> recursive_var:
$a
$b = 123
$c
TRUE
> logic_arr_var:
  1. TRUE
  2. FALSE
  3. TRUE
> seri_var:
0
1
1
2
2
3
3
3
4
3
5
3

The multi-dimension data is converted from SoS to R as follows:

In [5]:
In [6]:
> num_arr_var:
12
34
> mat_var:
01
12
34

The scalar data is converted from R to SoS as follows:

In [7]:
In [8]:
> null_var: NoneType
None
> num_var: int
123
> logic_var: bool
True
> char_var: str of length 4
'1"23'
> comp_var: complex
(1+2j)

The one-dimension (vector) data is converted from R to SoS as follows:

In [9]:
In [10]:
> num_vector_var: list of length 3
[1, 2, 3]
> logic_vector_var: list of length 3
[True, False, True]
> char_vector_var: list of length 3
['1', '2', '3']
> list_var: list of length 3
[1, 2, '3']
> named_list_var: dict of length 3
{'a': 1, 'b': 2, 'c': '3'}
> recursive_var: dict of length 2
{'a': 1, 'b': {'c': 3, 'd': 'whatever'}}
> seri_var: Series of shape (6,)
0    1
1    2
2    3
3    3
4    3
5    3
dtype: int64

The multi-dimension data is converted from R to SoS as follows:

In [11]:
In [12]:
> mat_var: ndarray of shape (2, 2)
array([[ 1.,  3.],
       [ 2.,  4.]])
> arr_var: ndarray of shape (2, 2, 2, 2)
array([[[[ 1,  3],
         [ 2,  4]],

        [[ 5,  7],
         [ 6,  8]]],


       [[[ 9, 11],
         [10, 12]],

        [[13, 15],
         [14, 16]]]])

It is worth noting that R's named list is transferred to Python as dictionaries but SoS preserves the order of the keys so that you can recover the order of the list. For example,

In [13]:

Although the dictionary might appear to have different order

In [14]:
Out[14]:
{'A': 1, 'B': 3, 'C': 'C', 'D': [2, 3]}

The order of the keys and values are actually preserved

In [15]:
Out[15]:
dict_keys(['A', 'C', 'B', 'D'])
In [16]:
Out[16]:
dict_values([1, 'C', 3, [2, 3]])

so it is safe to enumerate the R list in Python as

In [17]:
1 item of Rlist has key A and value 1
2 item of Rlist has key C and value C
3 item of Rlist has key B and value 3
4 item of Rlist has key D and value [2, 3]

Note that SoS uses the feather modules in Python and R to exchange Python Pandas Dataframe and R data.frame so this module is required if you would like to exchange data frames between these two languages.

Ruby

Basic data types recognised in Ruby are similar with Python's data types and there is a one-to-one correspondence for these types.

The convertion of datatype from SoS to Ruby (e.g. %get from Ruby) is as followings:

Python condition Ruby
None nil
boolean TrueClass or FalseClass
integer Integer
float Float
complex Complex
str String
Sequence (list, tuple, ...) Array
set Set
dict Hash
range Range
numpy.ndarray Array
numpy.matrix NMatrix
pandas.Series Hash
pandas.DataFrame Daru::DataFrame

Python objects in other datatypes are transferred as string "Unsupported datatype". Please let us know if there is a natural corresponding data type in Ruby to convert this data type.

Conversion of datatypes from Ruby to SoS (%get var --from Ruby from SoS) follows the following rules:

Ruby condition Python
nil None
Float::NAN numpy.nan
TrueClass or FalseClass boolean
Integer integer
String str
Complex complex
Float float
Array numpy.ndarray
Range range
Set set
Hash dict
NMatrix numpy.matrix
Array numpy.ndarray
Daru::DataFrame pandas.DataFrame

Ruby objects in other datatypes are transferred as string "Unsupported datatype".

For example, the scalar data is converted from SoS to Ruby as follows:

In [1]:
In [2]:
[nil, 123, true, "1\"23", (1.0+2.0i)]

Ruby supports DataFrame from its daru (Data Analysis in RUby) library so you will need to install this library before using the Ruby kernel. For example, a R dataframe is transfered as Daru::DataFrame to Ruby.

In [3]:
Loading required package: feather
Out[3]:
Daru::DataFrame(32x11)
mpg cyl disp hp drat wt qsec vs am gear carb
Mazda RX4 21.0 6.0 160.0 110.0 3.9 2.62 16.46 0.0 1.0 4.0 4.0
Mazda RX4 Wag 21.0 6.0 160.0 110.0 3.9 2.875 17.02 0.0 1.0 4.0 4.0
Datsun 710 22.8 4.0 108.0 93.0 3.85 2.32 18.61 1.0 1.0 4.0 1.0
Hornet 4 Drive 21.4 6.0 258.0 110.0 3.08 3.215 19.44 1.0 0.0 3.0 1.0
Hornet Sportabout 18.7 8.0 360.0 175.0 3.15 3.44 17.02 0.0 0.0 3.0 2.0
Valiant 18.1 6.0 225.0 105.0 2.76 3.46 20.22 1.0 0.0 3.0 1.0
Duster 360 14.3 8.0 360.0 245.0 3.21 3.57 15.84 0.0 0.0 3.0 4.0
Merc 240D 24.4 4.0 146.7 62.0 3.69 3.19 20.0 1.0 0.0 4.0 2.0
Merc 230 22.8 4.0 140.8 95.0 3.92 3.15 22.9 1.0 0.0 4.0 2.0
Merc 280 19.2 6.0 167.6 123.0 3.92 3.44 18.3 1.0 0.0 4.0 4.0
Merc 280C 17.8 6.0 167.6 123.0 3.92 3.44 18.9 1.0 0.0 4.0 4.0
Merc 450SE 16.4 8.0 275.8 180.0 3.07 4.07 17.4 0.0 0.0 3.0 3.0
Merc 450SL 17.3 8.0 275.8 180.0 3.07 3.73 17.6 0.0 0.0 3.0 3.0
Merc 450SLC 15.2 8.0 275.8 180.0 3.07 3.78 18.0 0.0 0.0 3.0 3.0
Cadillac Fleetwood 10.4 8.0 472.0 205.0 2.93 5.25 17.98 0.0 0.0 3.0 4.0
Lincoln Continental 10.4 8.0 460.0 215.0 3.0 5.424 17.82 0.0 0.0 3.0 4.0
Chrysler Imperial 14.7 8.0 440.0 230.0 3.23 5.345 17.42 0.0 0.0 3.0 4.0
Fiat 128 32.4 4.0 78.7 66.0 4.08 2.2 19.47 1.0 1.0 4.0 1.0
Honda Civic 30.4 4.0 75.7 52.0 4.93 1.615 18.52 1.0 1.0 4.0 2.0
Toyota Corolla 33.9 4.0 71.1 65.0 4.22 1.835 19.9 1.0 1.0 4.0 1.0
Toyota Corona 21.5 4.0 120.1 97.0 3.7 2.465 20.01 1.0 0.0 3.0 1.0
Dodge Challenger 15.5 8.0 318.0 150.0 2.76 3.52 16.87 0.0 0.0 3.0 2.0
AMC Javelin 15.2 8.0 304.0 150.0 3.15 3.435 17.3 0.0 0.0 3.0 2.0
Camaro Z28 13.3 8.0 350.0 245.0 3.73 3.84 15.41 0.0 0.0 3.0 4.0
Pontiac Firebird 19.2 8.0 400.0 175.0 3.08 3.845 17.05 0.0 0.0 3.0 2.0
Fiat X1-9 27.3 4.0 79.0 66.0 4.08 1.935 18.9 1.0 1.0 4.0 1.0
Porsche 914-2 26.0 4.0 120.3 91.0 4.43 2.14 16.7 0.0 1.0 5.0 2.0
Lotus Europa 30.4 4.0 95.1 113.0 3.77 1.513 16.9 1.0 1.0 5.0 2.0
Ford Pantera L 15.8 8.0 351.0 264.0 4.22 3.17 14.5 0.0 1.0 5.0 4.0
Ferrari Dino 19.7 6.0 145.0 175.0 3.62 2.77 15.5 0.0 1.0 5.0 6.0
... ... ... ... ... ... ... ... ... ... ... ...
Volvo 142E 21.4 4.0 121.0 109.0 4.11 2.78 18.6 1.0 1.0 4.0 2.0

Also, we choose NMatrix library in Ruby becuase its fast performance. Same as daru (Data Analysis in RUby), you will need to install nmatrix library before using the Ruby kernel.

In [4]:
Out[4]:
$$\left(\begin{array}{ccc} 2&3&4\\ 7&8&9\\ \end{array}\right)$$
In [5]:
'numpy.matrix([[2, 3, 4], [7, 8, 9]])'
In [6]:
Out[6]:
matrix([[2, 3, 4],
        [7, 8, 9]])

SAS

A SAS dataset is roughly equivalent to Python's DataFrame (Pandas) and R's dataframe, and this is the only datatype that can be exchanged between SAS and other languages.

For example, you can get a R dataframe into SAS as follows:

In [1]:
Loading required package: feather
Out[1]:
SAS Output
The SAS System

Obs mpg cyl disp hp drat wt qsec vs am gear carb
1 21.0 6 160.0 110 3.90 2.620 16.46 0 1 4 4
2 21.0 6 160.0 110 3.90 2.875 17.02 0 1 4 4
3 22.8 4 108.0 93 3.85 2.320 18.61 1 1 4 1
4 21.4 6 258.0 110 3.08 3.215 19.44 1 0 3 1
5 18.7 8 360.0 175 3.15 3.440 17.02 0 0 3 2
6 18.1 6 225.0 105 2.76 3.460 20.22 1 0 3 1
7 14.3 8 360.0 245 3.21 3.570 15.84 0 0 3 4
8 24.4 4 146.7 62 3.69 3.190 20.00 1 0 4 2
9 22.8 4 140.8 95 3.92 3.150 22.90 1 0 4 2
10 19.2 6 167.6 123 3.92 3.440 18.30 1 0 4 4
11 17.8 6 167.6 123 3.92 3.440 18.90 1 0 4 4
12 16.4 8 275.8 180 3.07 4.070 17.40 0 0 3 3
13 17.3 8 275.8 180 3.07 3.730 17.60 0 0 3 3
14 15.2 8 275.8 180 3.07 3.780 18.00 0 0 3 3
15 10.4 8 472.0 205 2.93 5.250 17.98 0 0 3 4
16 10.4 8 460.0 215 3.00 5.424 17.82 0 0 3 4
17 14.7 8 440.0 230 3.23 5.345 17.42 0 0 3 4
18 32.4 4 78.7 66 4.08 2.200 19.47 1 1 4 1
19 30.4 4 75.7 52 4.93 1.615 18.52 1 1 4 2
20 33.9 4 71.1 65 4.22 1.835 19.90 1 1 4 1
21 21.5 4 120.1 97 3.70 2.465 20.01 1 0 3 1
22 15.5 8 318.0 150 2.76 3.520 16.87 0 0 3 2
23 15.2 8 304.0 150 3.15 3.435 17.30 0 0 3 2
24 13.3 8 350.0 245 3.73 3.840 15.41 0 0 3 4
25 19.2 8 400.0 175 3.08 3.845 17.05 0 0 3 2
26 27.3 4 79.0 66 4.08 1.935 18.90 1 1 4 1
27 26.0 4 120.3 91 4.43 2.140 16.70 0 1 5 2
28 30.4 4 95.1 113 3.77 1.513 16.90 1 1 5 2
29 15.8 8 351.0 264 4.22 3.170 14.50 0 1 5 4
30 19.7 6 145.0 175 3.62 2.770 15.50 0 1 5 6
31 15.0 8 301.0 335 3.54 3.570 14.60 0 1 5 8
32 21.4 4 121.0 109 4.11 2.780 18.60 1 1 4 2

And if you have a SAS dataset,

In [2]:
Out[2]:
SAS Output
The SAS System

Obs NAME SEX AGE HEIGHT WEIGHT
1 JOHN M 12 59 99.5
2 JAMES M 12 57 83.0
3 ALFRED M 14 69 112.5
4 ALICE F 13 56 84.0

You can transfer the dataset to Python or R and analyze there

In [3]:
%preview CLASS
> CLASS: DataFrame of shape (4, 5)
  NAME   SEX   AGE   HEIGHT   WEIGHT  
0 JOHN M 12.0 59.0 99.5
1 JAMES M 12.0 57.0 83.0
2 ALFRED M 14.0 69.0 112.5
3 ALICE F 13.0 56.0 84.0

Stata

Stata, similar to SAS, has one and only one unnamed current dataset in memory. It also supports named, untyped macros. SoS supports the exchange of variables between Stata and other kernels as follows:

When pushing data to Stata

Python variable name Stata
int, float, str variable name become macro name local macro (string)
pandas.DataFrame unnamed in Stata Stata Dataset

When getting data from Stata

Stata variable name Type in Python
local macros name of macro str
global macros name of macro str
Dataset user specified pandas.DataFrame

Note that non-dataframe variables are transferred to Stata as local macros, and both local and global macros can be transferred although local macros will be checked first.

For example, if you have a directory and filenames defined in SoS

In [1]:

You can use the variables instantly in Stata with SoS' own %expand magic,

In [2]:
/Users/bpeng1/Downloads/StataIntro/dataSets


marital  age      educ     sex      inc      happy    region

However, if you prefer, you can transferred the variables as local macros in Stata

In [3]:
S_FNDATE:       11 Sep 2009 09:55
S_FN:           gss.dta
stata_kernel_graph_counter:
                0
S_ADO:          BASE;SITE;.;PERSONAL;PLUS;OLDPLACE;`"/Users/bpeng1/anaconda3/envs/sos/lib/python3.6/site-packages/stata_kernel/ado"'
S_level:        95
F1:             help advice;
F2:             describe;
F7:             save
F8:             use
S_StataMP:      MP
S_StataSE:      SE
S_CONSOLE:      console
S_FLAVOR:       Intercooled
S_OS:           Unix
S_MACH:         Macintosh (Intel 64-bit)
_filename:      gss.dta
_datadir:       ~/Downloads/StataIntro/dataSets

and use the macros in Stata

In [4]:
/Users/bpeng1/Downloads/StataIntro/dataSets


marital  age      educ     sex      inc      happy    region

Now if you have a dataframe, for example, mtcard in R, getting them in Stata will replace the current dataset, and the name mtcars no longer matters.

In [5]:
Loading required package: feather
    Variable |        Obs        Mean    Std. Dev.       Min        Max
-------------+---------------------------------------------------------
       index |          0
         mpg |         32    20.09062    6.026948       10.4       33.9
         cyl |         32      6.1875    1.785922          4          8
        disp |         32    230.7219    123.9387       71.1        472
          hp |         32    146.6875    68.56287         52        335
-------------+---------------------------------------------------------
        drat |         32    3.596563    .5346787       2.76       4.93
          wt |         32     3.21725    .9784574      1.513      5.424
        qsec |         32    17.84875    1.786943       14.5       22.9
          vs |         32       .4375    .5040161          0          1
          am |         32      .40625    .4989909          0          1
-------------+---------------------------------------------------------
        gear |         32      3.6875    .7378041          3          5
        carb |         32      2.8125      1.6152          1          8
In [6]:
(bin=5, start=52, width=56.6)

Sending macros and/or dataset to another kernel is a bit tricky because local and global macros can have the same name, and the dataset is unnamed. The rule is that

  1. If the name represents a local macro, send it as string
  2. Otherwise if the name represents a global macro, send it as string
  3. Otherwise send the current dataset as a DataFrame using the provided name

For example, because mydata is not one of the macros, it will become the name of the DataFrame representing the Stata dataset:

In [7]:
Out[7]:
index mpg cyl disp hp drat wt qsec vs am gear carb
0 Mazda RX4 21.0 6.0 160.0 110.0 3.90 2.620 16.46 0.0 1.0 4.0 4.0
1 Mazda RX4 Wag 21.0 6.0 160.0 110.0 3.90 2.875 17.02 0.0 1.0 4.0 4.0
2 Datsun 710 22.8 4.0 108.0 93.0 3.85 2.320 18.61 1.0 1.0 4.0 1.0
3 Hornet 4 Drive 21.4 6.0 258.0 110.0 3.08 3.215 19.44 1.0 0.0 3.0 1.0
4 Hornet Sportabout 18.7 8.0 360.0 175.0 3.15 3.440 17.02 0.0 0.0 3.0 2.0
5 Valiant 18.1 6.0 225.0 105.0 2.76 3.460 20.22 1.0 0.0 3.0 1.0
6 Duster 360 14.3 8.0 360.0 245.0 3.21 3.570 15.84 0.0 0.0 3.0 4.0
7 Merc 240D 24.4 4.0 146.7 62.0 3.69 3.190 20.00 1.0 0.0 4.0 2.0
8 Merc 230 22.8 4.0 140.8 95.0 3.92 3.150 22.90 1.0 0.0 4.0 2.0
9 Merc 280 19.2 6.0 167.6 123.0 3.92 3.440 18.30 1.0 0.0 4.0 4.0
10 Merc 280C 17.8 6.0 167.6 123.0 3.92 3.440 18.90 1.0 0.0 4.0 4.0
11 Merc 450SE 16.4 8.0 275.8 180.0 3.07 4.070 17.40 0.0 0.0 3.0 3.0
12 Merc 450SL 17.3 8.0 275.8 180.0 3.07 3.730 17.60 0.0 0.0 3.0 3.0
13 Merc 450SLC 15.2 8.0 275.8 180.0 3.07 3.780 18.00 0.0 0.0 3.0 3.0
14 Cadillac Fleetwood 10.4 8.0 472.0 205.0 2.93 5.250 17.98 0.0 0.0 3.0 4.0
15 Lincoln Continental 10.4 8.0 460.0 215.0 3.00 5.424 17.82 0.0 0.0 3.0 4.0
16 Chrysler Imperial 14.7 8.0 440.0 230.0 3.23 5.345 17.42 0.0 0.0 3.0 4.0
17 Fiat 128 32.4 4.0 78.7 66.0 4.08 2.200 19.47 1.0 1.0 4.0 1.0
18 Honda Civic 30.4 4.0 75.7 52.0 4.93 1.615 18.52 1.0 1.0 4.0 2.0
19 Toyota Corolla 33.9 4.0 71.1 65.0 4.22 1.835 19.90 1.0 1.0 4.0 1.0
20 Toyota Corona 21.5 4.0 120.1 97.0 3.70 2.465 20.01 1.0 0.0 3.0 1.0
21 Dodge Challenger 15.5 8.0 318.0 150.0 2.76 3.520 16.87 0.0 0.0 3.0 2.0
22 AMC Javelin 15.2 8.0 304.0 150.0 3.15 3.435 17.30 0.0 0.0 3.0 2.0
23 Camaro Z28 13.3 8.0 350.0 245.0 3.73 3.840 15.41 0.0 0.0 3.0 4.0
24 Pontiac Firebird 19.2 8.0 400.0 175.0 3.08 3.845 17.05 0.0 0.0 3.0 2.0
25 Fiat X1-9 27.3 4.0 79.0 66.0 4.08 1.935 18.90 1.0 1.0 4.0 1.0
26 Porsche 914-2 26.0 4.0 120.3 91.0 4.43 2.140 16.70 0.0 1.0 5.0 2.0
27 Lotus Europa 30.4 4.0 95.1 113.0 3.77 1.513 16.90 1.0 1.0 5.0 2.0
28 Ford Pantera L 15.8 8.0 351.0 264.0 4.22 3.170 14.50 0.0 1.0 5.0 4.0
29 Ferrari Dino 19.7 6.0 145.0 175.0 3.62 2.770 15.50 0.0 1.0 5.0 6.0
30 Maserati Bora 15.0 8.0 301.0 335.0 3.54 3.570 14.60 0.0 1.0 5.0 8.0
31 Volvo 142E 21.4 4.0 121.0 109.0 4.11 2.780 18.60 1.0 1.0 4.0 2.0

The macros can be obtained

In [8]:
Out[8]:
'BASE;SITE;.;PERSONAL;PLUS;OLDPLACE;`"/Users/bpeng1/anaconda3/envs/sos/lib/python3.6/site-packages/stata_kernel/ado"\''

and Stata supports the automatic transfer of variables (macros) with names starting with sos. For example,

In [9]:

but remember macros are untyped so they will be transferred as strings:

In [10]:
Out[10]:
'3.14159'

The %sessioninfo magic lists the version of Stata when the stata kernel is used:

In [11]:

SoS

SoS Version
0.17.7

Stata

Kernel
stata
Language
Stata
version 15.1

R

Kernel
ir
Language
R
R version 3.5.0 (2018-04-23)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: macOS High Sierra 10.13.6

Matrix products: default
BLAS/LAPACK: /Users/bpeng1/anaconda3/envs/sos/lib/libopenblasp-r0.2.20.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] feather_0.3.1

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.16    digest_0.6.15   crayon_1.3.4    IRdisplay_0.4.4
 [5] repr_0.13       jsonlite_1.5    magrittr_1.5    evaluate_0.10.1
 [9] pillar_1.2.1    rlang_0.2.0     stringi_1.1.7   uuid_0.1-2     
[13] IRkernel_0.8.11 tools_3.5.0     stringr_1.3.0   hms_0.4.2      
[17] compiler_3.5.0  pkgconfig_2.0.1 base64enc_0.1-3 htmltools_0.3.6
[21] pbdZMQ_0.3-2    tibble_1.4.2   

Python3

Kernel
python3
Language
Python3
Version
3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 11:07:29) 
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)]

TypeScript

SoS exchange data with a JypeScript kernel using JSON format. It passes all JSON serializable datatypes using the json module, and convert numpy arrays to list before translation (array.tolist()), and use DataFrame.to_jason to translate Pandas DataFrame to a list of records in JS.

Python condition TypeScript
None null
boolean, int, str, etc corresponding JS type
numpy.ndarray array
numpy.matrix nested array
pandas.DataFrame table with scheme and array of records (dictionary)

Translation from TypeScript to Python is easier because Python supports all JSON serializable TS datatypes.

The DataFrame translation is particularly interesting because it allows you to pass complex datatypes in Python and R for visualization. For example, for the following data.frame in R,

In [1]:
mpgcyldisphpdratwtqsecvsamgearcarb
Mazda RX421.0 6 160.0110 3.90 2.62016.460 1 4 4
Mazda RX4 Wag21.0 6 160.0110 3.90 2.87517.020 1 4 4
Datsun 71022.8 4 108.0 93 3.85 2.32018.611 1 4 1
Hornet 4 Drive21.4 6 258.0110 3.08 3.21519.441 0 3 1
Hornet Sportabout18.7 8 360.0175 3.15 3.44017.020 0 3 2
Valiant18.1 6 225.0105 2.76 3.46020.221 0 3 1
Duster 36014.3 8 360.0245 3.21 3.57015.840 0 3 4
Merc 240D24.4 4 146.7 62 3.69 3.19020.001 0 4 2
Merc 23022.8 4 140.8 95 3.92 3.15022.901 0 4 2
Merc 28019.2 6 167.6123 3.92 3.44018.301 0 4 4
Merc 280C17.8 6 167.6123 3.92 3.44018.901 0 4 4
Merc 450SE16.4 8 275.8180 3.07 4.07017.400 0 3 3
Merc 450SL17.3 8 275.8180 3.07 3.73017.600 0 3 3
Merc 450SLC15.2 8 275.8180 3.07 3.78018.000 0 3 3
Cadillac Fleetwood10.4 8 472.0205 2.93 5.25017.980 0 3 4
Lincoln Continental10.4 8 460.0215 3.00 5.42417.820 0 3 4
Chrysler Imperial14.7 8 440.0230 3.23 5.34517.420 0 3 4
Fiat 12832.4 4 78.7 66 4.08 2.20019.471 1 4 1
Honda Civic30.4 4 75.7 52 4.93 1.61518.521 1 4 2
Toyota Corolla33.9 4 71.1 65 4.22 1.83519.901 1 4 1
Toyota Corona21.5 4 120.1 97 3.70 2.46520.011 0 3 1
Dodge Challenger15.5 8 318.0150 2.76 3.52016.870 0 3 2
AMC Javelin15.2 8 304.0150 3.15 3.43517.300 0 3 2
Camaro Z2813.3 8 350.0245 3.73 3.84015.410 0 3 4
Pontiac Firebird19.2 8 400.0175 3.08 3.84517.050 0 3 2
Fiat X1-927.3 4 79.0 66 4.08 1.93518.901 1 4 1
Porsche 914-226.0 4 120.3 91 4.43 2.14016.700 1 5 2
Lotus Europa30.4 4 95.1113 3.77 1.51316.901 1 5 2
Ford Pantera L15.8 8 351.0264 4.22 3.17014.500 1 5 4
Ferrari Dino19.7 6 145.0175 3.62 2.77015.500 1 5 6
Maserati Bora15.0 8 301.0335 3.54 3.57014.600 1 5 8
Volvo 142E21.4 4 121.0109 4.11 2.78018.601 1 4 2

we can get the data from the TS script kernel as follows:

In [3]:
Out[3]:
{ mpg: 21,
  cyl: 6,
  disp: 160,
  hp: 110,
  drat: 3.9,
  wt: 2.62,
  qsec: 16.46,
  vs: 0,
  am: 1,
  gear: 4,
  carb: 4 }