HPCwiki - User contributions [en]

Conda for teaching

2020-05-27T10:01:46Z

Maas036: /* Setup */

You are going to give a teaching course, and you need a specific code environment.

== Setup ==
First - find a good location that everyone can read (and not write). I'd suggest somewhere under <code>/cm/shared/apps/SHARED/</code> as a starting point - this allows everyone to access this location. It's important not to put anything secret there - it's a public resource, so please bear that in mind.

Next - create a folder for your environment:

<source lang='bash'>
mkdir /cm/shared/apps/SHARED/my_conda_env
chmod +r /cm/shared/apps/SHARED/my_conda_env
</source>
You may want to manipulate the permissions for this folder if someone is going to set this up with you. Consider the commands in [[Shared folders]].

Then, install Anaconda into it (choose the year and month yourself!):

Also, note that you need to make the sh script executable before you can run it.

<source lang='bash'>
wget https://repo.anaconda.com/archive/Anaconda3-YEAR.MONTH-Linux-x86_64.sh
./Anaconda3-YEAR.MONTH-Linux-x86_64.sh -s -b -p /cm/shared/apps/SHARED/my_conda_env
</source>

Now you have a working conda environment in this folder. You can manipulate this here by running <code>/cm/shared/apps/SHARED/my_conda_env/bin/conda</code>, or, I would recommend creating a modulefile so that you can use it as default.

Create the following example modulefile in a matching <code>/cm/shared/modulefiles/SHARED/my_conda_env</code>:

<source lang='bash'>
#%Module -*- tcl -*-
##
## conda environment modulefile
##

set loadedmodules [split $::env(LOADEDMODULES) ":"]
set modulepath [split $ModulesCurrentModulefile "/"]
set envpath [lrange $modulepath 4 end]

set root /cm/shared/apps/[join $envpath "/"]

proc ModulesHelp { } {
global version

puts stderr "\tThis module provides the conda environment at $envpath"
}

if { [module-info mode] != "whatis" } {
puts stderr "[module-info mode] environent $envpath ."
}

module-whatis "Provides environment $envpath"

prepend-path PATH $root/bin
prepend-path LD_LIBRARY_PATH $root/lib
prepend-path LIBRARY_PATH $root/lib
prepend-path CPATH $root/include
prepend-path MANPATH $root/share/man
</source>

This will allow you to <code>module load SHARED/my_conda_env</code> and thus have <code>conda</code> pathed to the currently active environment.

== Jupyter Kernel ==

In order for students to be able to use this environment in jupyter, they will need a kernel definition.

Kernel definitions are usually a separate folder containing, in particular, a file called <code>kernel.json</code>, plus an icon that is displayed that represents this kernel, and other helper code.

The setup for this is that you should create the following folder for their access:

<source lang='bash'>
mkdir -p /cm/shared/apps/SHARED/my_conda_env/kernel/my_conda_env
chmod +r /cm/shared/apps/SHARED/my_conda_env/kernel/my_conda_env
</source>

This folder is something they will need to copy in place to their home directory, specifically <code>$HOME/.local/share/jupyter/kernels/</code>

Inside this folder, create the following <code>kernel.json</code> file. Watch out that the paths will need to match the current environment path if you're using a different location!

=== Python Kernel ===

<source lang='bash'>
vim /cm/shared/apps/SHARED/my_conda_env/kernel/my_conda_env/kernel.json
</source>
<source lang='bash'>
{
"env": {
"PATH":

"/cm/shared/apps/SHARED/my_conda_env/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin"
},
"language": "python",
"argv": [
"/cm/shared/apps/SHARED/my_conda_env/bin/python",
"-m",
"ipykernel",
"-f",
"{connection_file}"
],
"display_name": "my_conda_env"
}
</source>

For Python, that's it. Ipykernel is installed automatically on conda initialisation.

=== R Kernel ===

For an R kernel, you need to make sure that the IRkernel package is installed. This is the package that is used to communicate from Jupyter to your running R kernel.
<source lang='bash'>
#MISSING EXAMPLE CODE
#PROBABLY /cm/shared/apps/SHARED/my_conda_env/bin/conda install R_irkernel
</source>

You'll also need to create two files: <code>kernel.json</code> and <code>kernel.js</code>. the <code>kernel.js</code> is a helper script to allow jupyter to communicate to R effectively:

<source lang='bash'>
vim /cm/shared/apps/SHARED/my_conda_env/kernel/my_conda_env/kernel.json
</source>
<source lang='bash'>
{
"env": {
"LD_LIBRARY_PATH":
"/cm/shared/apps/SHARED/my_conda_env/lib:/cm/shared/apps/SHARED/my_conda_env/lib64"
},
"argv": ["/cm/shared/apps/SHARED/my_conda_env/bin/R", "--slave", "-e", "IRkernel::main()", "--args", "{connection_file}"],
"display_name": "MAE50806-AdvMolEcol/Sandbox_R",
"language": "R"
}
</source>

<source lang='bash'>
vim /cm/shared/apps/SHARED/my_conda_env/kernel/my_conda_env/kernel.js
</source>
<source lang='bash'>
const cmd_key = /Mac/.test(navigator.platform) ? 'Cmd' : 'Ctrl'

const edit_actions = [
{
name: 'R Assign',
shortcut: 'Alt--',
icon: 'fa-long-arrow-left',
help: 'R: Inserts the left-assign operator (<-)',
handler(cm) {
cm.replaceSelection(' <- ')
},
},
{
name: 'R Pipe',
shortcut: `Shift-${cmd_key}-M`,
icon: 'fa-angle-right',
help: 'R: Inserts the magrittr pipe operator (%>%)',
handler(cm) {
cm.replaceSelection(' %>% ')
},
},
{
name: 'R Help',
shortcut: 'F1',
icon: 'fa-book',
help: 'R: Shows the manpage for the item under the cursor',
handler(cm, cell) {
const {anchor, head} = cm.findWordAt(cm.getCursor())
const word = cm.getRange(anchor, head)

const callbacks = cell.get_callbacks()
const options = {silent: false, store_history: false, stop_on_error: true}
cell.last_msg_id = cell.notebook.kernel.execute(`help(\`${word}\`)`, callbacks, options)
},
},
]

const prefix = 'irkernel'

function add_edit_shortcut(notebook, actions, keyboard_manager, edit_action) {
const {name, shortcut, icon, help, handler} = edit_action

const action = {
icon, help,
help_index : 'zz',
handler: () => {
const cell = notebook.get_selected_cell()
handler(cell.code_mirror, cell)
},
}

const full_name = actions.register(action, name, prefix)

Jupyter.keyboard_manager.edit_shortcuts.add_shortcut(shortcut, full_name)
}

function render_math(pager, html) {
if (!html) return
const $container = pager.pager_element.find('#pager-container')
$container.find('p[style="text-align: center;"]').map((i, e) =>
e.outerHTML = `\\[${e.querySelector('i').innerHTML}\\]`)
$container.find('i').map((i, e) =>
e.outerHTML = `\$${e.innerHTML}\$`)
MathJax.Hub.Queue(['Typeset', MathJax.Hub, $container[0]])
}

define(['base/js/namespace'], ({
notebook,
actions,
keyboard_manager,
pager,
}) => ({
onload() {
edit_actions.forEach(a => add_edit_shortcut(notebook, actions, keyboard_manager, a))

pager.events.on('open_with_text.Pager', (event, {data: {'text/html': html}}) =>
render_math(pager, html))
},
}))
</source>

== Last Steps ==

In order to help your students get their kernel definitions into <code>$HOME/.local/share/jupyter/kernels/</code>, it's probably a good idea to write a small and simple notebook to do this when executed, or else instruct them to do:

<source lang='bash'>
cp -rv /cm/shared/apps/SHARED/my_conda_env/kernel/* .local/share/jupyter/kernels/
</source>

Conda for teaching

2020-05-27T10:00:06Z

Maas036: /* Setup */

You are going to give a teaching course, and you need a specific code environment.

== Setup ==
First - find a good location that everyone can read (and not write). I'd suggest somewhere under <code>/cm/shared/apps/SHARED/</code> as a starting point - this allows everyone to access this location. It's important not to put anything secret there - it's a public resource, so please bear that in mind.

Next - create a folder for your environment:

<source lang='bash'>
mkdir /cm/shared/apps/SHARED/my_conda_env
chmod +r /cm/shared/apps/SHARED/my_conda_env
</source>
You may want to manipulate the permissions for this folder if someone is going to set this up with you. Consider the commands in [[Shared folders]].

Then, install Anaconda into it (choose the year and month yourself!):
Also, note that you need to make the sh script executable before you can run it.

<source lang='bash'>
wget https://repo.anaconda.com/archive/Anaconda3-YEAR.MONTH-Linux-x86_64.sh
./Anaconda3-YEAR.MONTH-Linux-x86_64.sh -s -b -p /cm/shared/apps/SHARED/my_conda_env
</source>

Now you have a working conda environment in this folder. You can manipulate this here by running <code>/cm/shared/apps/SHARED/my_conda_env/bin/conda</code>, or, I would recommend creating a modulefile so that you can use it as default.

Create the following example modulefile in a matching <code>/cm/shared/modulefiles/SHARED/my_conda_env</code>:

<source lang='bash'>
#%Module -*- tcl -*-
##
## conda environment modulefile
##

set loadedmodules [split $::env(LOADEDMODULES) ":"]
set modulepath [split $ModulesCurrentModulefile "/"]
set envpath [lrange $modulepath 4 end]

set root /cm/shared/apps/[join $envpath "/"]

proc ModulesHelp { } {
global version

puts stderr "\tThis module provides the conda environment at $envpath"
}

if { [module-info mode] != "whatis" } {
puts stderr "[module-info mode] environent $envpath ."
}

module-whatis "Provides environment $envpath"

prepend-path PATH $root/bin
prepend-path LD_LIBRARY_PATH $root/lib
prepend-path LIBRARY_PATH $root/lib
prepend-path CPATH $root/include
prepend-path MANPATH $root/share/man
</source>

This will allow you to <code>module load SHARED/my_conda_env</code> and thus have <code>conda</code> pathed to the currently active environment.

== Jupyter Kernel ==

In order for students to be able to use this environment in jupyter, they will need a kernel definition.

Kernel definitions are usually a separate folder containing, in particular, a file called <code>kernel.json</code>, plus an icon that is displayed that represents this kernel, and other helper code.

The setup for this is that you should create the following folder for their access:

<source lang='bash'>
mkdir -p /cm/shared/apps/SHARED/my_conda_env/kernel/my_conda_env
chmod +r /cm/shared/apps/SHARED/my_conda_env/kernel/my_conda_env
</source>

This folder is something they will need to copy in place to their home directory, specifically <code>$HOME/.local/share/jupyter/kernels/</code>

Inside this folder, create the following <code>kernel.json</code> file. Watch out that the paths will need to match the current environment path if you're using a different location!

=== Python Kernel ===

<source lang='bash'>
vim /cm/shared/apps/SHARED/my_conda_env/kernel/my_conda_env/kernel.json
</source>
<source lang='bash'>
{
"env": {
"PATH":

"/cm/shared/apps/SHARED/my_conda_env/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin"
},
"language": "python",
"argv": [
"/cm/shared/apps/SHARED/my_conda_env/bin/python",
"-m",
"ipykernel",
"-f",
"{connection_file}"
],
"display_name": "my_conda_env"
}
</source>

For Python, that's it. Ipykernel is installed automatically on conda initialisation.

=== R Kernel ===

For an R kernel, you need to make sure that the IRkernel package is installed. This is the package that is used to communicate from Jupyter to your running R kernel.
<source lang='bash'>
#MISSING EXAMPLE CODE
#PROBABLY /cm/shared/apps/SHARED/my_conda_env/bin/conda install R_irkernel
</source>

You'll also need to create two files: <code>kernel.json</code> and <code>kernel.js</code>. the <code>kernel.js</code> is a helper script to allow jupyter to communicate to R effectively:

<source lang='bash'>
vim /cm/shared/apps/SHARED/my_conda_env/kernel/my_conda_env/kernel.json
</source>
<source lang='bash'>
{
"env": {
"LD_LIBRARY_PATH":
"/cm/shared/apps/SHARED/my_conda_env/lib:/cm/shared/apps/SHARED/my_conda_env/lib64"
},
"argv": ["/cm/shared/apps/SHARED/my_conda_env/bin/R", "--slave", "-e", "IRkernel::main()", "--args", "{connection_file}"],
"display_name": "MAE50806-AdvMolEcol/Sandbox_R",
"language": "R"
}
</source>

<source lang='bash'>
vim /cm/shared/apps/SHARED/my_conda_env/kernel/my_conda_env/kernel.js
</source>
<source lang='bash'>
const cmd_key = /Mac/.test(navigator.platform) ? 'Cmd' : 'Ctrl'

const edit_actions = [
{
name: 'R Assign',
shortcut: 'Alt--',
icon: 'fa-long-arrow-left',
help: 'R: Inserts the left-assign operator (<-)',
handler(cm) {
cm.replaceSelection(' <- ')
},
},
{
name: 'R Pipe',
shortcut: `Shift-${cmd_key}-M`,
icon: 'fa-angle-right',
help: 'R: Inserts the magrittr pipe operator (%>%)',
handler(cm) {
cm.replaceSelection(' %>% ')
},
},
{
name: 'R Help',
shortcut: 'F1',
icon: 'fa-book',
help: 'R: Shows the manpage for the item under the cursor',
handler(cm, cell) {
const {anchor, head} = cm.findWordAt(cm.getCursor())
const word = cm.getRange(anchor, head)

const callbacks = cell.get_callbacks()
const options = {silent: false, store_history: false, stop_on_error: true}
cell.last_msg_id = cell.notebook.kernel.execute(`help(\`${word}\`)`, callbacks, options)
},
},
]

const prefix = 'irkernel'

function add_edit_shortcut(notebook, actions, keyboard_manager, edit_action) {
const {name, shortcut, icon, help, handler} = edit_action

const action = {
icon, help,
help_index : 'zz',
handler: () => {
const cell = notebook.get_selected_cell()
handler(cell.code_mirror, cell)
},
}

const full_name = actions.register(action, name, prefix)

Jupyter.keyboard_manager.edit_shortcuts.add_shortcut(shortcut, full_name)
}

function render_math(pager, html) {
if (!html) return
const $container = pager.pager_element.find('#pager-container')
$container.find('p[style="text-align: center;"]').map((i, e) =>
e.outerHTML = `\\[${e.querySelector('i').innerHTML}\\]`)
$container.find('i').map((i, e) =>
e.outerHTML = `\$${e.innerHTML}\$`)
MathJax.Hub.Queue(['Typeset', MathJax.Hub, $container[0]])
}

define(['base/js/namespace'], ({
notebook,
actions,
keyboard_manager,
pager,
}) => ({
onload() {
edit_actions.forEach(a => add_edit_shortcut(notebook, actions, keyboard_manager, a))

pager.events.on('open_with_text.Pager', (event, {data: {'text/html': html}}) =>
render_math(pager, html))
},
}))
</source>

== Last Steps ==

In order to help your students get their kernel definitions into <code>$HOME/.local/share/jupyter/kernels/</code>, it's probably a good idea to write a small and simple notebook to do this when executed, or else instruct them to do:

<source lang='bash'>
cp -rv /cm/shared/apps/SHARED/my_conda_env/kernel/* .local/share/jupyter/kernels/
</source>

Conda for teaching

2020-05-27T09:32:17Z

Maas036: /* Setup */

You are going to give a teaching course, and you need a specific code environment.

== Setup ==
First - find a good location that everyone can read (and not write). I'd suggest somewhere under <code>/cm/shared/apps/SHARED/</code> as a starting point - this allows everyone to access this location. It's important not to put anything secret there - it's a public resource, so please bear that in mind.

Next - create a folder for your environment:

<source lang='bash'>
mkdir /cm/shared/apps/SHARED/my_conda_env
chmod +r /cm/shared/apps/SHARED/my_conda_env
</source>
You may want to manipulate the permissions for this folder if someone is going to set this up with you. Consider the commands in [[Shared folders]].

Then, install Anaconda into it (choose the year and month yourself!):

<source lang='bash'>
wget https://repo.anaconda.com/archive/Anaconda3-YEAR.MONTH-Linux-x86_64.sh
./Anaconda3-YEAR.MONTH-Linux-x86_64.sh -s -b -p /cm/shared/apps/SHARED/my_conda_env
</source>

Now you have a working conda environment in this folder. You can manipulate this here by running <code>/cm/shared/apps/SHARED/my_conda_env/bin/conda</code>, or, I would recommend creating a modulefile so that you can use it as default.

Create the following example modulefile in a matching <code>/cm/shared/modulefiles/SHARED/my_conda_env</code>:

<source lang='bash'>
#%Module -*- tcl -*-
##
## conda environment modulefile
##

set loadedmodules [split $::env(LOADEDMODULES) ":"]
set modulepath [split $ModulesCurrentModulefile "/"]
set envpath [lrange $modulepath 4 end]

set root /cm/shared/apps/[join $envpath "/"]

proc ModulesHelp { } {
global version

puts stderr "\tThis module provides the conda environment at $envpath"
}

if { [module-info mode] != "whatis" } {
puts stderr "[module-info mode] environent $envpath ."
}

module-whatis "Provides environment $envpath"

prepend-path PATH $root/bin
prepend-path LD_LIBRARY_PATH $root/lib
prepend-path LIBRARY_PATH $root/lib
prepend-path CPATH $root/include
prepend-path MANPATH $root/share/man
</source>

This will allow you to <code>module load SHARED/my_conda_env</code> and thus have <code>conda</code> pathed to the currently active environment.

== Jupyter Kernel ==

In order for students to be able to use this environment in jupyter, they will need a kernel definition.

Kernel definitions are usually a separate folder containing, in particular, a file called <code>kernel.json</code>, plus an icon that is displayed that represents this kernel, and other helper code.

The setup for this is that you should create the following folder for their access:

<source lang='bash'>
mkdir -p /cm/shared/apps/SHARED/my_conda_env/kernel/my_conda_env
chmod +r /cm/shared/apps/SHARED/my_conda_env/kernel/my_conda_env
</source>

This folder is something they will need to copy in place to their home directory, specifically <code>$HOME/.local/share/jupyter/kernels/</code>

Inside this folder, create the following <code>kernel.json</code> file. Watch out that the paths will need to match the current environment path if you're using a different location!

=== Python Kernel ===

<source lang='bash'>
vim /cm/shared/apps/SHARED/my_conda_env/kernel/my_conda_env/kernel.json
</source>
<source lang='bash'>
{
"env": {
"PATH":

"/cm/shared/apps/SHARED/my_conda_env/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin"
},
"language": "python",
"argv": [
"/cm/shared/apps/SHARED/my_conda_env/bin/python",
"-m",
"ipykernel",
"-f",
"{connection_file}"
],
"display_name": "my_conda_env"
}
</source>

For Python, that's it. Ipykernel is installed automatically on conda initialisation.

=== R Kernel ===

For an R kernel, you need to make sure that the IRkernel package is installed. This is the package that is used to communicate from Jupyter to your running R kernel.
<source lang='bash'>
#MISSING EXAMPLE CODE
#PROBABLY /cm/shared/apps/SHARED/my_conda_env/bin/conda install R_irkernel
</source>

You'll also need to create two files: <code>kernel.json</code> and <code>kernel.js</code>. the <code>kernel.js</code> is a helper script to allow jupyter to communicate to R effectively:

<source lang='bash'>
vim /cm/shared/apps/SHARED/my_conda_env/kernel/my_conda_env/kernel.json
</source>
<source lang='bash'>
{
"env": {
"LD_LIBRARY_PATH":
"/cm/shared/apps/SHARED/my_conda_env/lib:/cm/shared/apps/SHARED/my_conda_env/lib64"
},
"argv": ["/cm/shared/apps/SHARED/my_conda_env/bin/R", "--slave", "-e", "IRkernel::main()", "--args", "{connection_file}"],
"display_name": "MAE50806-AdvMolEcol/Sandbox_R",
"language": "R"
}
</source>

<source lang='bash'>
vim /cm/shared/apps/SHARED/my_conda_env/kernel/my_conda_env/kernel.js
</source>
<source lang='bash'>
const cmd_key = /Mac/.test(navigator.platform) ? 'Cmd' : 'Ctrl'

const edit_actions = [
{
name: 'R Assign',
shortcut: 'Alt--',
icon: 'fa-long-arrow-left',
help: 'R: Inserts the left-assign operator (<-)',
handler(cm) {
cm.replaceSelection(' <- ')
},
},
{
name: 'R Pipe',
shortcut: `Shift-${cmd_key}-M`,
icon: 'fa-angle-right',
help: 'R: Inserts the magrittr pipe operator (%>%)',
handler(cm) {
cm.replaceSelection(' %>% ')
},
},
{
name: 'R Help',
shortcut: 'F1',
icon: 'fa-book',
help: 'R: Shows the manpage for the item under the cursor',
handler(cm, cell) {
const {anchor, head} = cm.findWordAt(cm.getCursor())
const word = cm.getRange(anchor, head)

const callbacks = cell.get_callbacks()
const options = {silent: false, store_history: false, stop_on_error: true}
cell.last_msg_id = cell.notebook.kernel.execute(`help(\`${word}\`)`, callbacks, options)
},
},
]

const prefix = 'irkernel'

function add_edit_shortcut(notebook, actions, keyboard_manager, edit_action) {
const {name, shortcut, icon, help, handler} = edit_action

const action = {
icon, help,
help_index : 'zz',
handler: () => {
const cell = notebook.get_selected_cell()
handler(cell.code_mirror, cell)
},
}

const full_name = actions.register(action, name, prefix)

Jupyter.keyboard_manager.edit_shortcuts.add_shortcut(shortcut, full_name)
}

function render_math(pager, html) {
if (!html) return
const $container = pager.pager_element.find('#pager-container')
$container.find('p[style="text-align: center;"]').map((i, e) =>
e.outerHTML = `\\[${e.querySelector('i').innerHTML}\\]`)
$container.find('i').map((i, e) =>
e.outerHTML = `\$${e.innerHTML}\$`)
MathJax.Hub.Queue(['Typeset', MathJax.Hub, $container[0]])
}

define(['base/js/namespace'], ({
notebook,
actions,
keyboard_manager,
pager,
}) => ({
onload() {
edit_actions.forEach(a => add_edit_shortcut(notebook, actions, keyboard_manager, a))

pager.events.on('open_with_text.Pager', (event, {data: {'text/html': html}}) =>
render_math(pager, html))
},
}))
</source>

== Last Steps ==

In order to help your students get their kernel definitions into <code>$HOME/.local/share/jupyter/kernels/</code>, it's probably a good idea to write a small and simple notebook to do this when executed, or else instruct them to do:

<source lang='bash'>
cp -rv /cm/shared/apps/SHARED/my_conda_env/kernel/* .local/share/jupyter/kernels/
</source>