Connect to remote IPython notebook

Connect to a single remote computer

Suppose you are at your local computer and you want to connect to an IPython notebook running on a reomote computer. Here is what you need to do:

  1. From your local computer, create an alias for this ssh tunnel command as a script “setalias”:

    alias sshcluster="ssh remote.computer -L PORT:localhost:PORT"

  2. Source the ‘setalias’ script:

    . setalias

  3. Use the alias to create a ssh tunnel. Open screen to prevent a disconnection to kill your job, if necessary.

  4. Now you are at the remote computer, start the IPython notebook

    ipython notebook --port=PORT --no-browser

    this will start an IPython notebook server on specified PORT without automatically open a browser connecting to it, because apparantly you want a browser in your local computer to connect to the server.

  5. You are good to go! In your local computer, open a browser, direct it to localhost:PORT. You are now connected to the remote IPython notebook server!

  6. If another user also did the similar ssh tunnel to the same remote computer, can this user also connect to the IPython server you’ve started? I haven’t try this....can ssh tunnel be shared ?? Answer: YES it works with multiple users. Everyone can create a new notebook and work on it separately. Be carefully tho to open the notebook server at a user writable location, or use –notebook-dir= flag.

Connect to a cluster

Sometimes you want to connect to a computer cluster, through a head node and a job scheduler such as PBS. There are more steps involved. Here is a script https://github.com/pyHPC/ipynbhpc written by Andrea Zonca for SDSC Gordon.

The basic steps are:

  1. set the alias the same as the last section

  2. from local computer, run command ‘sshcluster’ to ssh to cluster head node, go to the directory where your project resides, create a “run-setup” script:

    #!/bin/bash

export IPYNB_PORT=7999
export IPYNB_MINUTES=30
export IPYNB_QSUB_TEMPLATE="qsub -I -V -l nodes=1:ppn=8 -l walltime=00:%d:00"
export IPYNB_NOTEBOOK_TEMPLATE="ipython notebook --port=%d --no-browser"

    to source the setup:

    . run-setup

  3. make sure “ssh localhost” works:

    ssh-keygen
cd ~/.ssh
cat id_rsa.pub >> authorized_keys

  4. If you haven’t done so, clone the git repo into the current directory:

    git clone https://github.com/pyHPC/ipynbhpc.git

  5. copy the script locally and do some editing:

    cd ipynbhpc
cp ipynbhpc ipynbhpc-yourlabel
vi ipynbhpc-yourlabel
save and quit
cd ..

  6. run the new script:

    ./ipynbhpc/ipynbhpc-yourlabel

    the script will submit qsub script to start an interactive job, get back the compute node’s hostname, then start IPython notebook server on compute node with no browser. Then build another ssh tunnel from compute node to headnode

  7. Now the two tunnels will work together to tunnel from compute node to local computer. Now local computer’s browser can be directed to localhost:PORT to connect to IPython server on compute node. Done.

Diagnose the script

The python script itself is written with some techniqes, here is detailed elaboration of the script and why we need to make the edits:

  1. use “ssh -t -t -4 localhost” to force pseudo terminal . A pseudo-terminal is mainly used to make a process believe that it interacts with a terminal (with stdin and stdout) although it actually interacts with one or more processes. This is used for python Popen subprocess to send command through STDIN pipe even tho STDIN doesn’t exist.
  2. On head node, open a subprocess to force pseudo terminal on localhost, then submit a qsub script to start an interactive job, and wait for the job to start. After the job starts, the subprocess is on compute node, use a hostname command to get the compute node’s hostname back, then start the IPython notebook server on compute node, wait until the notebook server starts.
  3. At this step, this script still runs at head node. Open another subprocess with command “ssh -L ipynbport:localhost:ipynbport computenode-hostname”, create a ssh tunnel from compute node to head node. use PIPE stdin and stdout to communcate with the compute node, make sure the tunnel is created, then print “sucess” message to user.

Note: the ipython subprocess running on compute node will not complete unless killed. The main process uses stdin and stdout/stderr PIPE to communiate with the subprocess. (If the stdout reach a end-of-file, then it means the subprocess is terminated/finished/completed, however you call it, Popen.wait() should return and set return code)