Openlab Jupyterhub @ ICS

Management Information(sgroup only)

ICS Instructors, researchers and students are invited to connect to the Openlab Jupyterlab https://hub.ics.uci.edu.

Openlab Jupyterhub turns a web browser into a personal Linux workstations offering a Linux terminal, VSCode IDE, Jupyter Notebooks, RStudio, an X11 desktop. ICS home directories are mounted to provide persistent storage.

Beginning in Fall 2024 Openlab Jupyterhub requires DUO. After logging in you will receive a DUO prompt so check your device.

NOTE For users with physical tokens who do not use the smartphone app, you must enter your DUO Passcode as part of your password:

Password:  password,passcode 

• Duo Device Management Portal
• DUO Support Desk
• OIT Duo Multi-factor Authentication

1. If you are off campus you will need to first connect to the Campus VPN
2. Visit https://hub.ics.uci.edu.
   1. Course specific sites are available, for a specific URL please talk to your instructor.
3. Log in with your ICS credentials.

You may open up the control panel to manage running servers and start new servers:

https://hub.ics.uci.edu/hub/home

This is especially useful if your running hub is unresponsive.

The complete list of hub sites available for Fall 2024 is in the table below.

• DUO Authentication: Users will be prompted for DUO Authentication now.
• Ubuntu 22.04 LTS
• Coder Version 4.91.1
• R-Studio Version 2024.04.2-764
• Python 3.10.12
• R version 4.4.1
• gcc 11.4.0
• gcc 9.5.0 on course jupyter hubs.
• jupyter C kernel
• Man pages.

• R-Studio 2023.06.1+524
• R version 4.3.1 (2023-06-16) – “Beagle Scouts”
• Python 3.11.4

• Ubuntu 22.04LTS
• Larger 8 core/16g ram containers available for datas ience applications
• Latest Coder Version 4.11
• Latest RStudio Version 2022.12.0+353
• New versions of NodeJS and Yarn
• Larger preinstalled R libraries
• Firefox install for Desktop
• Package self-management (e.g. `sudo apt install`)

Jupyterlab@ICS is built on donated equipment in the openlab cluster for research and instructional computing.

• Nothing to configure, no VM to run, no modules to load.
• Prior sessions can be reopened from the Running Terminals and Kernels pane.
• Persistent storage in your ICS home directory.
• Students and instructors work in the same cloud IDE.
• Works for any OS and any browser.
• Customizable per course of per research group.
• Users are isolated from one another.
• Mid-quarter update.

When you login to ICS Jupyterhub using any web browser, you get a personal docker container

• Ubuntu 22.04 LTS
• 18 nodes
• 360 CPU Cores
• 1.33 TiB Memory Total
• Up to up to 8 vcpu cores and 16 GB RAM More
• Backed by super quick all flash storage Faster
• Your own ICS Account home directory
• `sudo apt` allows self-service software installation.
• IDE's
  • VSCode 4.11
  • Rstudio 2022.12.0
  • Octave
• Servers
  • Shiny Server
• Notebooks
  • Python
  • JavaScript
  • Julia
  • R
  • Octave
• CLI Tools
  • gcc/g++ and debugger
  • python3
  • R
  • Java
• GIT Integration tools
• X11 DEsktop

• Jupyterhub.org
• Zero2Jupyterhub
• Git Repo
• Customization Guide
• Jupyterhub Git Integration Guide Good place for help

ICS is running a Jupyter lab.

1. If you are off campus you will need to first connect to the Campus VPN
2. Visit https://hub.ics.uci.edu.
   1. Course specific sites are available, for a specific URL please talk to your instructor.
3. Login with your ICS credentials.

	Notebooks: R,Python3, NodeJS, Julia
	VSCode
	X11 Desktop

In order to restart your Jupyterhub pod, navigate to the Hub Control Panel from the menu: “File > Hub Control Panel”

This will open a new tab similar to this:

If your hub is not responding, you may also go directly to the control panel at this URL: https://hub.ics.uci.edu/hub/home

If a Jupyterhub pod is currently running, there will large red button that says “Stop My Server”. Clicking on that button will stop the running Jupyterhub pod.

After a moment, the “Stop My Server” and “My Server” buttons will be replaced by a “Launch Server” Button, click on it and follow on screen prompts.

Users can start multiple Jupyter servers on a single JupyterHub instance.

In order to start a second server, navigate to the Hub Control Panel in “File > Hub Control Panel”

This will open a new tab.

Enter a name for your new server, then click on the link to the right: “Add New Server”.

The https://hub.ics.uci.edu site includes a wide number of extensions. These extensions will come and go during the quarter based on availability, usefullness, load, and security. Please send any requests for new extensions to helpdesk@ics.uci.edu.

Instructors may request a course specific hub. Available plugins will not change during the quarter except in response to extreme security vulnerabilities. Course hubs can be restricted to enrolled students and students on a list curated by the instructor. Upon request, instructors may be given rights to view student projects.

• Ubuntu 20.04
• Base notebook changed to https://hub.docker.com/r/jupyter/scipy-notebook
• R version 4.0.2

Please send email to helpdesk@ics.uci.edu if you have any problems running the following commands.

Start up a terminal from the launcher (you can start the launcher with ctrl + shift +l):

To install, run the following from the terminal command line.

sudo apt update 
sudo apt install -y postgresql

The following are commands you can run:

apt remove
apt install
apt upgrade
tcpdump
netstat

If you need to use other commands, email helpdesk@ics.uci.edu for consideration with an explanation as to why you need the command.

The following command create the database in your ICS home directory (which is persistent storage) and runs locally on the machine.

export PATH=$PATH:/usr/lib/postgresql/12/bin/
initdb -D ~/pgsql  -U your_user_name
pg_ctl -D ~/pgsql -l logfile -o "-k /tmp" start

If you restart your hub, you will need to install and run pgsql again, but you will not need to re-initialize the database.

NOTE: This works for small file sizes. Please use Samba to map your network drive to copy large file.

1. Open the jhub filebrowser by clicking on the indicated file browser icon:
   • 
2. Drag files from your local computer into the file tree on the left hand window pane in your browser. This uploads the file into your ICS account:
   • 

1. Simply right click on the file you would like to download and select “Download” from the popup menu:
   • 

These directions can be found in thei complete form here.

1. conda create –name myenv
2. conda install -c anaconda ipykernel
3. python -m ipykernel install –user –name=myenv

At this point, a new tile should appear in your launched called myenv

Use sshfs.

A Install the C/C++ IntelliSense extension for VSCode:

https://github.com/Microsoft/vscode-cpptools

Note that VSCode in hub is not true VSCode, and as such it won't have access to the same list of extensions.  The product used inside of hub is:

https://github.com/cdr/code-server

Please use a different workspace.
This workspace is already in use in another JupyterLab window. 
Please enter another workspace name.

You ran out of space or have too many files on your home directory (H:).  Delete files as needed.

A You already have a Jupyterlab browser open. Simply type in a name for the new desktop, anything you like, and click on “Switch Workspace”.

A We run Harbor locally.

We were very happy with it until version 1.9 implemented an AD specific directory search. We are currently running an ICS fork but, if the folks at goharbor do not fix it, we will need to find another solution.

A Use the following command to add the local conda library to your RStudio session

     .libPaths( c( "/opt/conda/lib/R/library/" , .libPaths() ) )
     
     

A In the past this has happened when moving from an older version of the hub to a newer one. See if the following steps resolve the issue:

1. Click on the stop server button in https://hub.ics.uci.edu/hub/admin#/
2. ssh into openlab.ics.uci.edu and `rm -rf ~/.jupyterhub ~/.config/xfce4`
3. Start a new server in https://hub.ics.uci.edu/hub/admin#/ (or log out and back into https://hub.ics.uci.edu

First and foremost, bring the problem to your instructor's or TA's attention. Chances are you are not the only one experiencing this problem and they will be able to help or provide you with the next steps to resolving your problem.

Your instructor may ask to you submit a trouble report to the ICS Computing Support Helpdesk if they don't have the solution or if you need more help than they can provide.

In order to submit a helpdesk, please include the following information in an email to helpdesk@ics.uci.edu:

• ICS Account Name
• Hub Instance (e.g. ics51, ics53)
• Location(e.g. Home, Openlab, Middle Earth)
• Campus VPN
• Operating System
• Browser
• Hub Applications (e.g. desktop, termlina, R)
• Time and date that the problem occurred
• Symptom/Issue.

This is a known problem in the Fall'23 image. The fix is to redeploy the container with client_max_body_size 500m; This is pending. In the meantime use ICS SFTP/SCP or ICS Mapped Network Drive to move the file to your homedir.

Check the following

• Permissions on the home directory are minimally 711 (although 777 is bad).
• Account is under quota (look up user on https://support.ics.uci.edu/elnino2 and click on Tardigrade usage)

If the user indicates the password has been recently changed, confirm either ElNino or using Apache Directory Studio.

Check course enrollment, which hub should the user be trying to attach to.

If everything looks good, write back to the user. Add that that was checked to the ticket.

Remind the user that they should be receiving a DUO authentication notice. If they aren't getting the Notice then they should goto OIT's Duo Support Desk to check for problems and remove any devices that they no longer use.

Check DUO authentication. Sometimes the account id disabled, the following will show up in duoldap:/opt/duoauthproxy/log/authevents.log:

 "msg": "Duo preauth result was Your account is disabled and cannot access this application. Please contact your administrator.", , 
"status": "Reject", "server_section": "ldap_server_auto", 

When logging in, they can open the app and see if DUO is silently prompting them.

If their DUO app is not silently prompting them then they can try appending a DUO passcode to their password when logging in. IF THIS WORKS we need to know why so keep them on the hook and let's work with them on OIT to figure out why they weren't getting a notification.

Dear Student,

It looks like your ICS account name is <username> and you are trying to login to <class>-hub.ics.uci.edu. This information is case-sensitive.

I see that your account is not locked, permissions are appropriate, and your account is under quota.  

Make sure that you are on campus or using the campus VPN.  Clients that are off campus, located in student housing, or using the guest wifi network will need to connect to the Campus VPN before continuing.  UCI OIT runs the campus VPN and more information is available on their site:

https://www.oit.uci.edu/services/security/vpn/  

You can test your password by logging into https://elnino.ics.uci.edu or ssh'ing into openlab using your password via VPN or from an on-campus host.

If you have not already done so, please try resetting your password at the following site:

https://password.ics.uci.edu/ssp/?action=sendtoken

Please note the menu bar at the top of the page with Question and Email.   Your recovery email address is a @gmail.com address.  More information on the password reset site is available on our wiki:

https://wiki.ics.uci.edu/doku.php/accounts:password_change_reset  

If you continue to have problems, it looks like you may not be receiving DUO prompts.  Check the current status of your DUO account and remove any defunct devices:

https://applications.oit.uci.edu/DuoSupportDesk/enrollment.htm

If everything looks proper and you continue to encounter problems, try appending a DUO passcode to your password with a comma:

password,passcode

More information is available here:  https://wiki.ics.uci.edu/doku.php/virtual_environments:jupyterhub#duo_authentication

For users that have already changed their password:

Dear Student,

It looks like your ICS account name is <username> and you are trying to login to ics45c-hub.ics.uci.edu.   Please make sure you are connected to the campus VPN and keep in mind that the login is case-sensitive (no-caps).

I see that your account is not locked, permissions are appropriate, and your account is under quota.

I see that you successfully reset your password recently. You can test your new password by logging into https://elnino.ics.uci.edu or ssh'ing into openlab using your password.  If you are off campus you will need to be connected  (you must be connected to the campus VPN).

It looks like you may not be receiving DUO prompts.  Check the current status of your DUO account and remove any defunct devices:

https://applications.oit.uci.edu/DuoSupportDesk/enrollment.htm

If everything looks proper and you continue to encounter problems, try appending a DUO passcode to your password with a comma:

password,passcode

More information is available here:  https://wiki.ics.uci.edu/doku.php/virtual_environments:jupyterhub#duo_authentication

Q. We had a student complaining that they were not receiving DUO prompts

A. They were directed to the OIT Duo Support Desk. They determined that they needed to update their device.

Q. How does one login with a DUO passcode if they don't have the smartphone app?

A. Append the passcode from the physical token to the password with a comma: password,passcode

Your using the Desktop on hub, the screensaver came on and locked the screen and now it doesn't accept your username/password.

Fix:

1. Go back to the main site, start up the launcher (ctrl-shift-L) and open a Terminal (click on the Terminal tile).
2. At the prompt, run

   kill -9  $(pidof /usr/bin/xfce4-screensaver)

3. Your screen is unlocked now.

Preventative:

1. Turn off the screen saver from the Desktop goto Applications → Settings → Screensaver and slide the “Enable Screensaver” switch to the left.

Root cause: Screen saver login is provided by PAM. PAM is not configured to use ICS LDAP, it is configured to use local files and you do not have a proper password entry in the local files.

sudo: The "no new privileges" flag is set, which prevents sudo from
running as root.
sudo: If sudo is running in a container, you may need to adjust the
container configuration to disable the flag.

Please add allowPrivilegeEscalation: yes to the hub config.yaml.

https://discourse.jupyter.org/t/issue-using-sudo-in-container-the-no-new-privileges-flag-is-set/17071/3

The hub container hangs when trying to open up a CSV file.

Use the Firefox browser. The CSV reader exhausts a memory limit that both Edge and Chrome have set.

Permament solution is pending.

The X11 desktop is blank or black.

You will need to restart the hub container using the directions above above.

This happens when the “Send CtlAltDel” button is clicked in the upper right corner of the screen. We are looking for a solution, but for the time being, do not click that button.

See helpdesk ticket 73286

It appears as if the server failed to start:

Running `kubectl -n jhub log jupyter-junjies1`, immediately after opening starting the jhub server, revealed log messages regarding permissions.

In this case, the user home directory was set to 0700. Setting the user home directory to 0711 resolved the problem:

chmod 711 $HOME

See https://helpdesk.ics.uci.edu/Ticket/Display.html?id=73077

kubectl  -n jhub-ics53 delete pod jupyter-dingruiz
...
Events:
  Type     Reason         Age                    From                  Message
  ----     ------         ----                   ----                  -------
  Warning  FailedKillPod  2m19s (x55 over 111m)  kubelet, centaurus-5  error killing pod: failed to "KillContainer" for "notebook" with KillContainerError: "rpc error: code = Unknown desc = operation timeout: context deadline exceeded"

Try this:

kubectl  -n jhub-ics53 delete pod jupyter-dingruiz

If that doesn't work:

kubectl  -n jhub-ics53 delete pod jupyter-dingruiz --now

When all else fails:

kubectl  -n jhub-ics53 delete pod jupyter-dingruiz --grace-period=0 --force  

This error seems to pop up when changing the hub container version, usually going backwards. If it happens, the solution I have been going with is deleting the entire helm chart and reinstalling.

After entering username and password, the following error is displayed:

403 : Forbidden
'_xsrf' argument missing from POST

We're still investigating. The problem occurs with k8s-hub images later than series 1. So far jupyterhub/k8s-hub:1.1.3-n023.h60c5e9f8 is the latest running container.

Possible solution https://github.com/ReviewNB/jupyterlab-gitplus/issues/3

jupyter labextension install @reviewnb/jupyterlab_gitplus

If the service is unavailable, this indicates that the kubernetes ingress cannot contact the node running the hub pod.

This problem occurs when the internal networking of the cluster fails. Recently we've been rebooting the cluster when this happens. Some nodes can reach the cert manager, some cannot. We're still working on determining the root cause and developing tools to figure out exactly what is broken.

This was a symptom that some nodes were unable to pull images form the harbor host last time. The harbor host might be on a host that isn't connected anylonger.

• Kubernetes@ICS
• Docker@ICS
• Instructional Virtual Machines