====== ICS Instructional Openlab Linux Cluster ====== ===== Summary ===== The ICS Instruction Openlab Linux Server cluster is for general purpose computing at the School of ICS and accessible to any user with an ICS shell account. Cluster priority is given to instruction, then undergraduate students, then graduate students, and then ICS researchers. ==== Guidelines ==== In order to ensure that Openlab cluster remain available to instruction and researchers fairly, users may expect the following: * Long running jobs must be submitted through our [[:services:slurm|SLURM]] job queueing system in order to balance resources. * Jobs executed from via SLURM queueing system will be given priority over jobs running on the command line. * Processes running for longer than 2 hours will be reniced 19. * Processes running for longer than 5 days will be suspended. * [[hardware:cluster:openlab#Exceptions|Exceptions]] * Processes for users with more than 100 processes will be suspended. * [[hardware:cluster:openlab#exceptions|Exceptions]] * Files in the /tmp directory will be cleared after two weeks. Please use [[services:slurm|slurm]] for long running or serial projects requiring more than one openlab node. Processes running through slurm are not subject to the above guidelines. Instructors may reach out to heldpesk@ics.uci.edu in order to request course exceptions to any of these guidelines. Also see [[courses:openlab_faq|]] ===== Home Directory ===== Your [[accounts:ics_home_directory|ICS home directory]] is shared across all ICS support linux hosts including the openlab hosts. ===== Connecting to Openlab ===== You may log in remotely from any host using an ssh client. See our [[software:commontools#ssh|common tools page]] for a list of clients. Login into one of the following domain names (e.g. //ssh openlab22.ics.uci.edu//). In this case though, openlab and openlab22 will take you to the same set of servers since we have moved the same OS for all openlab machines. ^ Domain Name ^ Operating System ^ | openlab.ics.uci.edu | Ubuntu 22.04 hosts only | | opengpu.ics.uci.edu | Ubuntu 22.04 hosts only| **NOTE**: If you are running a long running process, please take note of the hostname you are running on. The name "openlab.ics.uci.edu" is a load balancer and may not point you to the same host twice. You can determine the hostname by running the command `hostname` from the shell cli. When you come back to check on your process, you will ssh into this name instead of openlab.ics.uci.edu. ==== MOD/Splash Screen ==== When you login to the Openlab Linux cluster you will receive a lot of information that includes sections about fair use, system load and message from individual software [[commands:modules|ICS softward modules]] you add to your environment. This information may also include a message similar to the following: Expanded Security Maintenance for Applications is not enabled. 19 updates can be applied immediately. 11 of these updates are standard security updates. To see these additional updates run: apt list --upgradable 24 additional security updates can be applied with ESM Apps. Learn more about enabling ESM Apps service at https://ubuntu.com/esm You may safely ignore this message. This message is automatically managed by [[https://ubuntu.com/landscape/features|Ubuntu Landscape]] and regards the host that you have logged into, not your local system. This message indicates that there are patches available or that patches that have been applied require a system restart in order to fully install. Our systems administrators will take care of that, typically between quarters when systems are not heavily used. ==== Off Campus Access ==== **ADVISORY**: Off campus access requires [[accounts:ssh_keys|ssh keys]]. Users connecting from off campus must use keys. Users connecting from off campus cannot use passwords to login to the openlab cluster. What does ICS consider on campus: * Any host with a reverse dns name that ends in uci.edu. This includes most campus hosts and hosts connected via VPN. * Any host connected to the campus 172.23.0.0/20 network. These are the match rules in the ICS sshd_config file: Match Host *.uci.edu PasswordAuthentication yes Match Address 172.23.0.0/20 PasswordAuthentication yes ==== Load Balancing ==== The name openlab.ics.uci.edu takes you to a load balancer. The load balancer will send your connection request to the machine with the lowest load. **If your work requires you to work with multiple shells on the same host**, then login directly using a hostname instead of openlab.ics.uci.edu == Determining Hostname == In order to identify which openlab host you are on run the command: hostname In the following example, the user has logged into a host named circinus-30.ics.uci.edu: % hostname circinus-30 == SSH Tunneling == In order to create an ssh tunnel to the host that you are a process on, you **cannot** use the name openlab.ics.uci.edu. In the following example, the user has connected to openlab and run jupyterlab on port 5555. They have run hostname and determined that the name of the host they have run jupyterlab on is **circinus-2.ics.uci.edu**. % ssh -L 8080:localhost:5555 luser@circinsus-2.ics.uci.edu ==== Availability ==== The openlab cluster is available on and off campus via SSH and [[services:slurm|SLURM]]. There are limitations regarding authentication and how users can connect to the openlab cluster. Please refer to the following matrix for specific information: ^Location^[[accounts:ssh_keys|SSH with Keys]]^SSH with Password^[[services:slurm|SLURM]]^ ^On campus| Yes | Yes | Yes | ^Off Campus*| Yes | No | No | ^UCI VPN| Yes | Yes | Yes | *Off-campus locations include student housing, University Hills housing, and the campus WIFI network. ==== Using UCI VPN ==== Check here for directions connecting to the [[network:campus:campusvpn|]]. Make sure you download the software version and select "UCI" from the dropdown menu when choosing a connection type. ==== Windows Users ==== - Newer versions of Windows have ssh built in. Optionally, you can install [[https://docs.microsoft.com/en-us/windows/wsl/install-win10|Linux Subsystem for Windows]] to take advantage of running other Linux commands locally. - Open terminal, using [[https://docs.microsoft.com/en-us/windows/terminal/|the updated Windows Terminal]] is recommended. - ssh {ics_userid}@openlab.ics.uci.edu or Please reference this page on how to connect to Openlab using Putty software: [[accounts:putty|set up Putty for Openlab]]. ==== Mac Users ==== Open Terminal and type: ssh yourICSusername@openlab.ics.uci.edu ==== FOSS SSH Clients ==== You may use any SSH client to connect to the openlab servers. This short list provides a FOSS ssh clients: See our [[software:commontools#ssh|common tools page]] for a list of ssh clients. ===== Setup your SSH Keys ===== Off-campus users must use SSH keys to connect to the openlab cluster. Follow these directions to [[accounts:ssh_keys|]] (includes a link to PuTTY key setup). ===== Donations ===== Much of the openlab cluster runs on donated hardware. Feel free to contact support if you think you have some spare hardware that you would like to add. ===== Details ====== * Operating System: Ubuntu 22.04. * Provisioning: Puppet 6.x * Primary Investigator: Computing Support * Contact Email: helpdesk@ics.uci.edu ===== Hardware Specifications ===== The Openlab hosts are made up of two sets of hardware that are grouped with the following machines: ==== Circinus Cluster ==== ^Description ^Information | |Cluster|Circinus Cluster| |Count | 96 | |Model | HP Proliant SL390 G7 | |Processor | 2 x Intel Xeon X5680 @ 3.33 Ghz | |RAM | 96 GB | |HDD | Varies | |RAID |No | |Redundant Power |Yes| ==== Odin ==== ^Description ^Information | |Cluster|Openlab Cluster| |Count |1| |Model | Dell PowerEdge R815 | |Processor |AMD Opteron(tm) Processor 6378 | |RAM |512 GB | |HDD | 98GB | |RAID | 5 | |Redundant Power |Yes | ==== Tristram==== ^Description ^Information | |Cluster|Openlab Cluster| |Count |1| |Model | Dell PowerEdge R815 | |Processor |AMD Opteron(tm) Processor 6378 | |RAM |512 GB | |HDD | 98GB | |RAID | 5 | |Redundant Power |Yes | ===== User Access ===== * All Users ===== Services Provided ===== * [[services::ssh_access|Shell]] * [[services:slurm|SLURM job scheduling program]] * [[virtual_environments:|ICS Virtual Environments]] ===== Code Editor suggestions ===== * [[accounts:using_vscode_with_openlab|Visual Studio Code]] * vi * Cheat Sheets for learning vi can be easily be searched for. ===== Members ===== Complete rosters for the openlab cluster can be displayed with the following [[services:slurm|sinfo]] command: module load slurm sinfo ===== Long Running Processes ===== ==== Resuming Suspended Processes ==== If you find your process has been suspended, you may resume it by using the kill command ton send the CONT signal. kill -CONT You may list your processes using the command `ps ux`. Suspended processes will have a "T" in the 8th column USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND hans 2458438 0.0 0.0 6476 2244 pts/18 S+ 14:08 0:00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn --exclude-dir=.idea --exclude-dir=.tox T hank 3554779 0.0 0.0 20480 3164 ? Ts 2022 24:43 SCREEN -xxR In this case you would resume the SCREEN process, pid 3554779 by running: kill -CONT 3554779 ==== Exceptions ==== The following long running processes will not be automatically suspended on the Openlab Linux Cluster: * bash * rbash * csh * dash * ksh * ksh93 * tcsh * zsh * screen * tmux * nologin * nologin * ksh * rksh * csh * (sd-pam) * sshd * ssh-agent * systemd * dbus-daemon * node * vim * remote-dev-serv * fsnotifier * ion.clangd.main * ld-linux-x86-64 * fsnotifier * cpptools * cpptools-srv * xdg-mime * dbus-send =====Troubleshooting====== Also see [[courses:openlab_faq|]] ==== RSA host key for has changed ==== If you receive an error "Hot key verification failed" when logging in, you may, if you are confident you are logging into the correct host, clear the offending key by issuing the following command, on most platforms, to clear the old host key and resolve the error: ssh-keygen -R If this does not work on your platform, please send a message to helpdesk@ics.uci.edu along with the output from the command so we can extend this information. ==== RC Files ==== ICS users may occasionally damage their dot files, which results in unusual-looking SSH prompts and/or being unable to use the “module” command. If this is the case, click here for directions to [[https://swiki.ics.uci.edu/doku.php/accounts:restore_unix_dot_files|]] If the above fails, contact [[https://swiki.ics.uci.edu/doku.php/start?s[]=helpdesk#contacting_ics_computing_support_helpdesk|helpdesk]].