2023 ICS Computing Support Announcments

Beginning on 01/03/2023 As part of phase 3 of OIT’s Email Security Initiative, ICS will be adopting the OIT's Proofpoint Email Security. All mail addressed to @ics.uci.edu addresses will be scanned via Proofpoint Email Security

Proofpoint@ICS

Jupyterhub has updated with newer version of R and Python.

What's new:

• R-Studio 2023.06.1+524
• R version 4.3.1 (2023-06-16) – “Beagle Scouts”
• Python 3.11.4

Personal libraries will need to be recompiled to work with the new version of R and Python.

We have had reports of issues sending mail for UCI GMail accounts from Apple Mail via UCI SMTP servers.

If you are experiencing this issue please see if these instructions will fix the issue.

The instructions will inform you how to remove and re-add the account. This should set up your outgoing SMTP to use Google SMTP servers

Subject: ICS Proofpoint Deployment

Date: TBD

In order to conform with the UCI Campus Email Security Initiative all mail addressed to @ics.uci.edu email accounts, or any subdomain, will be scanned by ProofPoint and run through URL Defense.

ICS Proofpoint will improve spam, malware, and phishing detection. Campus has found that it does better with the “commodity” spam than the current MailScanner/SpamAssassin products that ICS is using. URL Defense has also helped in blocking and containing ongoing phishing campaigns.

ICS Mailscanner and spamassassin will be turned off. Any filters that rely on current mail headers (eg. spamscore=sssss) will no longer function.

Testing is available now. Any mail sent to your_username@pp-mail.ics.uci.edu will route through the ICS Proofpoint service and will be delivered into your ICS inbox. Feel free to use this alternate address to test the new service.

ICS users with a delivery point set to your ICS email account will be able to manage both their ICS and UCI Proofpoint settings via the OIT Proofpoint Portal

Check or change your UCI Email Delivery Point

Proofpoint Phase 2: Advanced Email Content Protection FAQ

In an effort to prepare to move to proofpoint ICS Computing Support will be turning off Spamhaus anti-spam measures for delivery to any @ics.uci.edu addresses. Users that read mail through GMail Web Interface will not be impacted. Users with mail that is delivered locally and read through IMAP clients such as Thunderbird, may notice an increase of spam mail. Please remain vigilant

Beginning the week of 5/23/2023, ICS Computing Support will enable DKIM signing on outgoing mail.

• ICS Email Services and Settings
• Wikepedia DomainKeys Identified Mail

ICS will be using the openDKIM package on its mailservers. Incoming messages that do not have a signature that matches the sender's DNS will be deferred. Outgoing messages will all be signed with the ICS signing key and will match the information in the ICS DNS.

Please contact helpdesk@ics.uci.edu or call x44222 to report problems and concerns.

Openlab compute resources are typically available to any user with and ICS shell account. Priority is given to instruction. Instructors may reach out to heldpesk@ics.uci.edu with questions or special requests for courses.

Using Jupyterhub @ ICS

Goto https://hub.ics.uci.edu

Jupyterlab@ICS turns any web browser into an Ubuntu 22.04LTS work station offering Jupyter Notebook, the lastest (Jupyter) VS Code, the latest Rstudio IDE, an X11 desktop and a direct connection to your ICS home direcotry

Using Slurm @ ICS

Slurm is an open source, fault-tolerant, and highly scalable cluster management and job scheduling system for large and small Linux clusters. Users may submit long running and massive serial jobs for processing on the Openlab Linux Cluster via slurm.

Using [https://swiki.ics.uci.edu/doku.php/hardware:cluster:openlab|ICS Instruction Openlab Linux Server]]

The Openlab Linux Cluster is a cluster of nearly 100 donated x86_64 compute nodes runnig Ubuntu 22.04LTS. This cluster is for general purpose computing at the School of ICS and accessible to any user with an ICS shell account with priority given to instruction.

The following policy change has been implemented for spring quarter:

• Files in the /tmp directory will be cleared after two weeks.
• Processes running for longer than 2 hours will be reniced 19.
• Processes running for longer than 5 days will be suspended.
• Processes for users with more than 100 processes will be suspended.

Long running jobs should be submitted via slurm. Long running jobs submitted via slurm are not subject to suspension.

About your ICS Account

Anyone associated with ICS will receive an ICS shell account. Your ICS shell account has the same username as your UCINedID but it will have a distinct password and can only provide access to ICS resources, such as Openlab Jupyter Hub or the Openlab Linux Cluster.

Every ICS account is associated with an ICS NFS (Network File Service) home directory.

• ICS NFS home directories are shared between computers.
• Secure Storage
• Self-serve file recovery is provided for 30 days via a daily snapshot backup.
• The space is copied to alternate storage to provide a long term disaster recovery option.

ICS NFS home directories are, typically, mounted by default on ICS managed or trusted Windows and Linux computers.

Users on unmanaged or untrusted computers may mount the ICS home directory via SSHFS and SMBD

In addition to your NFS home directory, the following, unquota'd spaces are available for ICS Graduate and Undergraduate students

• Grad students may use Graduate Space
• Undergrad students may use [https://swiki.ics.uci.edu/doku.php/hardware:storage:ugrad_space|ugrad_space]]

Setup your SSH Keys

Some ICS compute servers allow direct ssh connections from off campus without requiring the campus VPN. These compute servers, including openlab.ics.uci.edu, will make use of SSH keys in place of passwords. When using ssh-keys, you can also use the ssh-agent to automatically pass credentials and remove the need to type your password when logging into a host. Your ssh-keys can be used with any ssh-client.

Using Linux Group Accounts @ ICS

Formerly called groupleader and gsu accounts, these accounts allow processes and files to be shared between team members or within research groups.

Group accounts:

• Allow access to a shared NFS home directory from any computer.
• Grant access to shared ICS space that resides in ICS NFS servers
• Are backed up (see: snapshots)
• Can be created to leverage storage owned by research teams

When choosing cloud computing please consider the following:

• Cloud Cost vs.ICS Data Center (i.e.when is cloud less expensive than hardware in our DC).
• Hidden costs that can lock you in such as data egress.
• How to manage access and service continuity when teams change and students graduate.
• How much support do you need?

Engaging GCP @ ICS

ICS Computing Support can provide limited support to research teams setting up GCP projects including billing, access, and connection to ICS Data Center resources Please reach out to helpdesk@ics.uci.edut with any questions.

OIT provides researchers managed and self-supported platforms. Amazon offers campus discounts on virtual machines and storage. Feel free to reach out to ICS Computing Support to find out how we can help or goto the OIT Campus AWS site.

Use Vagrant/Oracle Virtual Box @ ICS

Oracle VM Virtual Box is a powerful x86 and AMD64 virtualization project. Virtual Box is availableon the Archimedes Linux Cluster for faculty and researchers and on other hosts upon request.

Vagrant is a command line utility for managing Virtual Box VM that is included on ICS hosts running Virtual Box.

Use Singularity @ ICS

Singularity is a widely-adopted container runtime that implements a unique security model to mitigate privilege escalation risks and provides a platform to capture a complete application environment into a single file (SIF). Both the ICS HP compute cluster and the UCI HP computer cluster run slurm and singularity containers for users to manage workloads.

Using Docker @ ICS

Due to docker's (lack of a) security model, availability is limited to some specific use cases for research groups that provide their own equipment. Please use singularity if at all possible, or send mail to helpdesk@ics.uci.edu to inquire about enabling docker on your equipment.

Kubernetes @ ICS has limited availability.

The ICS Computing Support group does run some small kubernetes installations and may have capacity to run some additional research related workloads. Please send mail to heldpesk@ics.uci.edu for more information.

Using slurm @ ICS

Slurm is an open source, fault-tolerant, and highly scalable cluster management and job scheduling system for large and small Linux clusters. Both the ICS HP compute cluster and the UCI HP compute cluster use slurm to distribute jobs across compute nodes. While ICS Computing Support manages several research clusters with slurm that are restricted to research group members, everybody at ICS is able to use the Openlab Slurm Cluster and the Openlab GPU Slurm Cluster.

The Fair Tree Fairshare Algorithm has been enabled and small job priority has been increased on the Openlab partitions in order to distribute consumable resources to as many users as possible and avoid situations where monolithic jobs lock up the cluster for hours and days at a time.

The Openlab GPU Openlab cluster is now slurm-only. This means that users may no longer ssh directly into servers that are part of Openlab GPU slurm partition and start processes outside the slurm scheduler. Users may still run a bash shell via srun in order to test or compile programs: `srun -N1 -n1 -p openlab.p bash -i`

ICS Computing Support is implementing the Kerberos authentication service for research groups that require docker or elevated privileges on their local machine. For complete details goto:

https://wiki.ics.uci.edu/doku.php/services:kerberos

Grad students are invited to help us test our Kerberos staging environment.

== Getting Started: ==

1. Goto https://kpassword.ics.uci.edu to activate your Kerberos login.
2. Connect to runamuck.ics.uci.edu via ssh using your ICS Kerberos login (not ssh key)
3. Grad students may test our new Kerberos5 Authenticated nodes:
   • runamuck.ics.uci.edu (Ubuntu 22.04LTS)
   • runabout.ics.uci.edu (CentOS7)

• Safe environment for researchers to run docker.
• Access to ICS home and extra directories via NFSv4 and sec=krb5p
• Expanded sudo access to researchers.
• Grads will have access to many sudo commands including docker, most commands in /bin and selected commands in /sbin. Use `sudo -l` to determine what commands are available to you.

ICS Computing Support provides the Community Edition of Gitlab.

More information about use Gitlab @ ICS

Goto https://gitlab.ics.uci.edu

Custom gitlab resources can be provided for instruction. Please send email to helpdesk@ics.uci.edu

UCI GitHub Enterprise is available to all Campus Faculty, Student, and Staff. It is only available on campus. For more information see.

Read more about OIT's offering here: https://www.oit.uci.edu/services/communication-collaboration/uci-github/

Recent changes to Campus licensing for Microsoft office can be found at the following link:

https://www.oit.uci.edu/services/end-point-computing/microsoft-365/

Campus has indicated that campus enterprise Adobe Creative Cloud license will be discontinued effective 7/1/2023. ICS will cover the licensing cost for Adobe Acrobat Pro for Faculty/Staff. Creative Cloud users will need to make other arrangements and campus is currently working on a new ordering process. The relevant message from the OIT follows.

The campus enterprise Adobe Creative Cloud license will be discontinued effective 7/1. This 
license was originally purchased during the COVID pandemic with federal aid funds that we 
are not able to continue given the budget challenges to the campus. School computing labs that 
need to continue a classroom computer license should make those requests via eTech. Orders 
for other single user licenses of Adobe products such as Acrobat Pro and Creative Cloud will 
follow a new process analogous to the old process via the system’s new software VAR, Dell. 

UC Irvine is participating in the InCommon Certificate program, which allows delegated administrators in campus departments to issue and renew digital certificates used for such purposes as securing web servers run on behalf of their department. Through the InCommon Certificate program, UC Irvine pays a site fee (sponsored by OIT) and is then entitled to issue unlimited digital certificates through Comodo, a well-established commercial Certificate Authority. More information about this program is available at http://www.incommonfederation.org/cert.

ICS Community members may request SSL Certificates from ICS Computing Support. We can create wildcard certificates, multi-domain certificates, or single site certificates. All certificates are good for 398 days and can be renewed annually.

ICS Community members may also self-support using certbot certificates (formerly named letsencrypt). Contact ICS Computing Support to get help configuring ICS DNS for the certbot DNS challenge.

Users are not using SLURM to request resources and share the usage of POISON, the openlab GPU server. To properly distribute and share POISON's resources, SSH access to the server will be disabled beginning Spring Quarter.

The server will accept SLURM batch jobs via an sbatch script and interactive shells via srun. All interactions will be via SLURM commands. Requesting a shell through SRUN will be the closest to logging into POISON directly but you will need to specify what resources you want.

In order to ensure that Openlab cluster remain available to instruction and researchers fairly, we will be making the following changes to the administration of the cluster:

• Files in the /tmp directory will be cleared after two weeks.
• Processes running for longer than 2 hours will be reniced 19.
• Processes running for longer than 48 hours will be suspended.
• Processes for users with more than 100 processes will be suspended.

Please use slurm for long running or serial projects requiring more than one openlab node. Processes running through slurm are not subject to the above guidelines.

Instructors may reach out to heldpesk@ics.uci.edu in order to request course exceptions to any of these guidelines.

Jupyterhub is available for students and researchers during Spring quarter. Users logging into one of the Jupyterhub URL's will receive a personal docker container running Ubuntu 22_04LTS.

https://hub.ics.uci.edu: General purpose hub intended for Winter quarter instruction(Linux shell, VSCode IDE, X11 Desktop)

What's new:

• Ubuntu 22.04LTS
• Expanded sudo access including apt install. Run `sudo -l` for a complete list. Send additional requests to heldpesk@ics.ucie.du
• R-Studio is back (v2022.12.0+353)
• R v4.2.2
• VSCode update (Coder Version 4.10.0)
• 8 core/16 gigabyte machine available for data science

Jupyterhub uses ICS home directory to provide access to persistent, backed up, data across all ICS hosts. Multiple server instances may be started. Hub containers will run indefinitely when logged into at least once per week.

OIT sent messages last year regarding adding ProofPoint scanning and quarantine to reduce campus risk related to email. Recently, users have let us know some of their emails are not making it to their inboxes. The emails are typically in the users quarantine or ProofPoint has tagged them as Spam. This has happened most often when a user forwards their email from @ics.uci.edu to @uci.edu.

We recommend sending emails to @uci.edu email addresses. This eliminates multiple server hops which can trigger a false spam report.

If you believe you are missing emails, we recommend you check your quarantine. Here is OIT's webpage with information on ProofPoint and how to check quarantine:

https://www.oit.uci.edu/services/communication-collaboration/proofpoint/

For those users who forward their ICS email to their UCI email address, you can set ics.uci.edu as a safe sender in ProofPoint.

• Login to ProofPoint (link available on webpage above)
• Click on New
• Type: ics.uci.edu
• Click Save

If you are forwarding your UCI email to ICS, it is not necessary to safelist your @uci.edu. If you try, you will receive the following message:

Cannot safelist or blocklist your own email address or domain.

ICS Computing Support it Testing a kerberos proof of concept cluster. For complete details goto:

• Kerberos@ICS

Getting Started:

• Goto https://kpassword.ics.uci.edu to activate your Kerberos login.
• Connect to runamuck.ics.uci.edu via ssh using your ICS Kerberos login (not ssh key)

Grad students may test our new Kerberos5 Authenticated nodes:

• runamuck.ics.uci.edu (Ubuntu 22.04LTS)
• runabout.ics.uci.edu (CentOS7)

Goals:

• Safe environment for researchers to run docker.
• Access to ICS home and extra directories via NFSv4 and sec=krb5p
• Expanded sudo access to researchers.

Grads will have access to many sudo commands including docker, most commands in /bin and selected commands in /sbin.  Use `sudo -l` to determine what commands are available to you.   Please review the following page on the best way to use sudo at ICS.

• Sudo@ICS

Send questions, suggestions to helpdesk@ics.uci.edu

Researchers and students may now fine GCP and AWS cloud computing utilities on ICS maanged Linux servers.

• awscli
• gcloud

You may send mail to heldpesk@ics.uci.edu to find out how ICS Computing Support can help researchers run cloud computing.

Releated ICS resources:

• Google Cloud