Helpdesk Ticket #101784

Brief flucuation in power, observed as lights blinking once, caused devices in data center to reset which led to widespread loss of services for approximately 5 minutes. The first email report arrived at 11:11am but may have been delayed due to down services. Services were significantly restored by 11:16. Twenty fives hosts were found to have lost power down during the event and it was necessary to power them up manually.

• Time Begun: ~11am
• Time Ended: 11:16 am

The following equipment looks like it reset

• 3023-c2960s-095.ics.uci.edu
• 3023-c2960g-080.ics.uci.edu
• 3023-c2960g-086.ics.uci.edu
• 3023-c2960g-086.ics.uci.edu
• 3023-c2950t-099.ics.uci.edu

Hi all, please be advised that we are currently experiencing a widespread power outage on the UCI campus.  Emergency Management will continue to monitor this event.  This email is just a heads-up and there is no CCT call planned at this time, but we may need to initiate one if the event continues. 
...
Appears  Bio Sci, Bren, and other areas.
...
Please stand by.

Dear ISEB Occupants,

We want to inform you that there is currently a power surge affecting our building and the campus as a whole. While all electronics should still be operational thanks to our backup generator, we have noticed that the autoclave is displaying a warning in some of the labs.

As a precaution, we advise your teams to proceed with caution and temporarily pause any experiments within the fume hood until UCI Facilities can resolve the issue Or until the autoclave no longer gives you the warning. In addition to that, you will also feel the building’s central air system working at a different rate which can affect the building’s air pressure. Please take extra precautions when opening and closing doors.

We appreciate your cooperation and will provide updates as soon as we have more information. Thank you for your patience.

• All ICS managed services and research located in the data center was affected
• All ICS stakeholders
• RCIC/CRSP (Imam Toufique @ RCIC)

• Link any relevant helpdesk and project tickets.
• netsail-4
• arcala-1
• datalab-12
• arcus-19
• Cantor
• Dizzy

• wdcohen
• vipada
• dutran
• hans

• Power recovered on its own.

• Determine which equipment was impacted by the outage

• The following equipment has failed:
  • zen.ics.uci.edu - disk
  • odin.ics.uci.edu - power supply
• Fix Power on Gravity
• Fix MySQL on sensoria-2