Category Archive: Incident Notification

SDSC Outage Notification – Cloud Storage & Compute – 4 Feb 2016, 12:39

[Update, 13:19] Service has been restored. A 10Gb cable was inadvertently removed which provided service to the load balancer. That cable has been replaced. The loadbalancer which provides access to both SDSC Cloud Storage and SDSC Cloud Compute has gone offline. Engineers are investigating.

SDSC Outage Notification – Datacenter Partial Power Loss ~16:00, 6 Jan 2016

Key services have been restored. Please contact datacenter operations or your SDSC system administrators if you are experiencing any lingering issues. Update [23:00]: All commvault services have been restored. Update [20:35]: Commvault backups from cvma4 are working. The media agent cvma3 is still down and operations is assisting in rebooting/recovery of the system. Update [19:30]: …

Continue reading »

SDSC/UCSD Intermittent Network Interruptions – 23 Aug 2015

Connectivity to UCSD and SDSC networks from the outside internet has been intermittently unavailable today, 23 August 2015. UCSD network engineers are working to resolve the issues. Additional information will also be available at http://status.ucsd.edu.

SDSC Outage Notification – Datacenter and building networking – 13:00-13:01, 18 Aug 2015

During a routine hardware swap on one side of a redundant networking switch pair, the network became unresponsive for approximately 60 seconds. The outage has cleared however some connections into or out of the datacenter may have been affected. If you continue to see any networking issues, please contact Datacenter Operations or your designated service …

Continue reading »

SDSC Outage Notification – cavern UNIX NFS / UNIX reference systems – 11:09, 28 July 2015

Update, 13:12, 28 July 2015: The ‘cavern’ fileserver was frozen. System engineers performed a reboot and service was restored at 11:32. We are investigating the cause of the outages. At approximately 1109 the reference systems relying upon ‘cavern’ NFS have frozen. System engineers are investigating the outage and we will post updates here.

SDSC Outage Notification – VMware – 21 May 2015, 13:53:00 PST

Description: Brief VMware outage Start Time: 13:53:00 PST End Time: 14:06:00 PST Date: 2015-05-21 A CPU error on one hypervisor caused its VMs to reboot around 14:01 PST after the VMs were automatically migrated by VMware High Availability to a fully functional hypervisor in order to restore service. Affected VMs: sc-osi-ops wcga-vm02 opentopo-vm rsm-bizanalytics delphidata rsm-sp2007-dev sc-pa-dev sc-osi-ops2-2 …

Continue reading »

SDSC Outage Notification – Networking Issues – 27 Feb 2015

Title: Networking Issues – 27 Feb 2015Description: Update 27 Feb 2015, 20:11: Service has been restored. A loose interconnect cable on a switch group found to be cause. Update 27 Feb 2015, 19:16: Services such as SDSC IMAP, reference system UNIX VM, Oracle database, MySQL database, continue to be unavailable. SDSC Sharepoint and VMWare services …

Continue reading »

SDSC Outage Notification – 27 Jan 2015, 00:20 – Networking

Title: SDSC Outage Notification – 27 Jan 2015, 00:20 – Networking Update 27 Jan 2015, 05:55: Internet Service Provider maintenance has been completed and service is fully functional. Please contact support or SDSC operations for assistance with any service outage that may still occur. Description: Internet-facing network connectivity to SDSC resources may be down, slow …

Continue reading »

SDSC Outage Report – 13 January 2015, 0530-0610 – Project Storage

Title: SDSC Outage Report – 13 January 2015, 0530-0610 – Project StorageDescription: SDSC Project Storage encountered an outage this morning (Tuesday 2015.01.13) between 5:30A-6:10A. If you had jobs running during that time please verify that they completed successfully. The following NFS and Samba shares were affected: ps-017.sdsc.edu:/ps-data/bglab ps-027.sdsc.edu:/ps-data/brain_observatory ps-002.sdsc.edu:/ps-data/camera ps-030.sdsc.edu:/ps-data/colemanlab ps-018.sdsc.edu:/ps-data/cwphs ps-032.sdsc.edu:/ps-data/dom ps-013.sdsc.edu:/ps-data/dsc ps-001.sdsc.edu:/ps-data/lca ps-003.sdsc.edu:/ps-data/matta2 …

Continue reading »

SDSC Outage Notification – 10 Dec 2014, 17:00-17:33 – Core NFS services

Title: SDSC Outage Notification – 10 Dec 2014, 17:00-17:33 – Core NFS servicesDescription: At approximately 17:00, the core NFS fileserver ‘cavern’ became unresponsive. Engineers successfully rebooted the server and restored service at 17:33. Services potentially affected: – hosted web services – managed Linux virtual machines – managed Linux physical systems – incoming @sdsc.edu emailStart Time: …

Continue reading »

Older posts «

» Newer posts