Update, 13:12, 28 July 2015: The ‘cavern’ fileserver was frozen. System engineers performed a reboot and service was restored at 11:32. We are investigating the cause of the outages. At approximately 1109 the reference systems relying upon ‘cavern’ NFS have frozen. System engineers are investigating the outage and we will post updates here.
Category Archive: Incident Notification
SDSC Outage Notification – VMware – 21 May 2015, 13:53:00 PST
Description: Brief VMware outage Start Time: 13:53:00 PST End Time: 14:06:00 PST Date: 2015-05-21 A CPU error on one hypervisor caused its VMs to reboot around 14:01 PST after the VMs were automatically migrated by VMware High Availability to a fully functional hypervisor in order to restore service. Affected VMs: sc-osi-ops wcga-vm02 opentopo-vm rsm-bizanalytics delphidata rsm-sp2007-dev sc-pa-dev sc-osi-ops2-2 …
SDSC Outage Notification – Networking Issues – 27 Feb 2015
Title: Networking Issues – 27 Feb 2015Description: Update 27 Feb 2015, 20:11: Service has been restored. A loose interconnect cable on a switch group found to be cause. Update 27 Feb 2015, 19:16: Services such as SDSC IMAP, reference system UNIX VM, Oracle database, MySQL database, continue to be unavailable. SDSC Sharepoint and VMWare services …
SDSC Outage Notification – 27 Jan 2015, 00:20 – Networking
Title: SDSC Outage Notification – 27 Jan 2015, 00:20 – Networking Update 27 Jan 2015, 05:55: Internet Service Provider maintenance has been completed and service is fully functional. Please contact support or SDSC operations for assistance with any service outage that may still occur. Description: Internet-facing network connectivity to SDSC resources may be down, slow …
SDSC Outage Report – 13 January 2015, 0530-0610 – Project Storage
Title: SDSC Outage Report – 13 January 2015, 0530-0610 – Project StorageDescription: SDSC Project Storage encountered an outage this morning (Tuesday 2015.01.13) between 5:30A-6:10A. If you had jobs running during that time please verify that they completed successfully. The following NFS and Samba shares were affected: ps-017.sdsc.edu:/ps-data/bglab ps-027.sdsc.edu:/ps-data/brain_observatory ps-002.sdsc.edu:/ps-data/camera ps-030.sdsc.edu:/ps-data/colemanlab ps-018.sdsc.edu:/ps-data/cwphs ps-032.sdsc.edu:/ps-data/dom ps-013.sdsc.edu:/ps-data/dsc ps-001.sdsc.edu:/ps-data/lca ps-003.sdsc.edu:/ps-data/matta2 …
SDSC Outage Notification – 10 Dec 2014, 17:00-17:33 – Core NFS services
Title: SDSC Outage Notification – 10 Dec 2014, 17:00-17:33 – Core NFS servicesDescription: At approximately 17:00, the core NFS fileserver ‘cavern’ became unresponsive. Engineers successfully rebooted the server and restored service at 17:33. Services potentially affected: – hosted web services – managed Linux virtual machines – managed Linux physical systems – incoming @sdsc.edu emailStart Time: …
SDSC Outage Notification – VMWare – 4 Nov 2014, 19:00
Title: SDSC Outage Notification – VMWare – 4 Nov 2014, 19:00 Update: 5 Nov 2014, 01:35 All services have been restored. Please contact SDSC support with any questions or concerns. Update: 5 Nov 2014, 01:20 Citrix, Sharepoint and Printing services have been restored. Update: 5 Nov 2014, 01:15 Commvault services have been restored. Linux guests …
SDSC Outage Notification – mysql – 2 Nov 2014, 01:30
Title: SDSC Outage Notification – mysql – 2 Nov 2014, 01:30Update 7:15, 2 Nov 2014: All services have been restoredDescription: SDSC is experiencing an outage of the production mysql services on ‘mysql2.sdsc.edu’. Engineers are investigating.Date: 2014-11-02
SDSC Outage Notification – Network – 31 Oct 2014, 13:15
Title: SDSC Outage Notification – Network – 31 Oct 2014, 13:15Description: SDSC networking experienced a brief outage around 13:15 on 31 Oct 2014. The outage lasted approximately five minutes. The networking team is investigating and we will post any additional information.Start Time: 13:15Date: 2014-10-31End Time: 13:20
SDSC MySQL Service Outage – 11:30, 20 Oct 2014
Title: MySQL Service Outage – 11:30, 20 Oct 2014Description: Database services on production MySQL may be unavailable while we restart the database to correct out of resource issues. We will update as we have more information.Start Time: 11:30Date: 2014-10-20