The SDSC East Building is experiencing intermittent network outages. UCSD Networking and Datacomm is aware and working on the issue. Users in the East Building may see outages to wired and wireless networking.
Category Archive: Incident Notification
SDSC Outage Notification – SDSC Cloud Compute – 17 Jun 2019, 14:30
(update 18:37) All issues have been resolved. Please contact support with any concerns or questions. (update 17:34) Networking has been restored. Some remaining issues with load balanced applications may be seen. The SDSC Cloud Compute service is experiencing network outages that engineers are investigating. More information will be posted here.
SDSC Outage – M47 network switch – 5/24 18:00
[update 22:16] The new switch is online and systems appear to be recovered. Please contact SDSC operations with any questions or concerns [update 20:32] engineers are on site swapping a chassis. No recovery ETA at this point The top of rack switch in data center rack M47 is offline. Some data center hosted systems are …
SDSC Outage Announcement – HPC, Commvault, Cloud resources unavailable – 6 May 2019, 18:00
[Update – 11:18] Networking service has been restored and appears stable. Please contact support if you continue to experience any issues. More updates will be posted once a cause is determined. [Update – 07:53] Commvault services are enabled and working — file restores are successful. [Update – 07:42] The reboot did not fully correct the …
SDSC Outage Announcement – Data Center Networking – 22 Feb 2019, 12:10
[Update: 12:15] Services have been restored. One switch group which feeds a number of datacenter racks rebooted. Engineers are still investigating the cause. The scope of outage included numerous web and database services and colocated server networking. —- There has been a network outage reported at SDSC. The scope of the outage is not yet …
SDSC Outage Notification – Firewall Network Issues – 26 Nov 2018, 14:35-14:50
On 26 November 2018 between roughly 14:30 and 14:50, systems behind the SDSC firewalls experienced limited network connectivity. Engineers have corrected the issue and systems are accessible. Please contact SDSC Operations or regular support channels with further questions.
SDSC Outage Notification – Project Storage ps-078:/ps-data/yeolab – 25 Sept 2018, 10:45
[Update – 11:19] The JBOD was brought back online and verified. The JBOD storage for ps-078:/ps-data/yeolab has faulted and the array is offline. Engineers are investigating and working to repair the array.
SDSC Outage Notification – Power Outage Resolved – 30 Aug 2018, 08:15
Update 8/30/2018 10:00 AM PT: A general power outage occurred this morning, Thursday, 8/30/2018, at SDSC around 8:10 AM PT. Power was restored by campus FM around 9:40 AM PT. Individual systems in the data center may need to be powered back on if they were affected by the outage. Generator-backed systems should have remained powered …
SDSC Post Incident Report – VMware Outage, 15 Aug 2018
On Wednesday, 15 August 2018, at approximately 1pm, the primary VMware cluster at SDSC experienced a complete outage. During the outage all VM guests were unable to access disk resources, and eventually needed to be powered off to allow cluster recovery. At 10pm the cluster was stable, at which point all VM guests were placed …
SDSC Outage Notification – multiple networks in building/datacenter – 15 Aug 2018, 13:15
[Update – 22:59] All guests are online and appear to be acting normally. Please contact support with any questions or concerns. [Update – 22:00] Engineers are slowly booting guests and monitoring stability of the VMware cluster. Additional updates will be posted once all guests are on and the environment is operating normally. [Update – 18:42] …