[Update – 20:14] The UPS has been reactivated and is protecting the systems. —– At approximately 10:40 the CDS2 UPS system lost power due to a short circuit condition. The power was immediately restored with the UPS system being temporarily bypassed. Updates will be forthcoming. The scope of the outage would be computer which were …
Category Archive: Incident Notification
SDSC Emergency Maintenance – Linux patch/reboot – 1 Nov 2016, 20:00-23:00
[Update – 20:41] All patches and reboots have been applied. —- SDSC will be applying critical patches to the Linux environment tonight, 1 Nov 2016 starting at 8pm. This maintenance will require a reboot of all systems listed below. Please contact support@sdsc.edu with any questions or concerns. Updates will be posted to http://status.sdsc.edu as the …
SDSC Outage Notification – Project Storage – 11:30-12:00, 19 Aug 2016 [Resolved]
The SDSC Project Storage Service experienced a partial outage from approximately 11:30AM – 12:00PM. A hung process on a single hotel node consumed a critical amount of processor and memory resources, effectively rendering the node unavailable during the outage. At this time the server is back up and functioning as expected. We apologize for any inconvenience caused by …
SDSC Outage Notification – Commvault – 21:00, 29 Feb 2016
[Update: 23:06, 29 Feb 2016] Deduplication stores have been sealed and systems requiring the deduplication are running backups. At approximately 21:00 on 29 Feb 2016 the Commvault media agent named ‘cvma3’ hung and was power cycled. This caused jobs which were running to pause. There were some jobs which required manual restarts. Some jobs utilizing …
SDSC Outage Notification – Cloud Storage & Compute – 4 Feb 2016, 12:39
[Update, 13:19] Service has been restored. A 10Gb cable was inadvertently removed which provided service to the load balancer. That cable has been replaced. The loadbalancer which provides access to both SDSC Cloud Storage and SDSC Cloud Compute has gone offline. Engineers are investigating.
SDSC Outage Notification – Datacenter Partial Power Loss ~16:00, 6 Jan 2016
Key services have been restored. Please contact datacenter operations or your SDSC system administrators if you are experiencing any lingering issues. Update [23:00]: All commvault services have been restored. Update [20:35]: Commvault backups from cvma4 are working. The media agent cvma3 is still down and operations is assisting in rebooting/recovery of the system. Update [19:30]: …
SDSC/UCSD Intermittent Network Interruptions – 23 Aug 2015
Connectivity to UCSD and SDSC networks from the outside internet has been intermittently unavailable today, 23 August 2015. UCSD network engineers are working to resolve the issues. Additional information will also be available at http://status.ucsd.edu.
SDSC Outage Notification – Datacenter and building networking – 13:00-13:01, 18 Aug 2015
During a routine hardware swap on one side of a redundant networking switch pair, the network became unresponsive for approximately 60 seconds. The outage has cleared however some connections into or out of the datacenter may have been affected. If you continue to see any networking issues, please contact Datacenter Operations or your designated service …
SDSC Outage Notification – cavern UNIX NFS / UNIX reference systems – 11:09, 28 July 2015
Update, 13:12, 28 July 2015: The ‘cavern’ fileserver was frozen. System engineers performed a reboot and service was restored at 11:32. We are investigating the cause of the outages. At approximately 1109 the reference systems relying upon ‘cavern’ NFS have frozen. System engineers are investigating the outage and we will post updates here.
SDSC Outage Notification – VMware – 21 May 2015, 13:53:00 PST
Description: Brief VMware outage Start Time: 13:53:00 PST End Time: 14:06:00 PST Date: 2015-05-21 A CPU error on one hypervisor caused its VMs to reboot around 14:01 PST after the VMs were automatically migrated by VMware High Availability to a fully functional hypervisor in order to restore service. Affected VMs: sc-osi-ops wcga-vm02 opentopo-vm rsm-bizanalytics delphidata rsm-sp2007-dev sc-pa-dev sc-osi-ops2-2 …