Category Archive: Incident Notification

SDSC Outage Notification – Sharepoint slow, some features unavailable – 12:00, 17 May 2017

[Update : 14:38] Engineers have found a number of stuck Sharepoints jobs from last night and are attempting to kill those jobs. Performance and feature outages are still unchanged. —- At approximately noon on 17 May 2017 we discovered that the hosted Sharepoint services were slow for rendering pages, and certain features like search were …

Continue reading »

SDSC Outage Notification – virtual machine guests – 6 Mar 2017, 10:00

[Update – 6 Mar 2017, 10:25] Networking has been fixed and virtual guests are back online. Engineers are continuing to investigate any remaining issues. — At approximately 10:00 on 6 Mar 2017 during routine network recabling on a redundant link, a VMWare hypervisor node stopped responding. System engineers are investigating and updates will be posted. …

Continue reading »

SDSC Outage Notification – Oracle services – 19 Jan 2017, 19:50

[Update : 20:56] The cause was found. During routine maintenance an assumed-unused package was removed. Once the package was re-added, Oracle began working again. Services are up — please contact support for any continuing issues. — At 19:50 on 19 Jan 2017 the primary Oracle nodes serving the SDSC Footprints system and SAM/ART accounting started …

Continue reading »

SDSC Outage Notification – limited Linux guest outage – 14 Dec 2016, 19:45

[Update, 00:05, 15 Dec 2016] guest ‘rupee’ is now online. [Update, 23:37] All guests except ‘rupee’ are online. Please contact support with any issues or concerns [Update, 22:05] Engineers replaced the faulty disk controller and verified the guests function. Guests were then shut down again so that engineers can perform a reboot and ensure the …

Continue reading »

SDSC Outage Notification – CDS2 / West UPS Power – 3 Nov 2016, 10:40

[Update – 20:14] The UPS has been reactivated and is protecting the systems. —– At approximately 10:40 the CDS2 UPS system lost power due to a short circuit condition. The power was immediately restored with the UPS system being temporarily bypassed. Updates will be forthcoming. The scope of the outage would be computer which were …

Continue reading »

SDSC Emergency Maintenance – Linux patch/reboot – 1 Nov 2016, 20:00-23:00

[Update – 20:41] All patches and reboots have been applied. —- SDSC will be applying critical patches to the Linux environment tonight, 1 Nov 2016 starting at 8pm. This maintenance will require a reboot of all systems listed below. Please contact with any questions or concerns. Updates will be posted to as the …

Continue reading »

SDSC Outage Notification – Project Storage – 11:30-12:00, 19 Aug 2016 [Resolved]

The SDSC Project Storage Service experienced a partial outage from approximately 11:30AM – 12:00PM. A hung process on a single hotel node consumed a critical amount of processor and memory resources, effectively rendering the node unavailable during the outage. At this time the server is back up and functioning as expected.  We apologize for any inconvenience caused by …

Continue reading »

SDSC Outage Notification – Commvault – 21:00, 29 Feb 2016

[Update: 23:06, 29 Feb 2016] Deduplication stores have been sealed and systems requiring the deduplication are running backups. At approximately 21:00 on 29 Feb 2016 the Commvault media agent named ‘cvma3’ hung and was power cycled. This caused jobs which were running to pause. There were some jobs which required manual restarts. Some jobs utilizing …

Continue reading »

SDSC Outage Notification – Cloud Storage & Compute – 4 Feb 2016, 12:39

[Update, 13:19] Service has been restored. A 10Gb cable was inadvertently removed which provided service to the load balancer. That cable has been replaced. The loadbalancer which provides access to both SDSC Cloud Storage and SDSC Cloud Compute has gone offline. Engineers are investigating.

SDSC Outage Notification – Datacenter Partial Power Loss ~16:00, 6 Jan 2016

Key services have been restored. Please contact datacenter operations or your SDSC system administrators if you are experiencing any lingering issues. Update [23:00]: All commvault services have been restored. Update [20:35]: Commvault backups from cvma4 are working. The media agent cvma3 is still down and operations is assisting in rebooting/recovery of the system. Update [19:30]: …

Continue reading »

Older posts «