Category Archive: Incident Notification

SDSC Outage Notification – (Sharepoint) – 21:12, 19 Oct 2017

[Update: 22:43, 19 Oct 2017] The Sharepoint site is now up and sites have been verified. —- The SDSC Sharepoint site,, is unavailable. Engineers are investigating the cause.

SDSC Incident Alert – Partial West Datacenter Power Outage – 11:40, 19 Oct 2017

[Update: 12:32, 19 Oct 2017] Power has been restored and services restored. —- A power circuit breaker has failed causing loss of power to six racks in the West datacenter. The affected rack customers include: LJI UCLA UCSD Library Campus electricians have been notified.

SDSC Outage Notification – Sharepoint slow, some features unavailable – 12:00, 17 May 2017

[Update : 14:38] Engineers have found a number of stuck Sharepoints jobs from last night and are attempting to kill those jobs. Performance and feature outages are still unchanged. —- At approximately noon on 17 May 2017 we discovered that the hosted Sharepoint services were slow for rendering pages, and certain features like search were …

Continue reading »

SDSC Outage Notification – virtual machine guests – 6 Mar 2017, 10:00

[Update – 6 Mar 2017, 10:25] Networking has been fixed and virtual guests are back online. Engineers are continuing to investigate any remaining issues. — At approximately 10:00 on 6 Mar 2017 during routine network recabling on a redundant link, a VMWare hypervisor node stopped responding. System engineers are investigating and updates will be posted. …

Continue reading »

SDSC Outage Notification – Oracle services – 19 Jan 2017, 19:50

[Update : 20:56] The cause was found. During routine maintenance an assumed-unused package was removed. Once the package was re-added, Oracle began working again. Services are up — please contact support for any continuing issues. — At 19:50 on 19 Jan 2017 the primary Oracle nodes serving the SDSC Footprints system and SAM/ART accounting started …

Continue reading »

SDSC Outage Notification – limited Linux guest outage – 14 Dec 2016, 19:45

[Update, 00:05, 15 Dec 2016] guest ‘rupee’ is now online. [Update, 23:37] All guests except ‘rupee’ are online. Please contact support with any issues or concerns [Update, 22:05] Engineers replaced the faulty disk controller and verified the guests function. Guests were then shut down again so that engineers can perform a reboot and ensure the …

Continue reading »

SDSC Outage Notification – CDS2 / West UPS Power – 3 Nov 2016, 10:40

[Update – 20:14] The UPS has been reactivated and is protecting the systems. —– At approximately 10:40 the CDS2 UPS system lost power due to a short circuit condition. The power was immediately restored with the UPS system being temporarily bypassed. Updates will be forthcoming. The scope of the outage would be computer which were …

Continue reading »

SDSC Emergency Maintenance – Linux patch/reboot – 1 Nov 2016, 20:00-23:00

[Update – 20:41] All patches and reboots have been applied. —- SDSC will be applying critical patches to the Linux environment tonight, 1 Nov 2016 starting at 8pm. This maintenance will require a reboot of all systems listed below. Please contact with any questions or concerns. Updates will be posted to as the …

Continue reading »

SDSC Outage Notification – Project Storage – 11:30-12:00, 19 Aug 2016 [Resolved]

The SDSC Project Storage Service experienced a partial outage from approximately 11:30AM – 12:00PM. A hung process on a single hotel node consumed a critical amount of processor and memory resources, effectively rendering the node unavailable during the outage. At this time the server is back up and functioning as expected.  We apologize for any inconvenience caused by …

Continue reading »

SDSC Outage Notification – Commvault – 21:00, 29 Feb 2016

[Update: 23:06, 29 Feb 2016] Deduplication stores have been sealed and systems requiring the deduplication are running backups. At approximately 21:00 on 29 Feb 2016 the Commvault media agent named ‘cvma3’ hung and was power cycled. This caused jobs which were running to pause. There were some jobs which required manual restarts. Some jobs utilizing …

Continue reading »

Older posts «