[Update 03:00] All work has been completed. Please contact colosupport@sdsc.edu with any questions or concerns [Update 02:25] Engineers are still working to verify and recover stale NFS mountpoints. Some webpages may be unavailable while the NFS filesystems are unavailable. [Update 01:16] Engineers are still verifying functionality of routes. Some services including project storage and USS …
Category Archive: Incident Notification
Networking issues on 172.25.119.0/26 subnet
[Update: 16:00] Networking engineers implemented a temporary fix to resolve the networking problems. A permanent solution is forthcoming with the planned reboot of the core router this evening. — We are seeing networking issues on the 172.25.119.0/26 subnet which may be causing slowness or missing network packets. SDSC network engineers are investigating the cause of …
[Resolved] SDSC Outage Notification – Partial Network Outage – 20 Feb 2021, 06:35-09:10 PST
[Update 11:15 PST] Restitution efforts are complete as of 10:45 PST with no lingering issues observed over the past half hour. Please email support@sdsc.edu to report any outstanding issues. —————————————————— SDSC received internal alerts of inaccessible services this morning at 06:35. The issue appears to be isolated to a subset of the SDSC network. The …
SDSC Outage Notification – Network – 9 Feb 2021, 10:50am
[Update – 11:30] Networking service has been failed back to original hardware and this has resolved the issue. SDSC will be in contact with hardware vendors to investigate and determine root cause. — SDSC has received notifications of inaccessible systems on some subnets. We are investigating and will post shortly with more information.
SDSC Outage Notification – Grafana portal – 20 Oct 2020, 13:10
[Update – 13:20] The issues have been resolved. Please contact SDSC Support with any questions or concerns. We have reports of dashboard errors which engineers are investigating. Updates will be posted to status.sdsc.edu
SDSC Outage Notification – Citrix Services – 13 Aug 2020, 08:15
[Update – 12:48] Citrix service has been restored. Please contact support with any questions or concerns. Citrix remote desktop services are unavailable and the outage is being investigated by engineers. More information will be posted to this location.
SDSC Outage Notification – VM guests rebooted due to hypervisor error – 11 July 2020, 14:25
[Update: 17:10] Hardware logs have been handed over to vendor support for review. At this time it appears that a hardware failure on the hypervisor node delayed I/O and caused the guests to automatically reboot on other hypervisors. At approximately 16:25 on 11 July 2020 the following systems rebooted due to a crash of the …
SDSC Outage Notification – Datacenter Networking Issues – 11:45, 30 Mar 2020
[Update 14:30] Different network paths are providing network connectivity. Please contact SDSC Operations if you have any additional questions or concerns. [Update 13:20] SDSC engineers are reconfiguring network devices to bring the connections to working equipment. They are also in communication with the hardware vendors for component replacement. [Update 12:43] The known scope of the …
SDSC Outage Notification – VMware guest outage – 22:00, 6 Dec 2019
[09:06 – 7 Dec 2019] Affected hosts include: actreal1 bahamut cycorephaser dev.mygeohub.org dev.purr.hubzero.org dev.smarteragriculture.org ea-repository elprado exec01.cdmhub.org exec01.communityhub.hubzero.org exec01.demohubzero.org exec01.dev.mygeohub.org exec01.help.hubzero.org exec01.mygeohub.org exec01.pharmahub.org exec01.smarteragriculture.org exec01.stemedhub.org gitlab grunwald16 itss-dynamics johndenver massive2 nexposevase osc-dev packages.help.hubzero.org qball3.sdsc.edu rsm-print rsm-print2 russell sc-dynamics sc-lansweeper sentinel-new stage.cdmhub.org web01.int.citscibio.org web01.int.extlearn.org web01.int.help.hubzero.org web01.int.vhub.org yellowfin [08:15 – 7 Dec 2019] Around 22:00 on …
SDSC Outage Notification – Network Uplink Outage – 18 Nov 2019, 08:48
(Update 09:40, 19 Nov) Crews repaired the two largest broken fiber bundles last night between 20:00 and 23:30. All severed links should be operational — please contact SDSC Operations if you continue to notice connection issues. (Update 16:48) Crews continue to work to repair the largest broken fiber with an anticipated completion of 01:00 on …