[Update – 23:58] All maintenance is complete and we are CANCELLING the maintenance window for 7 March 2021.
There was a card failure in the backup router for which we have a replacement in process. Once that arrives at the datacenter we will announce a replacement time — we expect no disruption as this is for the backup router.
[Update – 23:03] Engineers are diagnosing hardware issues on the redundant router. All traffic is pathed through the primary router at this time.
[Update – 21:53] Maintenance on the backup unit is progressed as planned.
[Update – 21:14] Routes have been reverted to the primary unit
[Update – 20:29] The next unit is being updated and rebooted.
[Update – 19:49] Core routers are still being upgraded and configurations syncronized.
[Update – 19:08] Core routers are now being upgraded.
[Update – 18:14] Core switch hardware has been upgraded. Some VLANs are still inaccessible and/or seeing 50% packet loss.
[Update – 17:53] Maintenance is still going. Some VLANs are still seeing additional outage when hardware is rebooted.
[Update – 17:18] Following the first maintenance, we see some subnets including some NFS storage down for over 15 minutes. Engineers are investigating
[Update – 16:46] Network maintenance is expected to start promptly at 17:00. Updates will be posted regularly as work continues.
Greetings,
We have scheduled an emergency maintenance window to perform a number of upgrades to our core infrastructure. We are scheduling two windows on consecutive days to allow us to address any further, unforeseen issues (testing/rollback/physical changes). The maintenance
will occur Saturday, March 6, 2021 1700-0000 PST and Sunday, March 7, 2021 1700-2200 PST.
During this window, we will be working directly with Juniper engineers to update each of our four main core devices in turn. We will be making every effort to minimize impact to users by manually manipulating traffic through their backup devices and redundant paths, but please be prepared for some intermittent connectivity during each window, as each device will require a reboot.
We will be keeping our status page up to date during these windows, so please refer to that for any updates/changes. http://status.sdsc.edu/
We appreciate your patience and understanding as we work to correct these issues.
Best Regards,
–Jeff
Jeff Filliez
Data Center Manager, SDSC
858-534-8368
jfilliez@sdsc.edu