[Update – 22:59] All guests are online and appear to be acting normally. Please contact support with any questions or concerns.
[Update – 22:00] Engineers are slowly booting guests and monitoring stability of the VMware cluster. Additional updates will be posted once all guests are on and the environment is operating normally.
[Update – 18:42] Most hypervisors have been recovered, while SDSC engineers continue to work with VMware support to recover the last hypervisor. Once all hypervisors are online, the guests will be booted.
[Update – 17:01] Engineers are working with VMware technicians to restore service of the VM cluster. All guests have been powered off and hypervisors are being rebooted.
[Update – 15:10] VMware connectivity is being restored. Many of the in-building problems may be related to the DHCP server being offline, as that is served by VMware.
Multiple networks both within the SDSC datacenter and building are experiencing connectivity issues. Network engineers are investigating.