Event Notice: Connectivity loss of switching through lightning – Resolved
Event Start: 9:01pm 10/05/2012
Event Start: 9:01pm 10/05/2012
Event Resolved: 5:00PM 10/08/2012
Event Known Impacts: Impact all floors west and central SDSC building, W77 – some rockstar resources were impacted
Event Known Impacts: Impact all floors west and central SDSC building, W77 – some rockstar resources were impacted
Summary of Issue:
Network disruption from mis-conifgured port(s) (loop, root issue) travels though the network until it is blocked at core level. Once cleared (manually or automatically) the blocks are not cleared both ways, requiring a router restart to correct the situation.
Update 10: ENS has made a change to the Router/Switch Fabric that should mitigate the current issue please let us know if you see any further issues.
Update 9:
- Actions taken at this time:
- In order to isolate the issue and stabilize services ENS has taken the following actions over the last 48 hours.
- Move the following off Lightning – Should be functioning normally
- All single attached hosts
- iSAFE has been moved to Router
- II36 has been moved to Z39 – (CMS, north and south pole, some rocks systems, some triton systems)
- East Floor Switch has been moved to 4900m
- PDB has been moved to DD60 switch 3
- UCSF has been moved
- All VRRP standby lines have been moved to Medusa
- CALIT link has been moved to Thor
- Still connected to Lightning(possibly subject to outages)
- West and Central Floor switches
- W77 part of rockstar system
- VLAN 1
- Detached Lightning from the East and West Aristas and moved it one switch back to determine if this is a compatibility issue between the routers and switches.
- Restricted access to networking management to local only at this time
- ENS has opened an event with the Vendor
Update 8: West and Central floor switches and wireless routing not working at this time 1:10pm 10/08/2012
Update 7: ENS had to reset the router Lightning at about 8:05 to restore service.
Update 6: 2:30AM we have lost a supervisor card, ENS reponded and services should be restored with the exception of PDB
Update 5: PDB reporting outage, ENS responding
Update 4: 4:40PM 10/06/2012 Service has been restored with the exception of vlan 474 – ii36, if you are experiencing any issues please notify operations@sdsc.edu
Update 3: 4:40pm 10/06/2012 ENS has had to restart the router Lightning to bring the system back to functional state.
Update 2: 3:32Pm We have experience another event at this time. ENS restarted the systems and is having to prune out some connections to isolate the issue. If you experience any problems please report it to operations@sdsc.edu.
Update : ENS has responded and most services appear to be running; However, there could be additional events. ENS is gathering additional information and will be working with the vendors involved