Cisco router failure causes system wide outage

On May 15, 2013 at 5:02pm PST, one of our Cisco edge routers experienced a hardware fault which caused it to begin flooding routing information into the network environment. Due to the cascading nature of this event, the automated failover protocols were unable to function properly.The corrective action taken to rectify this issue was to power down the defective router and reboot the adjacent network devices. Once rebooted, their routing tables were flushed and network traffic began flowing through the adjacent router.

As of 6:18pm PST, we confirmed that all network services were restored.

Our engineering and support staff continued troubleshooting all remaining services to confirm all issues had been resolved.

We have an open ticket with Cisco TAC and expect to have a new Edge router redeployed once new hardware arrives.

We apologize for any inconvenience that this failure caused. And we are working with Cisco to attempt to ensure that this type of failure does not affect the network traffic in the future.

Email This Article To A Friend - Print This Article