 |
 |
Attached is an excerpt from a message sent by Merit Network to their members and affiliates. Doug =+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+= Merit Community: Last Tuesday, November 13, 2007, we sent a message to you regarding the degradation of the network service your organization may have experienced. Today, we want to provide you with additional information regarding the November 13, 2007 service issue and also to provide you with an update on the most recent service issue that your organization may have experienced on Monday, November 19, 2007. Listed below is a recap of each service issue and the respective service ticket number for your reference. If you are currently experiencing any network performance problems please contact Merit's Network Operations Center (NOC) immediately. We have engineering, NOC, and support staff available to work with you on resolving any issues you are experiencing. Service Updates: --------------- Tuesday, November 13, 2007 Service Degradation, Service Ticket Number: 60530 Merit experienced this network service issue, because one of our commodity network service providers dropped Merit's BGP routes in Chicago. Further investigation with our commodity provider's NOC and Merit's NOC, it was diagnosed as a hardware failure with one of our commodity providers. Our commodity provider immediately dispatched an engineer to the location to fix the problem. One of Merit's major connections to the commodity Internet is in Chicago and the hardware service failure was from this commodity network service provider. Merit has 2 x 1Gig connections with this provider in Chicago. Each connection is a full 1Gig path and both of these paths were affected by the hardware failure. There is one a 1Gig connection to Merit's other commodity Internet service provider, Level-3, in Detroit. During this hardware failure, all of Merit's Members were being routed out this secondary commodity Internet connection through Detroit. Due to the capacity of the connection, 1Gig, it could not handle the whole load, which averages over 2Gigs of sustained commodity Internet service, therefore, some of Merit's Members experienced slow response times. Monday, November 19, 2007 Possible Service Degradation, Service Ticket Number: 60555 At approximately 8:00AM EST, November 19, 2007, Merit's 10Gig interface on our switch that services one of our two 1Gig connections to our network service provider in Chicago stopped forwarding packets, which caused congestion on our remaining connections. Troubleshooting the problem determined that it was an intermittent problem. Therefore, instead of replacing the interface card, we dispatched an engineer to have the switch replaced. The switch was replaced during the maintenance window: Tuesday, 20 November 2007, 00:00 EST to Tuesday, 20 November 2007, 02:00 EST. Since the failure was only 1 of our 2Gig connections, Merit Engineers were able to provide some traffic management to reroute some of the traffic to our 1Gig connection out of Detroit. Therefore, not all Merit Members may have experienced a degradation of service on Monday, November 19, 2007. We are currently not experiencing any service related issues on our core backbone. If you are currently experiencing any network performance problems please contact Merit's Network Operations Center (NOC) immediately. We have engineering, NOC, and support staff available to work with you on resolving any issues you are experiencing. [ed. note: Contact the ACNS Help Desk if you have any unresolved issues here at MSU. ] We sincerely apologize for the inconvenience these network service issues may have caused your organization. We continuously strive to provide the highest level of service to our Membership and regret these service issues occurred. (signed) Merit Network =+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+= -- Doug Nelson, Network Manager | [log in to unmask] Academic Computing and Network Services | Ph: (517) 353-2980 Michigan State University | http://www.msu.edu/~nelson/