Received: from mm-delp-elk-001.trustwave.com (204.13.202.74) by PPS-EXCHANGE2.pplant.msu.edu (35.8.180.123) with Microsoft SMTP Server (TLS) id 8.3.327.1; Wed, 25 Sep 2013 20:36:27 -0400 Received: (qmail 24639 invoked from network); 25 Sep 2013 20:36:51 -0400 Received: from unknown (HELO mm-scan-elk-007.trustwave.com) (172.29.7.42) by mm-delp-elk-001.trustwave.com with (DHE-RSA-AES256-SHA encrypted) SMTP; 25 Sep 2013 20:36:51 -0400 Received: (qmail 24970 invoked from network); 25 Sep 2013 20:36:51 -0400 Received: from unknown (HELO mm-recp-elk-001.trustwave.com) (172.29.130.68) by mm-scan-elk-007.trustwave.com with (DHE-RSA-AES256-SHA encrypted) SMTP; 25 Sep 2013 20:36:51 -0400 Received: (qmail 30184 invoked from network); 25 Sep 2013 20:36:51 -0400 Received: from unknown (HELO list.msu.edu) (35.9.75.22) by mm-recp-elk-001.trustwave.com with (DHE-RSA-AES256-SHA encrypted) SMTP; 25 Sep 2013 20:36:51 -0400 Received: from [35.9.75.22] (helo=list) by list.msu.edu with esmtp (Exim 4.69) (envelope-from <[log in to unmask]>) id 1VOzZX-0006qt-G9; Wed, 25 Sep 2013 20:36:50 -0400 Received: by LIST.MSU.EDU (LISTSERV-TCP/IP release 15.5) with spool id 106368282 for [log in to unmask]; Wed, 25 Sep 2013 20:36:34 -0400 Received: from [35.9.75.40] (helo=CAD-EX3.campusad.msu.edu) by list.msu.edu with esmtp (Exim 4.69) (envelope-from <[log in to unmask]>) id 1VOzZJ-0006qO-U1 for [log in to unmask]; Wed, 25 Sep 2013 20:36:34 -0400 Received: from CAD-EX2.campusad.msu.edu ([fe80::4186:508:e37d:4000]) by CAD-EX3.campusad.msu.edu ([fe80::594d:18e6:2391:3b72%20]) with mapi id 14.02.0328.009; Wed, 25 Sep 2013 20:36:28 -0400 From: "Davis, Thomas" <[log in to unmask]> To: "[log in to unmask]" <[log in to unmask]> Date: Wed, 25 Sep 2013 20:35:41 -0400 Subject: FW: Network outage Thread-Topic: Network outage Thread-Index: Ac66T28roGv2GM8RSQeQpcP3fICqkwAAMvyw Message-ID: <[log in to unmask]> References: <[log in to unmask]> List-Help: <http://list.msu.edu/cgi-bin/wa?LIST=ITEXCHANGE>, <mailto:[log in to unmask]> List-Subscribe: <mailto:[log in to unmask]> List-Unsubscribe: <mailto:[log in to unmask]> In-Reply-To: <[log in to unmask]> Reply-To: "Davis, Thomas" <[log in to unmask]> Accept-Language: en-US Content-Language: en-US X-MS-Exchange-Organization-AuthAs: Anonymous X-MS-Exchange-Organization-AuthSource: PPS-EXCHANGE2.pplant.msu.edu X-MS-Has-Attach: X-Auto-Response-Suppress: All X-MS-TNEF-Correlator: acceptlanguage: en-US list-archive: <http://list.msu.edu/cgi-bin/wa?LIST=ITEXCHANGE> list-owner: <mailto:[log in to unmask]> x-virus: None found by Clam AV x-originating-ip: [69.89.106.81] x-spam-report: All incoming messages to list.msu.edu are analyzed for typical spam characteristics. See http://techbase.msu.edu/article.asp?id=11475 for additional report information. Content preview: FYI. This is the first major backbone outage in several years. Once the root cause analysis is complete and we have a better understanding of what caused the outage, this information will be shared with the IT Exchange community. /Tom [...] Content analysis details: (1.3 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- 0.0 HTML_MESSAGE BODY: HTML included in message 1.3 RDNS_NONE Delivered to internal network by a host with no rDNS x-spam-score: 1.3 x-spam-level: + Content-Type: multipart/alternative; boundary="_000_D12A03D62BF0C84BB3CC10E550B5473F3ABE19C2CADEX2campusadm_" MIME-Version: 1.0 --_000_D12A03D62BF0C84BB3CC10E550B5473F3ABE19C2CADEX2campusadm_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable FYI. This is the first major backbone outage in several years. Once the roo= t cause analysis is complete and we have a better understanding of what cau= sed the outage, this information will be shared with the IT Exchange commun= ity. /Tom From: Davis, Thomas [mailto:[log in to unmask]] Sent: Wednesday, September 25, 2013 8:30 PM To: [log in to unmask] Subject: Network outage FYI. We experienced intermittent outages on the campus network backbone (MS= Unet) this afternoon. The source of the problem was traced to a core switch= ing device that had become unstable. Resolution of the problem took a coupl= e hours, unfortunately, as the unstable switch caused other network devices= to fail. In order to stop the cascading failures and restore everything to= working order, technicians were forced to shut down the entire backbone an= d restart each of the core switches. My apologies for the disruption this c= aused to the campus community. Tom Davis, Acting CIO P.S. Data from the failing switch has been sent to the vendor for root caus= e analysis. --_000_D12A03D62BF0C84BB3CC10E550B5473F3ABE19C2CADEX2campusadm_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable <html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr= osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:= //www.w3.org/TR/REC-html40"> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"= > <meta name=3D"Generator" content=3D"Microsoft Word 14 (filtered medium)"> <style><!-- /* Font Definitions */ @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} @font-face {font-family:Tahoma; panose-1:2 11 6 4 3 5 4 4 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; margin-bottom:.0001pt; font-size:11.0pt; font-family:"Calibri","sans-serif";} a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:purple; text-decoration:underline;} span.EmailStyle17 {mso-style-type:personal; font-family:"Calibri","sans-serif"; color:windowtext;} span.EmailStyle18 {mso-style-type:personal; font-family:"Calibri","sans-serif"; color:#1F497D;} span.EmailStyle19 {mso-style-type:personal-reply; font-family:"Calibri","sans-serif"; color:#1F497D;} .MsoChpDefault {mso-style-type:export-only; font-size:10.0pt;} @page WordSection1 {size:8.5in 11.0in; margin:1.0in 1.0in 1.0in 1.0in;} div.WordSection1 {page:WordSection1;} --></style><!--[if gte mso 9]><xml> <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" /> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext=3D"edit"> <o:idmap v:ext=3D"edit" data=3D"1" /> </o:shapelayout></xml><![endif]--> </head> <body lang=3D"EN-US" link=3D"blue" vlink=3D"purple"> <div class=3D"WordSection1"> <p class=3D"MsoNormal"><span style=3D"color:#1F497D">FYI. This is the first= major backbone outage in several years. Once the root cause analysis is co= mplete and we have a better understanding of what caused the outage, this i= nformation will be shared with the IT Exchange community. /Tom<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p> </o:p></spa= n></p> <div> <div style=3D"border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in = 0in 0in"> <p class=3D"MsoNormal"><b><span style=3D"font-size:10.0pt;font-family:"= ;Tahoma","sans-serif"">From:</span></b><span style=3D"font-s= ize:10.0pt;font-family:"Tahoma","sans-serif""> Davis, T= homas [mailto:[log in to unmask]] <br> <b>Sent:</b> Wednesday, September 25, 2013 8:30 PM<br> <b>To:</b> [log in to unmask]<br> <b>Subject:</b> Network outage<o:p></o:p></span></p> </div> </div> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">FYI. We experienced intermittent outages on the camp= us network backbone (MSUnet) this afternoon. The source of the problem was = traced to a core switching device that had become unstable. Resolution of t= he problem took a couple hours, unfortunately, as the unstable switch caused other network devices to fail. In order to s= top the cascading failures and restore everything to working order, technic= ians were forced to shut down the entire backbone and restart each of the c= ore switches. My apologies for the disruption this caused to the campus community.<o:p></o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">Tom Davis, Acting CIO<o:p></o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">P.S. Data from the failing switch has been sent to t= he vendor for root cause analysis. <o:p></o:p></p> </div> </body> </HTML> --_000_D12A03D62BF0C84BB3CC10E550B5473F3ABE19C2CADEX2campusadm_--