Print

Print


Received: from mm-delp-elk-001.trustwave.com (204.13.202.74) by
 PPS-EXCHANGE2.pplant.msu.edu (35.8.180.123) with Microsoft SMTP Server (TLS)
 id 8.3.327.1; Wed, 25 Sep 2013 20:36:27 -0400
Received: (qmail 24639 invoked from network); 25 Sep 2013 20:36:51 -0400
Received: from unknown (HELO mm-scan-elk-007.trustwave.com) (172.29.7.42)  by
 mm-delp-elk-001.trustwave.com with (DHE-RSA-AES256-SHA encrypted) SMTP; 25
 Sep 2013 20:36:51 -0400
Received: (qmail 24970 invoked from network); 25 Sep 2013 20:36:51 -0400
Received: from unknown (HELO mm-recp-elk-001.trustwave.com) (172.29.130.68)
  by mm-scan-elk-007.trustwave.com with (DHE-RSA-AES256-SHA encrypted) SMTP;
 25 Sep 2013 20:36:51 -0400
Received: (qmail 30184 invoked from network); 25 Sep 2013 20:36:51 -0400
Received: from unknown (HELO list.msu.edu) (35.9.75.22)  by
 mm-recp-elk-001.trustwave.com with (DHE-RSA-AES256-SHA encrypted) SMTP; 25
 Sep 2013 20:36:51 -0400
Received: from [35.9.75.22] (helo=list)	by list.msu.edu with esmtp (Exim 4.69)
	(envelope-from <[log in to unmask]>)	id 1VOzZX-0006qt-G9; Wed, 25
 Sep 2013 20:36:50 -0400
Received: by LIST.MSU.EDU (LISTSERV-TCP/IP release 15.5) with spool id
          106368282 for [log in to unmask]; Wed, 25 Sep 2013 20:36:34
 -0400
Received: from [35.9.75.40] (helo=CAD-EX3.campusad.msu.edu) by list.msu.edu
          with esmtp (Exim 4.69) (envelope-from <[log in to unmask]>) id
          1VOzZJ-0006qO-U1 for [log in to unmask]; Wed, 25 Sep 2013
          20:36:34 -0400
Received: from CAD-EX2.campusad.msu.edu ([fe80::4186:508:e37d:4000]) by
          CAD-EX3.campusad.msu.edu ([fe80::594d:18e6:2391:3b72%20]) with mapi
          id 14.02.0328.009; Wed, 25 Sep 2013 20:36:28 -0400
From: "Davis, Thomas" <[log in to unmask]>
To: "[log in to unmask]" <[log in to unmask]>
Date: Wed, 25 Sep 2013 20:35:41 -0400
Subject: FW: Network outage
Thread-Topic: Network outage
Thread-Index: Ac66T28roGv2GM8RSQeQpcP3fICqkwAAMvyw
Message-ID:
 <[log in to unmask]>
References:
 <[log in to unmask]>
List-Help: <http://list.msu.edu/cgi-bin/wa?LIST=ITEXCHANGE>,
           <mailto:[log in to unmask]>
List-Subscribe: <mailto:[log in to unmask]>
List-Unsubscribe: <mailto:[log in to unmask]>
In-Reply-To:
 <[log in to unmask]>
Reply-To: "Davis, Thomas" <[log in to unmask]>
Accept-Language: en-US
Content-Language: en-US
X-MS-Exchange-Organization-AuthAs: Anonymous
X-MS-Exchange-Organization-AuthSource: PPS-EXCHANGE2.pplant.msu.edu
X-MS-Has-Attach:
X-Auto-Response-Suppress: All
X-MS-TNEF-Correlator:
acceptlanguage: en-US
list-archive: <http://list.msu.edu/cgi-bin/wa?LIST=ITEXCHANGE>
list-owner: <mailto:[log in to unmask]>
x-virus: None found by Clam AV
x-originating-ip: [69.89.106.81]
x-spam-report: All incoming messages to list.msu.edu are analyzed for
 typical               spam characteristics. See
               http://techbase.msu.edu/article.asp?id=11475 for additional
               report information. Content preview:  FYI. This is the first
               major backbone outage in several years. Once the root cause
               analysis is complete and we have a better understanding of
 what               caused the outage,               this information will be
 shared with the IT Exchange community.               /Tom [...] Content
 analysis details:   (1.3 points,               5.0 required) pts rule name
              description ----               ----------------------
               -------------------------------------------------- 0.0
               HTML_MESSAGE           BODY: HTML included in message 1.3
               RDNS_NONE              Delivered to internal network by a host
               with no rDNS
x-spam-score: 1.3
x-spam-level: +
Content-Type: multipart/alternative;
	boundary="_000_D12A03D62BF0C84BB3CC10E550B5473F3ABE19C2CADEX2campusadm_"
MIME-Version: 1.0

--_000_D12A03D62BF0C84BB3CC10E550B5473F3ABE19C2CADEX2campusadm_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

FYI. This is the first major backbone outage in several years. Once the roo=
t cause analysis is complete and we have a better understanding of what cau=
sed the outage, this information will be shared with the IT Exchange commun=
ity. /Tom

From: Davis, Thomas [mailto:[log in to unmask]]
Sent: Wednesday, September 25, 2013 8:30 PM
To: [log in to unmask]
Subject: Network outage

FYI. We experienced intermittent outages on the campus network backbone (MS=
Unet) this afternoon. The source of the problem was traced to a core switch=
ing device that had become unstable. Resolution of the problem took a coupl=
e hours, unfortunately, as the unstable switch caused other network devices=
 to fail. In order to stop the cascading failures and restore everything to=
 working order, technicians were forced to shut down the entire backbone an=
d restart each of the core switches. My apologies for the disruption this c=
aused to the campus community.

Tom Davis, Acting CIO

P.S. Data from the failing switch has been sent to the vendor for root caus=
e analysis.

--_000_D12A03D62BF0C84BB3CC10E550B5473F3ABE19C2CADEX2campusadm_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr=
osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:=
//www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
>
<meta name=3D"Generator" content=3D"Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
	{font-family:"Cambria Math";
	panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
	{font-family:Calibri;
	panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
	{font-family:Tahoma;
	panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0in;
	margin-bottom:.0001pt;
	font-size:11.0pt;
	font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
	{mso-style-priority:99;
	color:blue;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{mso-style-priority:99;
	color:purple;
	text-decoration:underline;}
span.EmailStyle17
	{mso-style-type:personal;
	font-family:"Calibri","sans-serif";
	color:windowtext;}
span.EmailStyle18
	{mso-style-type:personal;
	font-family:"Calibri","sans-serif";
	color:#1F497D;}
span.EmailStyle19
	{mso-style-type:personal-reply;
	font-family:"Calibri","sans-serif";
	color:#1F497D;}
.MsoChpDefault
	{mso-style-type:export-only;
	font-size:10.0pt;}
@page WordSection1
	{size:8.5in 11.0in;
	margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
	{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=3D"EN-US" link=3D"blue" vlink=3D"purple">
<div class=3D"WordSection1">
<p class=3D"MsoNormal"><span style=3D"color:#1F497D">FYI. This is the first=
 major backbone outage in several years. Once the root cause analysis is co=
mplete and we have a better understanding of what caused the outage, this i=
nformation will be shared with the IT
 Exchange community. /Tom<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"color:#1F497D"><o:p>&nbsp;</o:p></spa=
n></p>
<div>
<div style=3D"border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in =
0in 0in">
<p class=3D"MsoNormal"><b><span style=3D"font-size:10.0pt;font-family:&quot=
;Tahoma&quot;,&quot;sans-serif&quot;">From:</span></b><span style=3D"font-s=
ize:10.0pt;font-family:&quot;Tahoma&quot;,&quot;sans-serif&quot;"> Davis, T=
homas [mailto:[log in to unmask]]
<br>
<b>Sent:</b> Wednesday, September 25, 2013 8:30 PM<br>
<b>To:</b> [log in to unmask]<br>
<b>Subject:</b> Network outage<o:p></o:p></span></p>
</div>
</div>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
<p class=3D"MsoNormal">FYI. We experienced intermittent outages on the camp=
us network backbone (MSUnet) this afternoon. The source of the problem was =
traced to a core switching device that had become unstable. Resolution of t=
he problem took a couple hours, unfortunately,
 as the unstable switch caused other network devices to fail. In order to s=
top the cascading failures and restore everything to working order, technic=
ians were forced to shut down the entire backbone and restart each of the c=
ore switches. My apologies for the
 disruption this caused to the campus community.<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
<p class=3D"MsoNormal">Tom Davis, Acting CIO<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
<p class=3D"MsoNormal">P.S. Data from the failing switch has been sent to t=
he vendor for root cause analysis.
<o:p></o:p></p>
</div>
</body>
</HTML>


--_000_D12A03D62BF0C84BB3CC10E550B5473F3ABE19C2CADEX2campusadm_--