A little more info on number 1:
MSU Telecom Systems transports /some/ long-distance calls over the Merit
connection (currently only for Avaya Campus PBX users). Our
long-distance vendor (Qwest) has a peering arrangement with Merit
networks, and has facilities located in the Merit POP in Chicago. We use
SSL/TLS encryption for these calls for the media traffic (meaning that
the actual voice conversation is fully encrypted while on this pipe).
Once our equipment sees any degradation in service, it will
automatically switch to other forms of connectivity (typically ISDN-PRI
service). This morning, we noticed a configuration issue that caused the
service not to automatically switch as it has done in the past. This
has since been corrected. Once our department was made aware that some
LD calls were not going through, we immediately took action and
corrected our equipment. This is the first service issue we have had
with this pipe, and we will continue to monitor it. We are coordinating
with ACNS Network Operations and ACNS Network Security for the
configuration of this service.
MSU Telecom Systems
From: MSU Network Administrators Group [mailto:[log in to unmask]] On
Behalf Of John Valenti
Sent: Tuesday, August 21, 2007 3:34 PM
To: [log in to unmask]
Subject: Re: [MSUNAG] Merit about yesterday's trouble
A few other tidbits I picked up at the meeting:
1. MSU's long distance phone calls are routed over the Internet. The
default path uses Merit, but it can be manually re-routed over an
AT&T connection. (someone at Telecomm re-routed things after an hour
of downtime this morning).
2. MSU is developing a remote hot site at Penn State. Currently in an
early stage of development, so there isn't a redundant network path
to it. Potentially the MSU web home page will be duplicated there
3. Merit had to send techs from Ann Arbor to Grand Rapids and
Kalamazoo to reboot hardware this morning (they had a good reason for
this). They will be installing remote power management hardware; also
trying to get some local people signed up for access to those fiber
On August 21, at 12:52 PM August 21, Doug Nelson wrote:
> For those of you who didn't attend Merit's web conference, here's a
> short update. Merit engineers traced the source of the Merit Network
> problems to the failure of an optical amplifier within the MiLR
> following scheduled maintenance by the fiber carrier. This failure
> created a marginal optical fiber path, and accounted for the network
> difficulties seen in the original outages between Thursday and Sunday,
> and again this morning, after a second scheduled fiber maintenance
> window. The fiber path failure was compounded by a software failure
> which repeatedly attempted to bypass the marginal link, and then
> restore the link to service. This process also caused the errant
> switches to exhaust their memory due to a second software failure,
> leading to additional packet switching and routing failures, cascading
> throughout the Merit Network.
> Once the problematic link was disabled, service returned to a somewhat
> stable state, but on limited bandwidth.
> As of 12:00 noon today, the Merit network appears to be operating
> normally and at full capacity, but we are awaiting final word from
> the Merit field and operations engineers on what was done and whether
> service is fully restored.
> I will pass on additional details once Merit releases them.