Dave Graff had the answer on this one. I thought I’d post it so that you all have some info about it if anyone else runs into the same issue.

Server 2008 changed the way some of the patching procedures operate, where instead of having the majority of the patch installation process occurring while the OS is up and running, you often see a large part of it being executed after the reboot during the pre-boot stage before the whole OS has been loaded. The operations here tend to be disk-intensive and can take a minute or two to complete due to our older storage system.

VMware uses an OS watchdog that is implemented through the VMware Tools integration with a special driver to catch for cases where a VM has been hung in an unresponsive state or bluescreen and gives it a hard reset to bring it back up. We currently run this feature with the most relaxed out of box setting which waits for two minutes before forcing a VM reboot.

These two features are conflicting due to the startup patch process taking more than two minutes to complete before the VMware Tools driver can load, which forces the VM to reset which Windows detects as a failed startup attempt and enters the recovery console during the next startup cycle

There is a way, though I don’t have those specifics yet, to disable the automatic recovery mode during a failed startup. Another option is to extend the heartbeat loss threshold.

Thanks.

-dak

----------------------------------------

Dak Aldrich

Michigan State University

Physical Plant Division Support

1147 Chestnut Road, Room 1
East Lansing, MI 48824

[log in to unmask]

517-432-0239

On Dec 13, 2012, at 8:57 AM, "Aldrich, Dakharai" <[log in to unmask]> wrote:

Here’s an odd one. I haven’t started digging deep into the server logs yet, but I thought I’d pull off of you all’s XP first to maybe get some direction.

This has happened twice in 3 months. It happened after the October MS update cycle, and then again in December.

What’s happening: After a server automatically installs updates, and then reboots, it comes back in Recovery Mode. We cancel out of recovery mode and reboot the server and then updates complete, and it’s back to business as normal.

This month it is the same set of machines as in October. All of the problem machines are VMs in a VMWare cluster. All of the problem machines are Server 2008 R2 Datacenter x64. A few of them do not have SP1 but most do. (Servers that are not R2 do not seem to have this issue.) All of the problem machines are set to automatically install updates and automatically reboot if an update requires it.

Everything I’ve found online points to people being stuck in the Recovery Mode loop. We are not. We cancel the recovery mode and reboot the server and updates complete successfully and everything works as expected.

Has anyone else seen this issue, or have an answer off the top of their head?

Thanks for any info anyone might have.