Exchange 2010 Service Pack 2 Upgrade Failed – FIXED!

INTRODUCTION

I was doing a spot of consultation recently. I received an urgent call about a serious issue a client was having with an attempted installation of Exchange 2010 Service Pack 2. What should have been a routine update quickly became a catastrophic failure of all Exchange related services causing major problems with all aspects of mail and mail access for the users.

I got on the scene and took a good look around. It’s a single site, single Exchange server, separate domain controller, all servers running Server 2008 R2. I documented all that appeared inconsistent and wrong against my reference machine, running a working version EX 2010 SP1.

As you’ll see, much of IIS was hosed and this is where the main thrust of the work was centred after determining whether or not SP2 had even installed correctly. What follows are my findings and my course of action over 3 days of concentrated diagnosis and troubleshooting to get them back online and to really try to get to the root cause of the issue so that we may learn from the lesson. At the expense of a short and punchy article, I’ve gone into much more granular detail of what happened. I feel, in this situation, more information is of greater value under the circumstances and certainly, the devil really is in the detail. In this case Exchange 2010 was installed on D:\ and this was originally a fresh installation of Exchange 2010 into the environment, not a migration from a previous version.

PROBLEMS

• EMC down
• Exchange Management Shell (EMS) down
• Mail transport down
• Did Service Pack 2 even install? Partially install? Or not install at all?
• OWA down internally and externally
• ActiveSync down
• AutoDiscover down

INITIAL FINDINGS

• A lot of the symptoms appeared to be centred on client access and IIS, notwithstanding a mailbox database in a “status unknown” state and multiple stopped services.
• When I checked the Exchange Server setup log I discovered that the installation/application of SP2 had errored out because of a service problem. Here’s the excerpt from the log:

[02/12/2012 21:15:49.0269] [1] [ERROR] The following error was generated when “$error.Clear();

if (get-service MSExchangeServiceHost* | where {$_.name -eq “MSExchangeServiceHost”})

{

restart-service MSExchangeServiceHost

}

” was run: “Service ‘Microsoft Exchange Service Host (MSExchangeServiceHost)’ cannot be started due to the following error: Cannot start service MSExchangeServiceHost on computer ‘.’.”.

[02/12/2012 21:15:49.0269] [1] [ERROR] Service ‘Microsoft Exchange Service Host (MSExchangeServiceHost)’ cannot be started due to the following error: Cannot start service MSExchangeServiceHost on computer ‘.’.

[02/12/2012 21:15:49.0269] [1] [ERROR] Cannot start service MSExchangeServiceHost on computer ‘.’.

[02/12/2012 21:15:49.0269] [1] [ERROR] The service cannot be started, either because it is disabled or because it has no enabled devices associated with it

[02/12/2012 21:15:49.0270] [1] [ERROR-REFERENCE] Id=BridgeheadRoleSetterComponent___115c1108e99e4560bd2c03c0fec99908 Component=EXCHANGE14:\Current\Release\Shared\Datacenter\Setup

[02/12/2012 21:15:49.0270] [1] Setup is stopping now because of one or more critical errors.

[02/12/2012 21:15:49.0270] [1] Finished executing component tasks.

[02/12/2012 21:15:49.0321] [1] Ending processing Install-BridgeheadRole

[02/12/2012 22:19:30.0147] [0] End of Setup

[02/12/2012 22:19:30.0147] [0] **********************************************

• When I got the EMC back up and running (see below) I could run the Exchange Best Practice Analyser and indeed there were Critical Issues. Client Access, Hub transport, Mailbox Roles showing not configured. BPA recommending run setup again.
• Browsing to: 443 (https) in most all Exchange related virtual directories In IIS Manager, below the Default Web Site, produced errors and inconsistencies.
• Testing connectivity to Active Sync from the Internet using Exchange Remote Connectivity Analyser produced multiple failures and errors.
• Multiple 2280 errors in the Application Log from IIS-W3SVC-WP stating “The Module DLL D:Program Files\Microsoft\Exchange Server\V14\Client Access\OWA\auth\exppw.dll failed to load”.
• Multiple 1310 warnings from APS.NET 2.0.50727.0 documenting configuration errors along the following application paths;
  • D:\Program Files\Microsoft\Exchange Server\V14\ClientAccess\Sync
  • D:\Program Files\Microsoft\Exchange Server\V14\ClientAccess\Autodiscover
  • D:\Program Files\Microsoft\Exchange Server\V14\ClientAccess\exchweb\EWS
• Multiple 5139 warnings in the System Log from WAS stating, “A listener channel for protocol ‘http’ in worker process ‘740’ serving application pool ‘DefaultAppPool’ reported a listener channel failure”.

EMC & EMS ISSUE

SYMPTOMS

• When trying to open EMC I got:
  • “Initialization Failed” The following error occurred while attempting to connect to the specified exchange server ‘servername.domain.com’: The attempt to connect to http://servername.domain.com/PowerShell using “Kerberos” authentication failed: connecting to remote server failed with the following error message: The WinRM client received an HTTP server error status (500), but the remote service did not include any other information about the cause of the failure. For more info, see the about Troubleshooting help topic.
• Multiple ‘red text’ errors when opening EMS. Apologies for the lack of detail here, I failed to get the screen grab.

SOLUTION & TROUBLESHOOTING STEPS

• Using a separate production Exchange 2010 server as a reference machine, I discovered the web.config file from D:\Program Files\Microsoft\Exchange Server\V14\ClientAccess\PowerShell\ to be missing complete paths.
• Over 200 path entries were displaying file:///%ExchangeInstallDir% compared to what it should read which is file:///D:\Program Files\Microsoft\Exchange Server\V14\ followed by bin\name of the DLL.
• I replaced all those bad entries with the correct ones using MS Visual Studio.
• EMC now throws a new “ConsoleInitialization.ps1” error stating;
  • “ConsoleInitialize.ps1″ is not recognized as the name of a cmdlet”.
• ConsoleInitialization.ps1 should be located along the install path in the D:\Program Files\Microsoft\Exchange Server\V14\RemoteScripts folder. I discovered that my ConsoleInitialization.ps1 file had been renamed ConsoleInitialization.ps1.txt! I renamed it to .ps1 and after I waited a few minutes, I launched EMC and it opened up first time just fine. Along the way, I verified that MSExchangePowerShellAppPool was running under the correct “LocalSystem” identity in IIS Manager. Once into EMC, I was able to access and run an Exchange BPA Health Check as above. EMS was fixed too!

MAIL TRANSPORT ISSUE

SYMPTOMS

• No mailflow
• Outlook disconnected from Exchange
• Mailbox database and Public Folder database status showing unknown
• Microsoft Exchange Mail Submission service stopped
• Microsoft Exchange Search Indexer stopped
• Microsoft Exchange Replication service stopped

SOLUTION & TROUBLESHOOTING STEPS

• Start Microsoft Exchange Mail Submission service
  • Check EMC for mounted Mailbox database – mounted!
  • Outlook clients – connected!
  • Mailflow – working!

SERVICE PACK 2 INSTALLATION – SUCCESS OR FAIL?

REMEMBER: Since 2007, whenever it comes to a Service Pack, it’s not going to install some updates or a patch. It completely uninstalls the previous version keeping the configuration and installs the new version. So what’s meant to happen here is the SP2 installer uninstalls SP1 and tries to install SP2. In this case, our BPA Health Check scan describes that for some reason, the SP2 installation was not completed for 3 roles; Mailbox, Client Access and Hub Transport.

ALSO REMEMBER: The first thing that happens when running SP2 setup is that the schema will be extended. In our case, we’re not sure if our installation was a success or not. A quick check of the configuration partition with ADSIEDIT on the DC will confirm this.

• ADSIEdit > Connect to > Naming Context > Configuration
  • Configuration > CN=Configuration > CN=Services > CN=Microsoft Exchange > CN=OrganizationName
  • If the schema has been extended for SP2, you should see CN=AddressBook Mailbox Policies. This was not here in earlier versions of Exchange. Ours is here, so we are confirmed on the schema update.

SOLUTION & TROUBLESHOOTING STEPS

• Do a Get-ExchangeServer | FL to check and verify;
  • AdminDisplayVersion: Version 14.2 (Build 247.5)
  • Server Role: Mailbox, ClientAccess, HubTransport.
  • The EMC is confirmed as showing the same Version 14.2 (Build 247.5) for all roles.

Clearly there’s something wrong because the shell output is saying its running SP2, yet the BPA Health Check scan says otherwise and IIS is broke! Let’s check the registry and the binaries.

• Open up registry editor to check registry entries as follows
  • HKLM\SOFTWARE\Microsoft\ExchangeServer\v14\MailboxRole
    • ConfiguredVersion is showing 14.1.218.15
    • UnpackedVersion is showing 14.2.247.5
• HKLM\SOFTWARE\Microsoft\ExchangeServer\v14\ClientAccessRole
  • ConfiguredVersion is showing 14.1.218.15
  • UnpackedVersion is showing 14.2.247.5
• HKLM\SOFTWARE\Microsoft\ExchangeServer\v14\HubTransportRole
  • UnpackedVersion is showing 14.2.247.5
  • There’s also a Watermark entry here too, meaning we may need to run SP2 setup again.
  • Check and verify the version numbers of the following binary files;
    • Store.exe: 14.2.247.3
    • Eseutil.exe: 14.2.247.1
    • Isinteg.exe: 14.2.247.3

The mailflow is working fine. We can see that the service pack upgraded successfully because the binary files show 14.2, however the registry was not modified. We can conclude that the BPA Health Check scan was displaying Critical Issues just because of these registry keys.

• Just to be on the safe side, we’ll dismount the database and do a copy backup of both Mailbox DB and Public Folder DB .edb files. We’ll also export a backup copy of the registry for good measure before we proceed.
• Now, we’re going to copy the unpacked version and paste it into the configured version leaving the watermark in the HubTransportRole registry key intact. Then let’s reboot the server.
• When the server comes back, we’ll verify all MS Exchange and related services are started. All is good except our MS Exchange Service Indexer is not started and attempts to start it produced “Error 1068”. I checked its dependencies and discovered Microsoft Search (Exchange) was not started so I set it to “Manual” Startup type and started it. I could then successfully start the MS Exchange Search Indexer service.
• I verified mailflow. All was good.
• Mailbox, Client Access and Hub Transport were all showing the correct version 14.2 in EMC.
• I checked the Configured Version vs. Unpacked Version numbers in the Registry as above and they matched for Mailbox and Client Access. However, the watermark was still present in the HubTransportRole key with an additional “Action” key and the versions did not match. I deleted the Watermark and Action keys and modified the Configured Version with the Unpacked Version. I rebooted the system.
• The system came back, I verified mailflow. I ran a BPA Health Check Scan – NO CRITICAL ISSUES!

OUTLOOK WEB APP ISSUE

SYMPTOMS

• OWA errored out with “Server Error in ‘/owa’ Application”. Followed by;

<!– Web.Config-configurationfile –>

<configuration>

<system.web>

<customErrors mode=”Off”/>

</system.web>

</configuration>

• Numerous 2280 error events in the Windows Application Log from IIS-W3SVC-WP as I mention in my initial findings. Further examination of the \OWA virtual directory Modules revealed that exppw was missing.
• Multiple repeated 1310 warnings in the Application Log from APS.NET 2.0.50727.0 documenting configuration errors as follows;
  • Could not load file or assembly ‘Microsoft.Exchange.Security, Version=14.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35’ or one of its dependencies. The system cannot find the file specified. (D:\Program Files\Microsoft\Exchange Server\V14\ClientAccess\sync\web.config line 1601)
  • Could not load file or assembly ‘Microsoft.Exchange.Security, Version=14.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35’ or one of its dependencies. The system cannot find the file specified. (D:\Program Files\Microsoft\Exchange Server\V14\ClientAccess\exchweb\EWS\web.config line 2392)
  • Could not load file or assembly ‘Microsoft.Exchange.Security, Version=14.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35’ or one of its dependencies. The system cannot find the file specified. (D:\Program Files\Microsoft\Exchange Server\V14\ClientAccess\Autodiscover\web.config line 940)
• Multiple 5139 warnings in the System Log from WAS, as above.

SOLUTION & TROUBLESHOOTING STEPS

• Let’s begin by tackling the 2280 Application Log errors i.e. the missing exppw registered module in IIS.
  • IIS Manager > servername\Sites\Default Web Site\owa > Modules  > Configure Native Modules
  • Check exppw > Ok
  • Module Type should be set to Native
  • Entry Type should be set to Local
  • CMD > iisreset
• Now we’ll address the 1310 Application Log warnings. Knowing what we know already about bad file paths in web.config files, I started with an examination of web.config files for the other Exchange services and sure enough, they were bad. There is a web.config file for each and every virtual directory. The same bad path entries as before with web.config from the ClientAccess\PowerShell directory were persistent with all of the following:
  • ClientAccess\owa\web.config (supporting the owa virtual directory)
  • ClientAccess\sync\web.config  (supporting the Microsoft-Server-ActiveSync virtual directory)
  • ClientAccess\Autodiscover\web.config (supporting the Autodiscover virtual directory)
  • ClientAccess\ecp\web.config (supporting the ecp virtual directory)
  • And ClientAccess\exchweb\EWS\web.config (supporting the EWS virtual directory)

I began by running the same operation on all 5 web.config files, as we did for the PowerShell web.config file, and replaced those bad path entries as above.

• After completion: CMD > iisreset
• Browsing OWA and the Default Web Site :443 (https) from IIS Manager now produced a blank page.
• Following on from the amendments to the web.config files, we’re now accompanied by 1105 and 1334 errors in the Application Log from MSExchange ActiveSync and ASP.NET respectively.
• A check of the IIS > servername> Worker processes revealed missing processes;
  • We should have 6 running in total.
  • Part of the problem is that DefaultAppPool and MSExchangeOWAAppPool are missing.
  • After a refresh, DefaultAppPool comes in and stays in a starting state.
• We need to get these worker Processes straightened out. Time to have a look at applicationHost.config from C:\Windows\System32\inetsrv\config\.
  • This file is the root file of the IIS configuration system. It defines Application Pools as well as all sites, applications, virtual directories and global defaults for the web server settings.
  • Check and verify that the following line is listed under globalModules:

<add name=”exppw” image=”D:\Program Files\Microsoft\Exchange Server\V14\ClientAccess\Owa\auth\exppw.dll” />

• Checked and verified.
• Now we’ll take a backup and then remove the owa folder from C:\Windows\Microsoft.NET\Framework64\v2.0.50727\Temporary ASP.NET Files.
• Now let’s check the advanced properties of the DefaultAppPool in IIS Manager > servername> Application pools;
  • Check and verify the Identity under Process Model is set to LocalSystem > Ok
  • Refresh and check IIS > servername> Worker processes
    • DefaultAppPool – RUNNING!
    • MSExchangeOWAAppPool – RUNNING!
• We still cannot browser OWA. It’s still taking us to a blank page.
• Next, we’re going to reset the OWA virtual directory.
  • First, let’s take a backup of the IIS config.
  • Navigate to C:\Windows\System32\inetsrv and run the following command;
    • Appcmd add backup IISBACKUP
• Next use EMC > Server Config > Client access > Reset Virtual Directory;
  • Browse to owa (Default Web Site), Next, Next, Reset
• Next do a CMD > iisreset
• Browse to OWA – WORKING! Internally and externally.
• Verify login to OWA using the Domain Admin – all good
• Verify login using other users – not working.
• Other (standard) users are not able to login to OWA but the Domain Admin is. This points to a permissions issue on the server.
  • Check and verify permissions on the owa\auth virtual directory
    • Authenticated Users should have Traverse Folder / execute file permission.
    • On our server, Authenticated Users have only Read permission.
• Change permission for Authenticated Users <not inherited> to Traverse Folder / execute file only.
  • Standard users still not able to login to OWA.
• Explore to D:\Program Files\Microsoft\Exchange Server\V14\ClientAccess and check security permissions
  • Add Authenticated Users with default permissions; Read & execute, List folder contents and Read.
• Verify External and Internal URLs are present and correct and set Authentication to User name only in EMC
  • EMC > Server > Client Access > Outlook Web App > properties of owa (Default Web Site)
    • General
    • Authentication: User name only and set to logon domain.
• CMD > iisreset
• Test login to OWA with standard user with username only – WORKING!

AUTO DISCOVER AND ACTIVE SYNC ISSUE

SYMPTOMS

• iPhones not synching.
• Testing connectivity to Active Sync from the Internet using Exchange Remote Connectivity Analyser produced multiple failures and errors.
• Multiple 1105 and 1334 errors in the Application Log from MSExchange ActiveSync and ASP.NET respectively.

SOLUTION & TROUBLESHOOTING STEPS

• Verify Basic Authentication is the only enabled authenticating method under IIS > Default Web Site > Microsoft-Server-ActiveSync > Authentication.
• Verify HTTP is not being redirected under IIS > Default Web Site > Microsoft-Server-ActiveSync > HTTP Redirect.
• Verify that we’re getting an appropriate challenge when attempting to browse Microsoft-Server-ActiveSync and that HTTP Error 401.1 – Unauthorized is the response.
• We’re now going to delete and re-create the Active Sync virtual directory.
  • First, we’ll backup the IIS config with CMD > Appcmd add backup IISBACKUP run from the C:\Windows\System32\inetsrv directory.
  • Second, we’ll reset the virtual directory, this time with the EMS
    • Remove-ActiveSyncVirtualDirectory -Identity “servername\Microsoft-Server-ActiveSync (Default Web Site)”
      • ‘Y’ to confirm
      • Verify with the virtual directory is gone from IIS with a quick refresh
    • New-ActiveSyncVirtualDirectory
      • Verify with the virtual directory is back in IIS again with a quick refresh
• Test connectivity to Active Sync from the Internet using Exchange Remote Connectivity Analyser manually specifying the FQDN of the server.
  • Better results than before, but failing on a FolderSync command
• Restart the IIS Admin Service.
• Restart World Wide Web Publishing Service.
• Check and verify in the Windows Application Event Viewer that the recurring 1105 and 1334 errors have STOPPED!
• Verify External and Internal URLs are present and correct under EMC > Server > Client Access > Exchange ActiveSync > properties of Microsoft-Server-ActiveSync (Default Web Site).
• Browsing to internal URL from internal client to Microsoft-Server-ActiveSync producing “503 Service Unavailable”.
• Time to check the web.config file from D:\Program Files\Microsoft\Exchange Server\V14\ClientAccess\sync
  • Paths entries are correct, something else is wrong with the file.
• Time to restore web.config from the web.config.bak file. After we restore from the .bak file we must replace the bad file:///%ExchangeInstallDir% path entries that have restored along with it with the good file:///D:\Program Files\Microsoft\Exchange Server\V14\, just like we did earlier.
  • Move web.config file to the desktop.
  • Copy and paste a copy of web.config.bak located in the \sync folder to the \sync folder.
  • Rename web.config.bak to web.config.
  • Replace the bad paths with good ones using Notepad++.
• Restart the IIS Admin Service.
• Browsing to Microsoft-Server-ActiveSync via IIS Manager producing “503 Service Unavailable”.
• A quick check of Application Pools in IIS Manager showed MSExchangeSyncAppPool not started.
  • Right-click MSExchangeSyncAppPool > Start
• CMD > iisreset.
• Browsing to Microsoft-Server-ActiveSync via IIS Manager now produced an appropriate challenge. The same is true from the Domain Controller and a client workstation.
• Test connectivity to Active Sync from the Internet using Exchange Remote Connectivity Analyser manually specifying the FQDN of the server producing good results – FIXED!
• Finally, a quick check on the client side confirmed that the iPhones were syncing again.

LESSONS & THE CAUSES

• It’s worth re-iterating here that, according to my information, this happened because of an attempted upgrade from Service Pack 1 to Service Pack 2. It’s still not known for certain what truly caused the problems that led to the aftermath. I was informed that the person who performed the upgrade was advised by the installer to remove all server roles for IIS 6, which strikes me as weird. He did so before re-installing them. This was second hand information I gathered before I got involved and we’ll never know exactly what was done.
• Antivirus – this is probably the most important lesson here. Symantec Endpoint Protection was installed on this machine prior to Service Pack 2. This may well have contributed to the cause. Antivirus software blocks Exchange from accessing the registry and its own files. It’s recommended to always use ‘Exchange aware’ antivirus. If not, then it may be possible to define exclusions to Exchange files. Check with the AV vendor.