As many of the organizations follow the Change Management process as part of ITSM, “Runbook” is a term which is widely used for the implementation or operations of the Infrastructure services. These runbooks need to be prepared with the pre-requisites, implementation steps and the rollback plan. This would then be reviewed by multiple stakeholders and change board for approval for the execution of the activity.

In this post, I would like to share the “Runbook Templates” for some of the common scenarios for Exchange Servers. Exchange Administrators can make use of these templates and modify to their organization’s standard for implementing the services.

  • Exchange 2013 Hybrid Upgrade to Exchange 2019

Environment: Existing environment is running Exchange 2013 CU 23 with Hybrid Exchange Online has been configured.

Requirement: To upgrade Exchange 2013 CU 23 to Exchange 2019 CU 7/8 with Hybrid

Assumptions: Exchange 2019 Calculator has been used for sizing and new servers with Windows server 2019 has been built.

Pre-requisites:

ID

Description

1 Administrator Access – (Enterprise/Schema/Domain/Organization Management)
2 Latest security updates installed on the OS
3 Export the current certificate from the Exchange 2013
4 Load balancer VIP for Exchange 2019
5 Firewall port communication for the below,

  • Exchange 2013 and Exchange 2019
  • Email gateway solution and Exchange 2019
  • Applications and Exchange 2019 for Relay
  • User subnets to the Exchange 2019 VIP
  • Exchange 2019 and Office365 for Hybrid Configuration
6 Azure AD Connect Server Access to refresh the Directory schema in case of any errors in AAD sync
7 M365 Global administrator login to re-run the Hybrid configuration wizard
8 Export the current Hybrid Configuration

Implementation Steps:

On a high level below are the steps,

  • Installing and configuring Exchange 2019
  • Migrating services from Exchange 2013 to 2019
  • Migrating the Recipients (mailboxes, PF,etc)
  • Decommissioning Exchange 2013
ID

Activities

1 Install the below pre-requisites on the Exchange 2019 server,

  • Install .NET Framework 4.8
  • Visual C++ Redistributable Package for Visual Studio 2012
  • Visual C++ Redistributable Package for Visual Studio 2013
  • Install-WindowsFeature Server-Media-Foundation
  • Install Unified Communications Managed API 4.0
  • Install-WindowsFeature Server-Media-Foundation, NET-Framework-45-Features, RPC-over-HTTP-proxy, RSAT-Clustering, RSAT-Clustering-CmdInterface, RSAT-Clustering-Mgmt, RSAT-Clustering-PowerShell, WAS-Process-Model, Web-Asp-Net45, Web-Basic-Auth, Web-Client-Auth, Web-Digest-Auth, Web-Dir-Browsing, Web-Dyn-Compression, Web-Http-Errors, Web-Http-Logging, Web-Http-Redirect, Web-Http-Tracing, Web-ISAPI-Ext, Web-ISAPI-Filter, Web-Lgcy-Mgmt-Console, Web-Metabase, Web-Mgmt-Console, Web-Mgmt-Service, Web-Net-Ext45, Web-Request-Monitor, Web-Server, Web-Stat-Compression, Web-Static-Content, Web-Windows-Auth, Web-WMI, Windows-Identity-Foundation, RSAT-ADDS
2 Reboot the server
3 Verify the AD attributes – rangeUpper, Objectversion before preparing the AD for Exchange 2019
4 Prepare the AD schema using Setup /prepareSchema /IAcceptExchangeServerLicenseTerms
5 Prepare the AD using Setup /prepareAD /IAcceptExchangeServerLicenseTerms
6 Prepare the AD domains using Setup /prepareAllDomains /IAcceptExchangeServerLicenseTerms
7 Verify the AD attributes – rangeUpper, Objectversion for Exchange 2019
8 Refresh the Directory schema using the AAD Connect – in case of any errors in AAD sync
9 Install the Exchange 2019 using Setup.exe either using GUI or unattended setup
10 Reboot the server
11 Verify the Exchange installation by using the https://localhost/ecp
12 Follow the steps 01, 02 & 09 – 11 on the other servers one by one if any
13 Activate the Microsoft Exchange 2019 Servers
14 Import the wildcard certificate to the Exchange 2019 servers
15 Assign the services to the certificate
16 Configure the AutoD SCP for Exchange 2019 servers using Set-ClientAccessService -AutoDiscoverServiceInternalURI
17 Configure the Internal URL & External URL for the Exchange 2019 Virtual directories and OA using EAC or PS

  • OA
  • ECP
  • OWA
  • EWS
  • EAS
  • MAPI
  • OAB
18 Create the DAG and add the Exchange 2019 members to it
19 Create the Databases by specifying the database & log files location (Quotas to be noted)
20 Add the mailbox database copies to the DAG with correct Activation preference
21 Verify the OAB in Exchange 2019
22 Enable IMAP & POP if required
23 Verify the Accepted domains are populated
24 Verify the Receive Connectors in the Exchange 2019 and create the required custom receive connectors like Exchange 2013
25 Create a new test mailbox in Exchange 2013
26 Perform client connectivity/mail flow – internal/external
27 Move the same mailbox to Exchange 2019
28 Perform tests by adding host file entry to the VIP of Exchange 2019 client connectivity/mail flow – internal/external. External email will go through Exchange 2013 at this time
29 Verify If the Applications can connect (if any) and send emails using Exchange 2019 VIP
30 Email gateway solution to be configured with new Exchange 2019 IP’s for send/receive emails
31 Add the Exchange 2019 servers to the Send Connector as the Source servers
32 External Public IP to be NAT’ed to the VIP of Exchange 2019 (This would allow not to change public DNS and in case of any issue, this can be changed to the existing VIP of Exchange 2013)
33 Internal DNS name to be changed to new VIP of Exchange 2019
34 Perform all the tests for users connecting to Exchange 2019 with their mailboxes on Exchange 2013
35 Re-run the Hybrid Configuration Wizard
36 Create the migration batches and start moving the mailboxes
37 Move the Public folders/Public folder mailboxes (if any)
38 once all mailboxes are migrated, power off Exchange 2013 servers and monitor it for a week
39 Power on the Exchange 2013 servers
40 Uninstall Exchange 2013 servers using GUI or unattended setup mode

Rollback Plan:

ID

Activities

1 Uninstall Exchange 2019 incase if issues arise before the services migration
2 Change the NAT from public IP to Existing Exchange 2013 so services stay in Exchange 2013
3 Change internal DNS back to Exchange 2013
4 Change the Email gateway solution to send/receive emails to Exchange 2013
5 Remove the Exchange 2019 servers from send connector and add Exchange 2013
6 Application teams to connect to Exchange 2013 VIP
7 Set the AutodiscoverServiceInternalURI to Null on the Exchange 2019 servers
  • Exchange 2019 Cumulative Update

Environment: Exchange 2019 in 2-node DAG

Requirement: To Install the latest cumulative update

Note: This is applicable to Exchange 2016 as well.

Pre-requisites:

ID Description
1 Administrator Access – (Enterprise/Schema/Domain/Organization Management) – Active Directory permissions are required in case if the new CU has schema changes
2 Backup of Active Directory, Exchange Servers and Databases are UpToDate
3 Backup Exchange customizations like OWA, configuration files, registries, etc
4 Temporarily disable Anti-virus, monitoring (if any)
5 Download the Pre-requisites especially .NET

Implementation Steps:

ID Activities
1 Prepare the Active Directory using Setup file /PrepareSchema and /PrepareAD
2 Place the first node in Maintenance mode

  • Set-ServerComponentState EXCH1 –Component HubTransport –State Draining –Requester Maintenance
  • Redirect-Message -Server EXCH1 -Target EXCH2
  • Suspend-ClusterNode –Name EXCH1
  • Set-MailboxServer EXCH1 –DatabaseCopyActivationDisabledAndMoveNow $true
  • Set-MailboxServer EXCH1 –DatabaseCopyAutoActivationPolicy Blocked
  • Set-ServerComponentState EXCH1 –Component ServerWideOffline –State InActive –Requester Maintenance
3 Verify that the server is marked as Down/Offline in the load balancer
4 Reboot the server and make sure no pending updates
5 Install the Update using GUI or unattended mode
6 Reboot the server
7 Remove the server from Maintenance mode

  • Set-ServerComponentState EXCH1 –Component ServerWideOffline –State Active –Requester Maintenance
  • Resume-ClusterNode –Name EXCH1
  • Set-MailboxServer EXCH1 –DatabaseCopyAutoActivationPolicy Unrestricted
  • Set-MailboxServer EXCH1 –DatabaseCopyActivationDisabledAndMoveNow $false
  • Set-ServerComponentState EXCH1 –Component HubTransport –State Active –Requester Maintenance
8 Repeat the steps from 2 to 6 on the other node
9 Post upgrade tasks:

  • Redistribute the databases across the DAG nodes
  • Restore the customizations like OWA, configuration files, registries, etc
  • Check the cluster services, DAG status, Service health, MAPI health, Replication health, database copy status, server component status
  • Client connectivity and mail flow
10 Enable the Anti-Virus, monitoring (if any)

Roll back Plan:

ID Activities
1 If the upgrade fails on the first node, make sure to check the logs, troubleshoot and fix the issue and complete the upgrade
2 As the last option, uninstall the CU, install the previous CU, restore the databases from backup

Runbook templates can be downloaded here

Happy learning!! 🙂

By Ashok M

Microsoft Certified Professional with key technical skills including Microsoft Exchange, Windows Server, Microsoft Azure, Office 365, Intune, EMS, Skype for Business, Active Directory, ADFS and has got more exposure to Hyper V, System Center Configuration Manager, Virtualization, Video conferencing room systems, SQL. Have experience in design, implementation, migration & support for various Microsoft infrastructure products. Currently working as "Implementation Engineer" with the UAE's first tier IV Data Center design certification in the region.

Leave a Reply

Your email address will not be published. Required fields are marked *