As many of the organizations follow the Change Management process as part of ITSM, “Runbook” is a term which is widely used for the implementation or operations of the Infrastructure services. These runbooks need to be prepared with the pre-requisites, implementation steps and the rollback plan. This would then be reviewed by multiple stakeholders and change board for approval for the execution of the activity.
In this post, I would like to share the “Runbook Templates” for some of the common scenarios for Exchange Servers. Exchange Administrators can make use of these templates and modify to their organization’s standard for implementing the services.
- Exchange 2013 Hybrid Upgrade to Exchange 2019
Environment: Existing environment is running Exchange 2013 CU 23 with Hybrid Exchange Online has been configured.
Requirement: To upgrade Exchange 2013 CU 23 to Exchange 2019 CU 7/8 with Hybrid
Assumptions: Exchange 2019 Calculator has been used for sizing and new servers with Windows server 2019 has been built.
Pre-requisites:
| ID |
Description |
| 1 | Administrator Access – (Enterprise/Schema/Domain/Organization Management) |
| 2 | Latest security updates installed on the OS |
| 3 | Export the current certificate from the Exchange 2013 |
| 4 | Load balancer VIP for Exchange 2019 |
| 5 | Firewall port communication for the below,
|
| 6 | Azure AD Connect Server Access to refresh the Directory schema in case of any errors in AAD sync |
| 7 | M365 Global administrator login to re-run the Hybrid configuration wizard |
| 8 | Export the current Hybrid Configuration |
Implementation Steps:
On a high level below are the steps,
- Installing and configuring Exchange 2019
- Migrating services from Exchange 2013 to 2019
- Migrating the Recipients (mailboxes, PF,etc)
- Decommissioning Exchange 2013
| ID |
Activities |
| 1 | Install the below pre-requisites on the Exchange 2019 server,
|
| 2 | Reboot the server |
| 3 | Verify the AD attributes – rangeUpper, Objectversion before preparing the AD for Exchange 2019 |
| 4 | Prepare the AD schema using Setup /prepareSchema /IAcceptExchangeServerLicenseTerms |
| 5 | Prepare the AD using Setup /prepareAD /IAcceptExchangeServerLicenseTerms |
| 6 | Prepare the AD domains using Setup /prepareAllDomains /IAcceptExchangeServerLicenseTerms |
| 7 | Verify the AD attributes – rangeUpper, Objectversion for Exchange 2019 |
| 8 | Refresh the Directory schema using the AAD Connect – in case of any errors in AAD sync |
| 9 | Install the Exchange 2019 using Setup.exe either using GUI or unattended setup |
| 10 | Reboot the server |
| 11 | Verify the Exchange installation by using the https://localhost/ecp |
| 12 | Follow the steps 01, 02 & 09 – 11 on the other servers one by one if any |
| 13 | Activate the Microsoft Exchange 2019 Servers |
| 14 | Import the wildcard certificate to the Exchange 2019 servers |
| 15 | Assign the services to the certificate |
| 16 | Configure the AutoD SCP for Exchange 2019 servers using Set-ClientAccessService -AutoDiscoverServiceInternalURI |
| 17 | Configure the Internal URL & External URL for the Exchange 2019 Virtual directories and OA using EAC or PS
|
| 18 | Create the DAG and add the Exchange 2019 members to it |
| 19 | Create the Databases by specifying the database & log files location (Quotas to be noted) |
| 20 | Add the mailbox database copies to the DAG with correct Activation preference |
| 21 | Verify the OAB in Exchange 2019 |
| 22 | Enable IMAP & POP if required |
| 23 | Verify the Accepted domains are populated |
| 24 | Verify the Receive Connectors in the Exchange 2019 and create the required custom receive connectors like Exchange 2013 |
| 25 | Create a new test mailbox in Exchange 2013 |
| 26 | Perform client connectivity/mail flow – internal/external |
| 27 | Move the same mailbox to Exchange 2019 |
| 28 | Perform tests by adding host file entry to the VIP of Exchange 2019 client connectivity/mail flow – internal/external. External email will go through Exchange 2013 at this time |
| 29 | Verify If the Applications can connect (if any) and send emails using Exchange 2019 VIP |
| 30 | Email gateway solution to be configured with new Exchange 2019 IP’s for send/receive emails |
| 31 | Add the Exchange 2019 servers to the Send Connector as the Source servers |
| 32 | External Public IP to be NAT’ed to the VIP of Exchange 2019 (This would allow not to change public DNS and in case of any issue, this can be changed to the existing VIP of Exchange 2013) |
| 33 | Internal DNS name to be changed to new VIP of Exchange 2019 |
| 34 | Perform all the tests for users connecting to Exchange 2019 with their mailboxes on Exchange 2013 |
| 35 | Re-run the Hybrid Configuration Wizard |
| 36 | Create the migration batches and start moving the mailboxes |
| 37 | Move the Public folders/Public folder mailboxes (if any) |
| 38 | once all mailboxes are migrated, power off Exchange 2013 servers and monitor it for a week |
| 39 | Power on the Exchange 2013 servers |
| 40 | Uninstall Exchange 2013 servers using GUI or unattended setup mode |
Rollback Plan:
| ID |
Activities |
| 1 | Uninstall Exchange 2019 incase if issues arise before the services migration |
| 2 | Change the NAT from public IP to Existing Exchange 2013 so services stay in Exchange 2013 |
| 3 | Change internal DNS back to Exchange 2013 |
| 4 | Change the Email gateway solution to send/receive emails to Exchange 2013 |
| 5 | Remove the Exchange 2019 servers from send connector and add Exchange 2013 |
| 6 | Application teams to connect to Exchange 2013 VIP |
| 7 | Set the AutodiscoverServiceInternalURI to Null on the Exchange 2019 servers |
- Exchange 2019 Cumulative Update
Environment: Exchange 2019 in 2-node DAG
Requirement: To Install the latest cumulative update
Note: This is applicable to Exchange 2016 as well.
Pre-requisites:
| ID | Description |
| 1 | Administrator Access – (Enterprise/Schema/Domain/Organization Management) – Active Directory permissions are required in case if the new CU has schema changes |
| 2 | Backup of Active Directory, Exchange Servers and Databases are UpToDate |
| 3 | Backup Exchange customizations like OWA, configuration files, registries, etc |
| 4 | Temporarily disable Anti-virus, monitoring (if any) |
| 5 | Download the Pre-requisites especially .NET |
Implementation Steps:
| ID | Activities |
| 1 | Prepare the Active Directory using Setup file /PrepareSchema and /PrepareAD |
| 2 | Place the first node in Maintenance mode
|
| 3 | Verify that the server is marked as Down/Offline in the load balancer |
| 4 | Reboot the server and make sure no pending updates |
| 5 | Install the Update using GUI or unattended mode |
| 6 | Reboot the server |
| 7 | Remove the server from Maintenance mode
|
| 8 | Repeat the steps from 2 to 6 on the other node |
| 9 | Post upgrade tasks:
|
| 10 | Enable the Anti-Virus, monitoring (if any) |
Roll back Plan:
| ID | Activities |
| 1 | If the upgrade fails on the first node, make sure to check the logs, troubleshoot and fix the issue and complete the upgrade |
| 2 | As the last option, uninstall the CU, install the previous CU, restore the databases from backup |
Runbook templates can be downloaded here
Happy learning!! 🙂