Exchange Server

Exchange 2010 includes the capability to ensure that deleted items are retained within the dumpster till the deleted item retention period. This prevents accidentally or maliciously deleted items from being deleted permanently, and allows an administrator to recover such items. This feature is known as Single Item Recovery and it enables organizations to change their backup paradigms (i.e., you no longer need to maintain backups for single item recovery), and to retain and allow discoverability of the data to meet compliance requirements.

Essentially there are two steps:

1. Search – Determining the location of the missing items.
2. Recovery – Retrieving the missing items.

Remember, in order to discover and recover the data, each mailbox needs to have Single Item Recovery enabled prior to the accidental purge event. Therefore, we recommend enabling Single Item Recovery for mailboxes as part of the Exchange 2010 upgrade process.

The Scenario

Ross sent his administrative assistant, Julie, a message regarding his upcoming trip to Seattle, specifically requesting Julie to book his itinerary. Unfortunately, before she could work on Ross’ request, Julie shift-deleted the message while cleaning out her mailbox. Like most users, Julie has done this before and is familiar with the Recover Deleted Items capability within Outlook. However, this time, Julie made the mistake of clicking the delete button for the message in question instead of clicking the recover button. Panicking, Julie calls Help Desk to request recovery of the item.

Step 1: Search

The help desk ticket results in a workflow process that is performed by an IT administrator who has necessary rights to perform searches (in this scenario, the Help Desk technician’s user account has been delegated the Discovery Management role).

Note: By default, no accounts have the ability to perform mailbox searches. You can either create a custom role group to allow an administrator to search only a subset of mailboxes, or add the administrator to the Discovery Management built-in role group (which allows them to search all mailboxes in the Exchange organization) by using the following command:
Add-RoleGroupMember “Discovery Management” -Member <user account>

The Help Desk technician has two choices for performing discovery, and the choice will depend on the target user’s client access license (CAL):

1. If the users included in the search have Standard CALs, the Help Desk technician can only use the Search-Mailbox cmdlet.
2. If the users included in the search have Enterprise CALs, the Help Desk technician can also use the New-MailboxSearch cmdlet, or the Multi-Mailbox Search feature in the Exchange Control Panel (ECP).

In Julie’s case, she provided the Help Desk technician with the following information:

• The message was sent from her boss.
• The message contains the word “Seattle”.

Searching messages by using Search-Mailbox

When a mailbox with a Standard CAL will be searched, the Search-Mailbox cmdlet will be used. The Search-Mailbox cmdlet requires the following information:

• The mailbox to be searched
• The search query criteria
• The mailbox and folder where the results will be placed

1. Knowing this information, the Help Desk technician executes the following command from the Shell:Search-Mailbox sec -SearchQuery “from:’boss’ AND seattle” -TargetMailbox “Discovery Search Mailbox” -TargetFolder “Secretary Recovery” -LogLevel FullNote: Search-Mailbox does not allow the target mailbox to be the same as the source mailbox. Search-Mailbox does allow you to be very specific in your search criteria. Besides scoping the search with the SearchQuery parameter using Advanced Query Syntax (AQS), in Exchange 2010 SP1 you can also use the SearchDumpsterOnly switch to search only items in the dumpster.The Help Desk technician receives the following output:RunspaceId : fb25cadf-a63f-4e88-8567-cb4ae1b30ade
   Identity : corp.contoso.com/Users/Secretary
   TargetMailbox : corp.contoso.com/Users/DiscoverySearchMailbox {D919BA05-46A6-415f-80AD-7E09334BB852}
   TargetPSTFile :
   Success : True
   TargetFolder : \Secretary Recovery\Secretary-4/14/2010 6:28:33 AM
   ResultItemsCount : 1
   ResultItemsSize : 1.577 KB (1,615 bytes)
2. The Help Desk technician then logs into OWA and opens the Discovery Search Mailbox via the Open Other Mailbox option:Note: The OWA and ECP screenshots are from Exchange 2010 SP1. These are preliminary screen shots from pre-Beta software that are subject to change before the final release of SP1.
3. The Help Desk technician navigates the folder structure within the Discovery Search Mailbox and verifies that he has recovered the item in question:

Searching messages by using Multi-Mailbox Search

When a mailbox with an Enterprise CAL will be searched, the administrator can use the Multi-Mailbox Search feature in the Exchange Control Panel. The Help Desk technician takes the following steps:

1. He launches the Exchange Control Panel via https://mail.contoso.com/ecp and logs on using his credentials.
2. From the Options drop-down, he selects Manage My Organization.
   
3. He clicks on Service Level and selects the Mailbox Searches applet.
4. He clicks New to create a new search request which requires at least the following information:
   1. The search query criteria
      
   2. The mailbox to be searched
      
   3. The mailbox and folder where the results will be placed
      
5. When the results are obtained, he can either click on the [Open] link in the Mailbox Searches Results pane, or open the Discovery Search Mailbox via the Open Other Mailbox option from within OWA.
6. He navigates the folder structure within the Discovery Search Mailbox and verifies that he has recovered the item in question:
   

Step 2: Recovery

At this point the Search phase is complete and the Recovery phase begins. There are two options for how to recover and return the item back to the user and it depends on the version of Exchange 2010 you have deployed:

1. If you are running Exchange 2010 RTM or later, you can utilize the Search-Mailbox cmdlet to restore the item back to the user.
2. If you are running Exchange 2010 SP1, you can utilize the PST import and export cmdlets to restore the item back to the user.

Search-Mailbox Recovery Process

1. The Help Desk technician executes the following command from the Shell:Search-Mailbox “Discovery Search Mailbox” -SearchQuery “from:’boss’ AND seattle” -TargetMailbox sec -TargetFolder “Recovered by HelpDesk” -LogLevel Full -DeleteContentHe receives the following output:RunspaceId : fb25cadf-a63f-4e88-8567-cb4ae1b30ade
   Identity : corp.contoso.com/Users/DiscoverySearchMailbox {D919BA05-46A6-415f-80AD-7E09334BB852}
   TargetMailbox : corp.contoso.com/Users/Secretary
   TargetPSTFile :
   Success : True
   TargetFolder : \Recovered by HelpDesk\Discovery Search Mailbox-4/14/2010 6:32:49 AM
   ResultItemsCount : 1
   ResultItemsSize : 1.577 KB (1,615 bytes)
2. He notifies Julie that the item is recovered. Julie logs into her mailbox and verifies she has the correct item:
   

It’s important to note that due to the two-step process involved with Search-Mailbox (copying the results to the Discovery Mailbox and then copying the results back to the user’s mailbox) the hierarchy is same for the end user – the root of the Discovery Search Mailbox, as well as the folder target that was used to place the item in the Discovery Search Mailbox, are both visible.

PST Export/Import Recovery Process

Exchange 2010 SP1 includes infrastructure that allows administrators to perform bulk import and export of PST files without requiring the installation of the Outlook client. This infrastructure, supported by the cmdlets *-MailboxImportRequest and *-MailboxExportRequest, leverages the Mailbox Replication Service and the framework that exists for moving mailboxes between databases (see Understanding Move Requests for more information).

To use this functionality, two prerequisites must be met:

1. The person performing the import or export must have the appropriate permissions within Exchange. By default, no RBAC role group provides this functionality. To grant the ability for a help desk administrator, compliance officer, or Exchange administrator to perform bulk import/export capabilities against all mailboxes, the following commands must be executed:New-RoleGroup “Mailbox Import-Export Management” -Roles “Mailbox Import Export”
   Add-RoleGroupMember “Mailbox Import-Export Management” -Member <user account>The first command creates a new role group that grants access to the *-MailboxImportRequest and *-MailboxExportRequest cmdlets. The second command adds a user to the role group.
2. The Exchange Trusted Subsystem security group must have Full Control/Owner permissions on the file share that will be used to temporarily store the PST files.

In this scenario, the Help Desk technician is a member of the Mailbox Import-Export Management role group and thus can utilize the Import and Export cmdlets. The Help Desk technician:

1. Runs the following command from the Shell to export the recovered data from the Discovery Search Mailbox to a PST file:New-MailboxExportRequest -Mailbox “Discovery Search Mailbox” -FilePath “\\exchsvr\HelpDeskPst\SecretaryRecovery.pst” -ContentFilter {Subject -eq “april travel plans”} -SourceRootFolder “Secretary Recovery”
2. Runs the following command from the Shell to import the recovered data into Julie’s mailbox:New-MailboxImportRequest -Mailbox sec -FilePath “\\exchsvr\HelpDeskPst\SecretaryRecovery.pst” -TargetRootFolder “Recovered By HelpDesk”
3. Notifies Julie that the item is recovered.

At this point, Julie logs into her mailbox and verifies she has the correct item:

Conclusion

Exchange 2010 provides you the means to ensure data is not deleted from the system prior to the expiration of its deleted item retention. In the event that a message is accidentally or maliciously purged from the user’s dumpster, it can be easily recovered and restored using built-in tools.

Perform Integrity Checks
Exchange performs the automated maintenance tasks on its databases every night. You should still perform a manual integrity check on a quarterly basis. Manual checks let you see if there are any problems with the databases and take corrective action if necessary.

Before performing an integrity check, make sure you have a full backup of the database. In rare situations, performing manual database maintenance can cause database corruption. You also need to make sure you have adequate disk space. If you have to perform any type of repair on the database, you’ll need enough free space on the volume for a full copy of the database, plus another 10 to 20 percent for overhead.

To perform a server-level integrity check, you first need to dismount the Store. In Exchange 2007, open Exchange Management Console and navigate through the console tree to Server Configuration\Mailbox. Next, right-click the database you want to check and select Dismount Database from the shortcut menu. To dismount a database in Exchange 2003, open Exchange System Manager, navigate through the console tree to your store, right-click it, and choose Dismount Store from the shortcut menu.

After the database is dismounted, you can use Isinteg to check for errors in the database. Open a command prompt window, navigate to the \Program Files\Microsoft\Exchange Server\Bin folder, and enter the following command:

isinteg -s <servername> -test allfoldertests

When you run this command, you’ll receive a list of the databases on the server, as Figure 2 shows. Next, enter the number from the list for the database you want to test. Isinteg prompts you for confirmation; press Y to start the tests. If any errors are reported, Isinteg tells you what corrective action to take, and you should perform such actions right away.

If the server-level integrity check with Isinteg doesn’t return any errors, you should perform a database-level integrity check by using Eseutil. To do so, enter the following command:

eseutil /G "<database file path>"

In the above command, you would replace <database file path> with the actual database path (in quotes). For example, the command might be

eseutil /G "Q:\program Files\<br>     MicrosoftExchange Server\<br>     Mailbox First Storage Group\Database.edb"

Isinteg and Eseutil work the same in Exchange 2007 and Exchange 2003.

Check Your Database for Free Space
As already mentioned, Exchange defragments its databases as a part of the nightly maintenance cycle. However, an online defragmentation doesn’t actually shrink the size of the database. Instead, empty database pages, known as free space, are simply grouped together so they can be efficiently reused.

Usually this technique doesn’t present much of a problem, but there are circumstances when you might need to shrink a database. For example, if you moved some mailboxes to a different store as a way of freeing up disk space, you wouldn’t accomplish your goal unless you ran an offline defragmentation afterward.

Even if you aren’t trying to reduce the amount of space consumed by your databases, it’s a good idea to perform a quarterly check to make sure that the databases don’t contain excessive amounts of free space. Generally, free space is considered to be excessive if it occupies more than 15 percent of the total database. The easiest way to find out how much free space is in a database is to search your server’s application log for event ID 1221. As you can see in Figure 3, the Event Properties dialog box tells you how many megabytes of free space are in the database. Use this number along with the database’s total size to figure out the percentage of free space.

If you need to remove free space from a database, you can do so with the Eseutil command. You’ll have to dismount the database first, and be sure to follow the earlier words of caution about having a full backup and enough disk space for a backup copy. You would enter the command

eseutil /D "<database file path>"

where <database file path> is the actual database path.

I came across this error during an Exchange 2010 Unified Messaging deployment. The EMC would not connect:

“The attempt to connect to http://server.domain.com/PowerShell using “Kerberos” authentication failed: connecting to remote server failed with the following error message : The WinRM client cannot complete the operation within the time specified.  Check if the machine name is valid and is reachable over the network and firewall exception for Windows Remote Management service is enabled.  For more information, see the about_Remote_Troubleshooting Help topic.”

There are various blog posts on the internet around how to fix the connectivity problem to the server, but in this case the server EMC was pointing to had been decommissioned properly and was no longer listed in AD. EMS would connect fine to a different working server.

In my case I had to take two actions to fixed it.

Close EMC

Under C:\users\<specific user>\AppData\Roaming\Microsoft\MMC\Exchange Management Console\ there is a config file. Delete it

In the registry under HKCU\Software\Microsoft\Exchangeserver\v14\AdminTools\NodeStructureSettings delete the value NodeStructureSettings

Only after both of these are done, did the EMC correctly rediscover an active Exchange 2010 server. If one or there other is done, the incorrect server information is retained.

Getting the Database into a Clean Shutdown State

In order for Exchange to mount a database, it needs to be in a clean shutdown state. I’ll use the eseutil tool to play any outstanding transactions into the database to get it clean. Before I begin, I’ll open a command prompt, switch to the directory that contains the database and logs, and use the following command to view the status:

eseutil /mh DB01.edb

When reviewing the output, the database state will be reported as Dirty Shutdown:

What I will do next is perform a soft recovery to get the database consistent. I’ll run the following command to do this:

eseutil /r e01 /d

The /r specifies that I’m doing a soft recovery. The e01 is the log generation prefix for the database. I’m using the /d switch without any arguments to specify the database path, which is in the current directory. Since the logs are also located here, I don’t need to use the /l switch, as it defaults to the current path. Once the operation has completed successfully, I can runeseutil again with the /mh switch to verify the database is clean shutdown:

Now that my database has been restored and brought to a clean shutdown state I can create the Recovery Database.

Creating the Recovery Database

The next step in the process is to create the Recovery database using the database files restored from the backup. To do this, I’ll use the New-MailboxDatabase cmdlet with the following syntax:

New-MailboxDatabase -Name RecoveryDB -EdbFilePath E:\RecoveryDB\E_\DB01\DB01.edb -LogFolderPath E:\RecoveryDB\E_\DB01 -Recovery -Server mbx1

Notice that I’ve specified the path to the database and log files using the location where the database was restored. Also, the key to creating the Recovery database is to make sure you use the -Recovery switch parameter, as shown above. You can see I got a warning message after running the command stating the Recovery database was created using an existing file, and that I need to ensure that the database is in a clean shut down state before I try to mount it. I already did this in the previous step, so I can safely ignore this message and mount the database using the following command:

Mount-Database RecoveryDB

The Recovery database is now mounted, and I’m ready to restore mailbox data.

Finding Mailboxes and Performing a Simple Mailbox Restore

Now that my Recovery database is online, I need to be able to see what mailboxes are available for restores. I can use the Get-MailboxStatistics cmdlet to do this:

Get-MailboxStatistics -Database RecoveryDB

If you’re looking for a specific mailbox, you can filter the results using the following syntax:

Get-MailboxStatistics -Database RecoveryDB | ?{$_.DisplayName -like 'Mike*'}

You can see in this command I’ve used the ? alias for the Where-Object cmdlet. I’m using the -like operator to filter the results and only show me the mailboxes that start with Mike.

When restoring mailbox data from a Recovery database in Exchange 2010 SP1, we use the New-MailboxRestoreRequestcmdlet. When running this cmdlet, the source mailbox in the recovery database needs to be identified using one of three possible values; the DisplayName, MailboxGUID, or LegacyExchangeDN values. Keep in mind that you cannot reference the source mailbox using the Exchange Alias when performing a restore.

So, let’s take a look at the restore process. Based on the previous commands I can see that there is a copy of my mailbox in the Recovery database. To do a complete restore of the mailbox data to the original mailbox that is currently active in the production database I’ll use the following command:

New-MailboxRestoreRequest -SourceDatabase RecoveryDB -SourceStoreMailbox 'Mike Pfeiffer' -TargetMailbox mpfeiffer

Depending on the size of the mailbox, it may take quite some time to perform the restore. I can keep tabs on the progress using the following one-liner:

Get-MailboxRestoreRequest | Get-MailboxRestoreRequestStatistics

Dealing with Multiple Mailboxes with the same DisplayName

It’s possible that a Recovery database will have multiple mailboxes with the same display name. This can happen if there were one or more disconnected versions of a mailbox, in addition to an active mailbox, in the same database during the time of the back up. In this case, you can use the MailboxGuid value to identify the source mailbox when doing a restore. Consider the following:

Get-MailboxStatistics -Database RecoveryDB | ?{$_.DisplayName -like 'Isabel*'} | fl DisplayName,MailboxGuid,DisconnectDate

Here you can see that there are two mailboxes with the same display name in the Recovery database. One has aDisconnectDate defined, meaning it is a disconnected mailbox, and the other one does not which means it was the active mailbox in the database at the time of the backup. If you run into a scenario where there are multiple mailboxes in a database with the same display name, use the above command to determine the MailboxGuid of each mailbox. You can then use this value to identify the correct mailbox when performing a restore.

New-MailboxRestoreRequest -SourceDatabase RecoveryDB -SourceStoreMailbox 4a1d2118-b8cc-456c-9fd9-cd9af1f549d0 -TargetMailbox ihill

Restoring Individual Mailbox Folders

Here you can see that I am using the -IncludeFolders parameter to specify that only data from the Inbox should be restored from the mailbox in the recovery database:

New-MailboxRestoreRequest -SourceDatabase RecoveryDB -SourceStoreMailbox administrator -TargetMailbox administrator -IncludeFolders '#Inbox#'

The -IncludeFolders will accept a list of one or more mailbox folders. You can specify well-known folder names as well as personal folders using this parameter. Notice that the value needs to be enclosed in hash marks (#). For example, if you wanted to restore only the contacts folder, use #Contacts#, or #Tasks# for the Tasks folder, and so on. For more details, check out the help for this parameter in the TechNet documentation for the New-MailboxRestoreRequest cmdlet. If you simply want to restore a single root folder, check out the -SourceRootFolder parameter.

Restoring to an Archive Mailbox

Restoring a mailbox to a users archive is as simple as tacking on the -TargetIsArchive switch parameter to our original restore command. Here’s the command and output:

New-MailboxRestoreRequest -SourceDatabase RecoveryDB -SourceStoreMailbox 'Mike Pfeiffer' -TargetMailbox mpfeiffer -TargetIsArchive

Obviously, you’ll need to ensure that the target mailbox has been archive enabled for this to work.

Restoring to an Alternate Mailbox

By default, the New-MailboxRestoreRequest cmdlet looks for a matching LegacyExchangeDN on the source and destination mailbox, or checks to see that an X500 proxy address on the target mailbox corresponds to the LegacyExchangeDN on the source mailbox. This ensures that you do not accidentially restore mailbox data to the wrong location. If you need to restore data to an alternate mailbox, you can use the -AllowLegacyDNMismatch switch parameter:

New-MailboxRestoreRequest -SourceDatabase RecoveryDB -SourceStoreMailbox 'Mike Pfeiffer' -TargetMailbox administrator -TargetRootFolder Restore -AllowLegacyDNMismatch

In this example, I am restoring the data from my mailbox in the recovery database to a sub-folder of the administrator mailbox called Restore. Here’s a screen shot of the administrator mailbox after running the above restore command:

Be careful when restoring to alternate mailboxes. If you omit the -TargetRootFolder parameter, the data will be restored and merged into the existing folders in the target mailbox. On the other hand, that might be exactly what you want — if so, just remove the -TargetRootFolder parameter.

Bulk Restores

You might find yourself in a situation where you need to restore data from all mailboxes in a recovery database. For example, let’s say you need to restore the Contacts folder for all of your mailboxes. In this case, you could use a foreach loop to iterate through each mailbox in the recovery database and restore that particular folder to the active mailboxes:

foreach($mailbox in Get-MailboxStatistics -Database RecoveryDB) {
  New-MailboxRestoreRequest -SourceDatabase RecoveryDB -SourceStoreMailbox $mailbox.DisplayName -TargetMailbox $mailbox.DisplayName -SourceRootFolder Contacts
}

This might take a while; you can monitor the progress using the Get-MailboxRestoreRequest with the -Status parameter:

Get-MailboxRestoreRequest -Status Queued

As you’ve seen, there are a lot of steps and multiple options when it comes to restoring data from a recovery database. Obviously, this is not something you want to learn on the fly when a disaster strikes. I’d highly recommend documenting and testing your restore procedure on a regular basis.

Here’s a little time-saving tip for working with mailboxes in Exchange 2010.  Normally if you need full access to another user’s mailbox in Exchange 2007/2010 you would highlight their mailbox in the Exchange Management Console and the choose “Manage Full Access…” from the action pane.  You would have to do this individually for each mailbox that you wanted to manage full access permissions for.

Here’s an easier way to grant full access to a user for every mailbox in a mailbox database.  Just edit the bracketed fields below and paste the cmdlet in the Exchange Command Shell:

Get-MailboxDatabase -identity “[mailbox database name]” | Add-ADPermission -user [username] -AccessRights GenericAll

Voila!  You now have full access to all mailboxes in the selected database.  This even applies to new accounts created after you run the cmdlet.