We have been assisting a customer with their move from on-premises Exchange Server 2013 (CU23) to Exchange Online. After running the Hybrid Configuration Wizard, federation testing unveiled a pre-existing condition with Autodiscover which resulted in the hybrid wizard not creating the Federation Trust or Organization Relationships properly.
There are some highly relevant articles explaining the way free/busy works between on-premises Exchange Server and Office 365. These should be reviewed as a first step in troubleshooting:
- Demystifying Hybrid Free/Busy: what are the moving parts?
- How to troubleshoot free/busy issues in a hybrid deployment of on-premises Exchange Server and Exchange Online in Office 365
In our case, following these articles still left us with a scenario where any on-premises mailbox could not see free/busy information for any mailboxes in Exchange online.
Digging deeper, other articles got us closer to the root of the problem and lead us to run the ‘Get-AuthConfig’ command from on-premises Exchange. The results showed the following:
As can be seen, the certificate used for OAuth is expired. As it turns out, this certificate expires 5 years after installing Exchange. While online searches could not provide a definitive answer, it seemed to be the most likely cause. We then executed the commands in the following article to generate a new certificate:
Even with the new certificate assigned, the issue persisted.
This is where the Microsoft Test Connectivity Tool became our best friend. With Autodiscover working, the tool can be run to test free/busy. This revealed the following result:
Performing Free/Busy LookupFree/Busy Lookup failed.
Additional Details
Free/Busy Lookup failed with exception: Autodiscover failed for email address
This exact error was hard to find, but the following article makes reference to very similar errors and issues:
Piecing solutions together lead us to running the following from the on-premises Exchange Server:
Test-OAuthConnectivity -Service EWS -TargetUri 'https://outlook.office365.com/ews/exchange.asmx' -Mailbox newsig2@blah.com -Verbose |fl
The results of which returned the following error:
Error:[ACSTokenBuildRequest:GetActorTokenFromAuthServer] Unable to get the token from auth server blah. The request has token {blah, blah, the error from ACS is {"error":"invalid_client","error_description":"AADSTS700027: Client assertion contains an invalid signature. [Reason - The key was not found., Thumbprint of key used by client:'D687XXXXXXXXXXXXXXXXC8CE']\r\nTrace ID: blah: The remote server returned an error: (401) Unauthorized. blah Error:Unable to get token from Auth Server. Error code: 'invalid_client'. Description: 'AADSTS700027: Client assertion contains an invalid signature. [Reason - The key was not found., Thumbprint of key used by client: 'D687XXXXXXXXXXXXXXXXC8CE'] Trace ID: blah, blah
Exchange Response Details:
HTTP response message:
Exception:
System.Net.WebException: The request was aborted: The request was canceled. ---> Microsoft.Exchange.Security.OAuth.OAuthTokenRequestFailedException: Unable to get token from Auth Server. Error code: 'invalid_client'. Description: 'AADSTS700027: Client assertion contains an invalid signature. [Reason - The key was not found., Thumbprint of key used by client: 'D687XXXXXXXXXXXXXXXXC8CE']
Trace ID: blah ---> System.Net.WebException: The remote server returned an error: (401) Unauthorized.
at System.Net.HttpWebRequest.GetResponse()
at blah, blah, blah
From that we were able to determine that the OAuth certificate that was replaced also had to be uploaded to Azure. We confirmed this by running the following command against Exchange online:
Get-MsolServicePrincipalCredential -ServicePrincipalName “00000002-0000-0ff1-ce00-000000000000” -ReturnKeyValues $true
Though somewhat buried and hard to find, a different article had the steps/scripts to upload the new certificate (steps 3 and 4 only):
Voilà! Free/busy time can now be seen for both incoming and outgoing requests.
Though documents could be found to help with the troubleshooting process, nothing could be found that tied everything together neatly and back to the expired OAuth cert. Hopefully this post helps someone along in a similar scenario.