Page 1 of 23
1 LYNC/ EXCHANGE TROUBLESHOOT GUIDE
TABLE OF CONTENTS
1 Lync/ Exchange Troubleshoot Guide ................................. 1
2 Troubleshooting Exchange Configuration/Functionality .... 2
2.1.1 ENABLING EXCHANGE TRACING ........................................................................... 2
2.1.2 EXCHANGE PROTOCOL LOGS ............................................................................... 3
2.1.3 VERIFYING CONNECTION BETWEEN LYNC AND EXCHANGE SERVER ............................... 4
2.1.4 TSG: CHECKING TENANT PROVISION/EXISTING IN ACS .............................................. 5
2.1.5 TSG: AUTODISCOVER AND/OR EWS REQUESTS FAIL WITH SSL/TLS ERRORS, “THE
UNDERLYING CONNECTION WAS CLOSED: COULD NOT ESTABLISH TRUST RELATIONSHIP FOR THE
SSL/TLS SECURE CHANNEL” ....................................................................................... 6
2.1.6 TSG: "UNABLE TO READ DATA FROM THE TRANSPORT CONNECTION: AN EXISTING
CONNECTION WAS FORCIBLY CLOSED BY THE REMOTE HOST." ........................................... 7
2.1.7 TSG: CREATEAPPACTASTOKEN FAILED, EX=STORECONFIGEXCEPTION:
CODE=ERROROAUTHCONFIGISSUER, REASON=CANNOT LOCATE ISSUER. ENSURE CONFIGURED
LOCAL/TENANT/GLOBAL ISSUERS ARE ACCEPTED ............................................................ 8
2.1.8 TSG: LYNCACS RST REQUEST FAILS WITH ACS50024: JWT TOKEN IS INVALID. ACS50028:
PRINCIPAL WITH NAME… .......................................................................................... 9
2.1.9 TSG: LYNC ACS RST REQUEST FAILS WITH ACS50027: JWT TOKEN IS INVALID ............ 9
2.1.10 TSG: CLIENT SIGNS IN USING WINDOWS AUTH .................................................... 11
2.1.11 TSG: AUTODISCOVER/EWS FAILS WITH OAUTHCONFIGEXCEPTION/
CRYPTOGRAPHICEXCEPTION KEYSET DOES NOT EXIST ................................................... 12
2.1.12 TSG: AUTODISCOVER/EWS REQUESTS FAIL WITH 401 REASON="THE TOKEN HAS INVALID
SIGNATURE.";ERROR_CATEGORY="INVALID_SIGNATURE" ............................................... 14
2.1.13 TSG: NOT ABLE TO REGISTER LYNC PARTNER APPLICATION AT EXCHANGE SERVER ....... 16
2.1.14 TSG: REDIRECTION STILL POINTS TO EXCHANGE ONPREM ...................................... 17
2.1.15 TSG: PICKING UP RANDOM WEB PROXY AUTOD ISSUE ........................................... 18
2.1.16 TSG: ACCESS DENIED WHEN LYNC STORAGE CMDLET EXECUTED .............................. 20
3 Lync-ACS-Exchange Message Flow ................................. 22
Page 2 of 23
2 TROUBLESHOOTING EXCHANGE CONFIGURATION/FUNCTIONALITY
2.1.1 ENABLING EXCHANGE TRACING
Run "extra" from a command shell to bring up Microsoft Exchange Troubleshooting Assistant. Click “Cancel this check” if Checking for Updates starts executing.
Click “Select a task”.
Click “Trace Control”, note the default trace folder/file path, e.g. C:\Users\Administrator\ExchangeDebugTraces.etl
Click “Set Manual Trace Tags”
Select trace tags and components relevant to the functionality being troubleshooted.
For AuthN/AuthZ issues, select Security component and all
trace tags (should include OAuth).
Click “Start Tracing”,
Repro the issue and run through the scenario you’re troubleshooting.
Click “Stop Tracing Now”, note the trace log path, e.g c:\Users\Administrator\ExchangeDebugTraces.etl
Convert .ETL files to text using one of the following methods :
For Exchange deployed via Lync VM Test infra: Use RPC
Trace Decoder tool
Exchange support tool has a RPC Trace Decoder tool includes a Get-EtwTrace command.
1. Install Exchange Support Tool from c:\Exchange15\Debugging\Exchange14SupportTools.ms
i, must use .msi built with Exchange.
2. Copy
"c:\Exchange15\Debugging\internal.exchange.shared
.Win32.dll"
Page 3 of 23
"C:\Program Files\Microsoft\Exchange
Support\Tools\"
3. In the support shell (You can launch the support shell by
opening <D:\Program Files\Microsoft\Exchange Support\Tools and double clicking
RpcTD_DebugConsole.psc1>), run
Get-EtwTrace
c:\Users\Administrator\ExchangeDebugTraces.etl |
export-csv
c:\users\Administrator\ExchangeDebugTraces.csv
For Exchange deployed via http://tdsweb: Use ExTrace.exe
Use ExTrace.exe to convert .ETL files to text, for example:
extrace
c:\Users\Administrator\ExchangeDebugTraces.etl
For more info read… How to collect Exchange Product traces
More verbose version of above steps.
How to view Exchange trace files.
Describes how to view tracing real time and convert .ETLs to CSV.
2.1.2 EXCHANGE PROTOCOL LOGS
CSV based protocol logs used for diagnostics and reporting are stored in: %ProgramFiles%\Microsoft\Exchange
Server\V15\Logging\Ews
%ProgramFiles%\Microsoft\Exchange
Server\V15\Logging\AutoDiscover
Below are some of the fields logged… DateTime,RequestId,AuthenticationType,IsAuthenticated,AuthenticatedUser,Organization,UserAgent,ClientIpAddress,ServerHostName,SoapAction,HttpStatus,RequestSize,ResponseSize,ErrorCode,ImpersonatedUser,Cookie,CorrelationGuid,BeginBudgetConnections,EndBudgetConnections,BeginBudgetH
Page 4 of 23
angingConnections,EndBudgetHangingConnections,BeginBudgetAD,EndBudgetAD,BeginBudgetCAS,EndBudgetCAS,BeginBudgetRPC,EndBudgetRPC,BeginBudgetFindCount,EndBudgetFindCount,BeginBudgetSubscriptions,EndBudgetSubscriptions,MDBResource,MDBHealth,MDBHistoricalLoad,ThrottlingPolicy,ThrottlingDelay,ThrottlingRequestType,TotalDCRequestCount,TotalDCRequestLatency,TotalMBXRequestCount,TotalMBXRequestLatency,TotalExchangePrincipalLatency,TotalAuthNLatency,TotalAuthZLatency,PreExecutionLatency,CoreExecutionLatency,TotalRequestTime,GenericInfo,AuthenticationErrors,GenericErrors
2.1.3 VERIFYING CONNECTION BETWEEN LYNC AND EXCHANGE
SERVER
1) Run Lync Server Management Shell in FE
2) Checking connection => Test-CsExStorageConnectivity –Sipurl [email protected] –Verbose
3) Checking notification => Test-CsExStorageNotification –Sipurl [email protected] –Verbose
Notification works only if Storage Web Service is running. Refer the screenshot below.
Page 5 of 23
2.1.4 TSG: CHECKING TENANT PROVISION/EXISTING IN ACS
If tenant does not provision correctly or configured, it may cause problem
in service scenario between Lync and Exchnage.
Copy this to your local machine and execut “AcsConfig CheckTenantExist -
Env INT-SN1-004 -Name 7e36e953-bd60-499f-9227-9c0958bb0ebb”
highlights are different based on your target/tenant. The following
screenshot is expected if tenant provisioned correctly. “acsconfig ?” gives
us more option.
Page 6 of 23
2.1.5 TSG: AUTODISCOVER AND/OR EWS REQUESTS FAIL WITH
SSL/TLS ERRORS, “THE UNDERLYING CONNECTION WAS
CLOSED: COULD NOT ESTABLISH TRUST RELATIONSHIP FOR
THE SSL/TLS SECURE CHANNEL”
Autodiscover and/or EWS requests fail with SSL/TLS errors
The request failed. The underlying connection was closed: Could not
establish trust relationship for the SSL/TLS secure channel. --->
System.Net.WebException: The underlying connection was closed: Could
not establish trust relationship for the SSL/TLS secure channel. --->
System.Security.Authentication.AuthenticationException: The remote
certificate is invalid according to the validation procedure.
at
Microsoft.Exchange.WebServices.Data.EwsHttpWebRequest.Microsoft.Exch
ange.WebServices.Data.IEwsHttpWebRequest.GetResponse()
at
Microsoft.Exchange.WebServices.Autodiscover.AutodiscoverRequest.Intern
alExecute()
--- End of inner exception stack trace ---
at
Microsoft.Exchange.WebServices.Autodiscover.AutodiscoverRequest.Intern
alExecute()
As of CL#977086 for Office15:2245853, Lyss no longer ignores cert errors
(e.g. subject mismatch), need to either:
Page 7 of 23
A) Ensure IIS hosting EWS and Autodiscover web services is configured
with server certificates exactly/wildcard matching the fqdn of requests that
LYSS sends.
B) Or, less preferably, disable checking certs in LYSS's LysSvc.exe.config
file (see LyssTestSetup.cmd in CL#977086).
Try using Fiddler (enabled SSL) to track down which requests are failing
due to DNS and/or cert issues, if you are
new to fiddler consider reading VoIP is
not a Four Letter Word.
HACK: Suppress “Mismatch Address”
ceritificate errors when running Fiddler
while LYSService and unit test client are
running,configure via tools menu
fiddler options menu item HTTPs tab to capture HTTPS, decrypt
HTTPS and ignore server cert errors:
2.1.6 TSG: "UNABLE TO READ DATA FROM THE TRANSPORT
CONNECTION: AN EXISTING CONNECTION WAS FORCIBLY
CLOSED BY THE REMOTE HOST."
- Autodiscover, OWA and other functionality dependant on backend IIS
hosted web services fails with
"Unable to read data from the transport connection: An existing connection
was forcibly closed by the remote host."
- Application event log contains warning entries logged by FrontEnd HTTP
Proxy about unhandled exceptions.
Page 8 of 23
Backend website is misconfigured.
Configure binding for backend website with appropriate certificate. Select a
wildcard cert for a single Exchange Server deployment handling requests
for multiple endpoints (e.g. Autodiscover, EWS, etc..).
2.1.7 TSG: CREATEAPPACTASTOKEN FAILED,
EX=STORECONFIGEXCEPTION:
CODE=ERROROAUTHCONFIGISSUER, REASON=CANNOT
LOCATE ISSUER. ENSURE CONFIGURED
LOCAL/TENANT/GLOBAL ISSUERS ARE ACCEPTED
CreateAppActAsToken failed, ex=StoreConfigException: code=ErrorOAuthConfigIssuer, reason=Cannot locate issuer. Ensure configured local/tenant/global issuers are accepted, host=autodiscover.exhb-99915dom.extest.microsoft.com, acceptedIssuers=microsoft.lync@ocsmgmt1.rtmp.selfhost.corp.microsoft.com,[email protected]
Neither the local or tenant configuyred issuers match the "acceptedIssuers"
listed. Could be Lync and/or Exchange misconfiguration.
Examine previous tracing statements for this activity and check/fix issuer
identifier(s) registered with Lync/Exchange.
Page 9 of 23
2.1.8 TSG: LYNCACS RST REQUEST FAILS WITH ACS50024: JWT
TOKEN IS INVALID. ACS50028: PRINCIPAL WITH NAME…
ACS50024: JWT token is invalid. ACS50028: Principal with name
\u002700000004-0000-0ff1-ce00-000000000000@b269bfba-7188-4a1f-
93a7-a42831454e77\u0027 is not a known principal.
Mismatch between -ServiceName value specified in Lync's Set-
CSOAuthConfiguration and the application identifier registered with ACS
(via STSCfg.exe -addclient)
Ensure Lync's configured application identifier (default value 00000004-
0000-0ff1-ce00-000000000000) is registered with ACS and Exchange.
2.1.9 TSG: LYNC ACS RST REQUEST FAILS WITH ACS50027:
JWT TOKEN IS INVALID
Lync to ACS RST request fails with ACS50027: JWT token is
invalid. Example failure response:
HTTP/1.1 400 Bad Request
Cache-Control: private
Content-Type: application/json; charset=utf-8
x-ms-request-id: 18a9461e-e407-42ca-865f-4172c2cf16cb
X-Content-Type-Options: nosniff
Date: Tue, 07 Aug 2012 19:04:03 GMT
Content-Length: 273
Page 10 of 23
{"error":"invalid_client","error_description":"ACS50027: JWT token is
invalid. \r\nTrace ID: 18a9461e-e407-42ca-865f-
4172c2cf16cb\r\nTimestamp: 2012-08-07
19:04:04Z","error_codes":[50027],"timestamp":"2012-08-07
19:04:04Z","trace_id":"18a9461e-e407-42ca-865f-4172c2cf16cb"}
Failure can be due to variety of configuration issues, most probable cause
is that certificate used to sign the token doesn’t match the certificate
registered with the ACS tenant.
Verify configuration is correct. ACS traces maybe required to troubleshoot
some issues, traces for ACS INT environment can be accessed by dev/test
folks. Search for ACS traces by traceID and timestamp returned in the RST
response from ACS. Traces for Dogfood/Production environments are
restricted to ACS Team/Ops.
Viewed ACS traces using failure response above: https://test1.diagnostics.monitoring.core.windows.net/content/search/search.html?table=AadIntSN1WADLogsTable&start=2012-08-07+19%3a00%3a00Z&end=2012-08-07+19%3a10%3a00Z&query=%22Message.Contains(%22%2218a9461e%22%22)%22&utc=True Reading through the ACS Traces revealed that request failed due to token signature mismatch, the token was signed with a different certificate to what’s registered with ACS “ACS50027: JWT token is invalid.Microsoft.IdentityModel.Tokens.FailedAuthenticationException : Invalid signature.”
Page 11 of 23
Fixed by verifying cert registered with ACS and fixed Lync configuration: set-CsCertificate -Type OAuthTokenIssuer -Thumbprint ac14159171f9b7a763300debd09057feaf044f38
2.1.10 TSG: CLIENT SIGNS IN USING WINDOWS AUTH
Lync Client application cannot sign in
Authentication may be using Live ID
Page 12 of 23
In Lync Management Shell of Server VM:
1) Import-Module lync
2) Import-Module lynconline
3) Set-CsHostedWebAuthConfiguration -
UseClientCertAuthForWindowsAuth 0 -UseWsFedAuth 0 -Verbose
4) Set-CsWebServiceConfiguration -UseWindowsAuth 1 -
UseCertificateAuth 1 –Verbose
5) Set-CsProxyConfiguration -DisableNtlmFor2010AndLaterClients 0 –
Verbose
6) Edit hosts file : Server IP address(192.168.0.240) pool0.vdomain.com
2.1.11 TSG: AUTODISCOVER/EWS FAILS WITH
OAUTHCONFIGEXCEPTION/ CRYPTOGRAPHICEXCEPTION
KEYSET DOES NOT EXIST
OAuth fails, event log entry/tracing contains…
UnsupportedStoreException: code=ErrorIncorrectExchangeServerVersion,
reason=GetUserSettings failed, [email protected],
Autodiscover
Uri=https://autodiscover.pocket.org/autodiscover/autodiscover.svc,
Autodiscover WebProxy=<NULL> --->
Microsoft.Exchange.WebServices.Data.ServiceRequestException: The
request failed. The request was aborted: The request was canceled. --->
System.Net.WebException: The request was aborted: The request was
canceled. ---> Microsoft.Rtc.Internal.Storage.OAuthConfigException:
Certificate with <SerialNumber, 791eebc300000000004f> by
<IssuerName, CN=myca> does not have private key or it is inaccessible or
not RSA, ex=System.Security.Cryptography.CryptographicException:
Keyset does not exist
Page 13 of 23
Either private key is missing from personal certificate imported into Lync
Front End’s machine store, and/or “Network Service” has not been granted
permissions to access the private key. Enable access from certificate
manager MMC snap-in:
Grant permissions to “Network Service”, for example:
Page 14 of 23
2.1.12 TSG: AUTODISCOVER/EWS REQUESTS FAIL WITH 401
REASON="THE TOKEN HAS INVALID
SIGNATURE.";ERROR_CATEGORY="INVALID_SIGNATURE"
Lync to Exchange autodiscover fails, Exchange returns 401 with x-ms-
diagnostics: 2000000;reason="The token has invalid
signature.";error_category="invalid_signature"
TL_VERBOSE(TF_DIAG) [1]35B4.1E14::05/11/2012-
22:55:37.451.0198ad72
(Lyss,ExchangeContext.EwsTraceListener.Trace:exchangecontext.cs(631))[
3676575611]type=AutodiscoverResponseHttpHeaders, msg=<Trace
Tag="AutodiscoverResponseHttpHeaders" Tid="29" Time="2012-05-11
22:55:37Z">
HTTP/1.1 401 Unauthorized
request-id: 3f2ca142-ab5b-48c3-bd04-14d60d3fccb9
X-FEServer: L04-OCG
x-ms-diagnostics: 2000000;reason="The token has invalid
signature.";error_category="invalid_signature"
Server: Microsoft-IIS/7.5
WWW-Authenticate: Bearer client_id="00000002-0000-0ff1-ce00-
000000000000", trusted_issuers="00000004-0000-0ff1-ce00-
[email protected],00000002-0000-0ff1-ce00-
[email protected]", error="invalid_token",Basic
realm="autodiscover.pocket.org",Negotiate,NTLM
X-Powered-By: ASP.NET
Date: Fri, 11 May 2012 22:55:35 GMT
Content-Length: 0
</Trace>
Page 15 of 23
TL_ERROR(TF_STACKTRACE) [1]35B4.1E14::05/11/2012-
22:55:37.452.0198ad73
(Lyss,ExchangeContext.GetUserEwsSettings:exchangecontext.cs(568))[367
6575611]UnsupportedStoreException:
code=ErrorIncorrectExchangeServerVersion, reason=GetUserSettings
failed, [email protected], Autodiscover
Uri=https://autodiscover.pocket.org/autodiscover/autodiscover.svc,
Autodiscover WebProxy=<NULL> --->
Microsoft.Exchange.WebServices.Data.ServiceRequestException: The request
failed. The remote server returned an error: (401) Unauthorized. --->
System.Net.WebException: The remote server returned an error: (401)
Unauthorized.
Exchange does not like the token signature, this indicates a mismatch
between the cert Lync signed the token with and the cert that Exchange is
configured with for the Lync Partner application.
Verify/correct cert that Lync is configured with and cert registered with
Exchange. Verify cert Exchange is configured with using Exchange
management shell’s Get-PartnerApplication, copy-paste the base64
encoded certificate data into a .cer text file, save and open the .cer file
from explorer to see the cert details.
Lync Storage Service picks up configuration changes within
seconds. Exchange requires iisreset in order for EWS to pickup recent
configuration changes. In addition, Exchange only periodically refreshes
cert data by query the AuthMetadataUrl configure for Lync partner
application. Force Exchange to query Lync’s autodiscover endpoint for
latest certs by setting AuthMetadataUrl to the existing value, e.g. Set-
PartnerApplication "Lync" -AuthMetadataUrl
https://pool1.pocket.org/metadata/json/1
Page 16 of 23
2.1.13 TSG: NOT ABLE TO REGISTER LYNC PARTNER
APPLICATION AT EXCHANGE SERVER
Sometimes, Exchange Management Shell cmdelt, New-PartnerApplication
returns 502 bad gateway in spite of AuthMetaUrl can be reachable. This is
Exchange bug and tracking #3076377.
[PS] D:\Program Files\Microsoft\Exchange Server\V15\Scripts>New-
PartnerApplication [email protected] -Enabled $true -AuthMetadataUrl
https://O04-mcs.exchangedc4.com/metadata/json/1 -LinkedAccount
"exchangedc4.com/Users/Exchange Online-ApplicationAccount"
Cannot acquire auth metadata document from 'https://O04-
mcs.exchangedc4.com'/metadata/json/1'. Error: The underlying connection
was closed: Could not establish trust relationship for the SSL/TLS secure
channel..
+ CategoryInfo : ResourceUnavailable: (:) [New-
PartnerApplication], AuthMetadataClientException
+ FullyQualifiedErrorId :
150AC300,Microsoft.Exchange.Management.SystemConfigurationTasks.Ne
wPartnerApplication
+ PSComputerName : o04-mcs.exchangedc4.com
Cannot acquire auth metadata document from 'https://O04-
mcs.exchangedc4.com'. Error: The remote server returned an
error: (502) Bad Gateway..
+ CategoryInfo : ResourceUnavailable: (:) [New-
PartnerApplication], AuthMetadataClientException
+ FullyQualifiedErrorId :
AFABAD18,Microsoft.Exchange.Management.SystemConfigurationTasks.Ne
wPartnerApplication
+ PSComputerName : o04-mcs.exchangedc4.com
Page 17 of 23
1) Use IPAddress instead of O04-mcs.exchangedc4.com
2) Download metadata manually and copy to Exchange IIS
2.1.14 TSG: REDIRECTION STILL POINTS TO EXCHANGE ONPREM
The redirection scenario, such as mailbox user migrated from Exchange
onprem to online, the redirected autodiscover url points still 1st Exchange
onprem as screenshot shown below.
This blocks to access Exchange online box and redirection scenario will not
work.
Check the following [PS] C:\Program Files\Microsoft\Exchange Server\V15\Scripts>New-AcceptedDomain lysshybrid.msol-test.com -DomainName lysshybrid.msol-test.com
Page 18 of 23
2.1.15 TSG: PICKING UP RANDOM WEB PROXY AUTOD ISSUE
This failed lyss connectivity to Exchange.
Add the following to %programfiles%\Microsoft Lync Server
2013\Server\Core\LysSvc.exe.config
<system.net>
<settings>
<servicePointManager
checkCertificateName="true" />
</settings>
<defaultProxy>
<proxy
usesystemdefault="true"
Page 19 of 23
proxyaddress="http://itgproxy.redmond.corp.microsoft.com:80"
bypassonlocal="True"
/>
<bypasslist>
<add address="[a-z]+\.lcspeer\.lcesa\.pri\.local" />
<add address="[a-z]+\.lcesa\.pri\.local" />
<add address="firstsite\.exchangelabs\.live-int\.com" />
</bypasslist>
</defaultProxy>
</system.net>
Page 20 of 23
2.1.16 TSG: ACCESS DENIED WHEN LYNC STORAGE CMDLET
EXECUTED
The current account does not belong to RTC Group.
Page 21 of 23
Add the account logged in currently to RTC Group (Computer
Management=>Local Users and Groups => Groups) below and then log
out/back in.
RTC Component Local Group
RTC Server Local Group
RTC Local User Administrators
Page 22 of 23
3 LYNC-ACS-EXCHANGE MESSAGE FLOW
|Agent LYSS| |ACS| | Exchange | |AutoD | EWS| ------> - Lync agent/LYSS-client sends request to LYSS WCF based Storage Service. LYSS internally dispatches request to an Adaptor which may need to perform some EWS operation. -----------------> - LYSS sends get user setting request to Exchange Autodiscover WS. <----------------- - Exchange fails to find OAuth token in request header returns 401. -------> - LYSS sends RST request to ACS for app token signed with cert trusted by Exchange, requests access for AutoD fqdn resource. - This step is skipped if token is already cached. <------- - ACS returns app token for AutoD resource. -----------------> - LYSS resends get user setting request with AppActAs token containing app token signed by ACS. <----------------- - Exchange Autodiscover WS verifies OAuth token and internally does AD/store lookups for User settings such as EWS endpoint. ---------------------------> - LYSS sends EWS request (e.g. CreateItem) to EWS endpoint, <--------------------------- - Exchange fails to find OAuth token in request header returns 401. -------> - LYSS sends RST request to ACS for app token signed with cert trusted by Exchange, requests access for EWS fqdn resource. - This step is skipped if token is already cached. <------- - ACS returns app token for EWS resource. ---------------------------> - LYSS resends EWS request (e.g. CreateItem) to EWS endpoint with AppActAs token containing app token signed by ACS. <--------------------------- - Exchange performs EWS operation and internally interacts with various resources depending on the operation. Eventually returning a response.
Page 23 of 23
Top Related