Complex XenDesktop use cases; common mistakes; tools and techniques for resolution

Baptiste Duflos

Manager, Escalation Services

May 8th, 2012

Introduction and objectives

#CitrixSummit

Tweet about this session with hashtag #SUM301and #CitrixSummit

#CitrixSummit

Focusing on the major components of XenDesktop

4

WI Controllers

VDAsAD

VM Host(XenServer, Hyper-V, VMware)

Licensing

User

SQL Database

#CitrixSummit

Deploying Controller Servers

5

• All Controllers load balance session

launch and VDA registrations

• Configuring Controllers in an N+1

configuration allows for resiliency in

case of a failure

• All Controllers talk to the SQL database

and should deployed as close as

possible

Controllers

#CitrixSummit

Controller Server Scalability

6

XD4.x:

XD5.x:

Broker

Broker (ZDC)

Broker

Hypervisor Pool

Controller

Controller

Controller(failed)

WI

WI

Hypervisor Pool

Hypervisor Pool

Hypervisor Pool

Hypervisor Pool

Hypervisor Pool

#CitrixSummit

Controllers – Scalability and Best Practices

7

• Can overwhelm the hosting infrastructure with power state requests during

peak times when many users logon and off.

• You can throttle the amount of power commands sent per Controller with

“MaximumTransitionRate” – default is 20, do NOT increase it won’t speed up

power up times

#CitrixSummit

Deploying SQL for XD Databases

8

• XD 5 uses a single database with

multiple schemas that map to XD

services

• Stored procedures are leveraged to

reduce load on database

• Database is critical to XD 5 – all

Controllers have heartbeat to database

SQLDatabases

#CitrixSummit

SQL – Database Mirroring

9

• Database failure = Controller Failure

○ Only impacts new connections – existing or disconnected sessions not affected

• Citrix recommends leveraging SQL Mirroring for fault tolerance

Principal database

Mirror database

Transaction log

• Mirroring sends transaction log

from Principal database to the

redundant database

• If the principal database fails,

user intervention is required to

fail over the database

• Citrix recommends using

synchronous database

mirroring with witness

Witness Server

#CitrixSummit

SQL – Best Practices

10

• SQL transaction log is critical to monitor○ Connection launches and idle desktops consume transaction log space○ Use a fixed-size transaction log – auto-growth feature could impact response times○ Leverage SQL Alerts when log reaches thresholds (recommend 50%)

• Database failover tuning – adjust Controller heartbeat interval ○ Default heartbeat is 30secs and requires a SQL operation○ Controllers unregister workers that do not heartbeat for over one minute○ Controlled by Regkey: HKLM\Software\Citrix\DesktopServer\HeartbeatPeriodMs

#CitrixSummit

Deploying Virtual Desktop Agents

11

• VDA now uses “registry based”

registration by default

• Verify ports are open and firewall

configured

• Forward and Reverse DNS is required

Virtual Desktop Agents

#CitrixSummit

VDA – Scalability and best practice

12

• Increase the Service timeouts if you expect periods with large amount of VMs

rebooting – increase to 3 mins recommended

• Optimize the logon process – improves desktop performance

• Plan staged deployments and consider leveraging tools such like LoginVSI to

perform scale and load testing before adding large groups of users to

environment

#CitrixSummit

Key points to remember

13

• Controllers are resilient and scale well – keep deployments simple

• SQL server plays pivotal role in infrastructure – protect it!

• Make your end users happy – tune your VDAs for performance

#CitrixSummit

Troubleshooting a session launch failure

14

• Users were reporting they got an error

when trying to launch their desktops

• Admin noticed that intermittently VDAs

would de-register at session launch

Case StudyWalkthrough

#CitrixSummit

Environment overview

15

• XenDesktop deployment with:○ Web Interface 5.4○ XD 5.6 ○ SQL 2008○ VMWare 5.0○ Windows 2008 R2 Active Directory○ Virtual Desktop Agent OS – Win7 32-bit○ Citrix Receiver 3.1

Web Interface 5.4XD 5.6SQL 2008VMWare 5.0Active DirectoryVDAsReceiver

#CitrixSummit

What did failure look like?

16

#CitrixSummit 17

User attempts to start the session

1011011010 SSL 1011011010 SSL 1011011010 SSL 1011011101101110 111011011010 SSL 1011011010 SSL 1011011010 SSL 1011011101101110 11

#CitrixSummit

Initial Troubleshooting

18

• How often does it happen?

• Any particular timeframe it happens?

• Any specific users or images it happens more frequently with?

• What changed?

• Any event viewer messages?

#CitrixSummit

What changed?

19

WI

Controller #1

VDAs

Controller #2 VDAs register on Controller #1

Customer had single server deployment

Customer added second Controller for redundancy

#CitrixSummit

Where do we start looking?

20

• We found 4 interesting messages in Event Viewer:

Warning – Event ID 2103:An unexpected exception occurred while the Citrix Broker Service processed an XML transaction. An incompatible client might be trying to access the XML service. Verify the compatibility of clients accessing the service. If this problem persists, reinstall the Citrix XenDesktop Controller.

Error details: Transaction: 'RequestAddress' Exception Type: 'System.ServiceModel.Security.SecurityAccessDeniedException'

Application Warning – Event ID 1060:The Citrix Broker Service failed to apply settings on the virtual machine 'KB-WIN7-01.get.services.citrite.net'.

Check that the virtual machine can be contacted from the Controller and that any firewall on the virtual machine allows connections from the Controller. See Citrix Knowledge Base article CTX126992.

Error details: Exception 'Access is denied.' of type 'System.ServiceModel.Security.SecurityAccessDeniedException'.

Warning – Event ID 1039:The Citrix Broker Service failed to contact virtual machine 'KB-WIN7-01.get.services.citrite.net' (IP address ).

Check that the virtual machine can be contacted from the Controller and that any firewall on the virtual machine allows connections from the Controller. See Citrix Knowledge Base article CTX126992.

Error details: Exception 'Access is denied.' of type 'System.ServiceModel.Security.SecurityAccessDeniedException'.

Warning – Event ID 1101:The Citrix Broker Service failed to broker a connection for user 'GET\atladmin' to resource 'KB-Win7-PW'.

The Citrix Broker Service cannot find any available virtual machines. Please add more virtual machines to the site. If the problem is due to existing virtual machines not becoming available, see Citrix Knowledge Base article CTX126992.

#CitrixSummit

Troubleshooting Methodology – verify environment

21

• Check Firewall configuration

• Active Directory mis-configuration

• Forward DNS and Reverse DNS

• Environmental checks:○ Check for time skew○ Default ports○ Port conflicts

#CitrixSummit

Troubleshooting Methodology – gathering data

22

• Run Citrix Scout

• TaaS beta

• Enable logging on both Controllers

• Run a CDFTrace

#CitrixSummit

Citrix Scout / XD Collector (CTX130147)

23

• Push button easy data collection system

•Makes data collection and upload push button easy

• Integrates data collected by Scout with the Citrix Tools as a Service

(TaaS) backend

•Simplifies data collection & analysis

#CitrixSummit

Tools as a Servicehttp://Taas.Citrix.com/Beta

Auto analysis health check

2 3

24

Data Collection

Recommendations tailored to YOU

1

Quickly collect and upload your data

#CitrixSummit

Enabling logging

• Enabling Controller Service

Logging - CTX127492

• CDF Control - CTX111961

25

Controller

#CitrixSummit

Digging deeper – Controller log analysis

26

CdsBroker:1:1:UpdateWorkerSettings configurationService.Set failed:System.ServiceModel.Security.SecurityAccessDeniedException: Access is denied.Server stack trace: at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg) at System.Runtime.Remoting.Proxies.RealProx"CdsBroker:1:1:UpdateWorkerSettings reject the worker (S-1-5-21-1123877020-465626563-3648135752-3586)"BrokerDAL:8:5:DAL >>> DeregisterWorker(S-1-5-21-1123877020-465626563-3648135752-3586, CommunicationFailure)"

BrokerDAL:8:5:DAL >>> DeleteBrokeredSessionOnPrepareFailure(LaunchToken=54711b77-4fce-4edc-b31e-937bc7dca341, SinBin=True)

#CitrixSummit

Using CDF Control

27

• With CDFControl you

can download the

public TMF files which

will allow you to parse

the CDF trace and

troubleshoot your issue

• Parsing the CDF trace and enabling the expert shader feature allows us to quickly find exceptions which are typically highlighted in orange

• High level failure is: “CdsWorkerAgent:8:5:UserAllowed: found no matching Controllers, access not allowed for user”

#CitrixSummit

Digging deeper – CDF trace log analysis

28

Initial trust failure:CdsWorkerAgent:8:5:CheckAccessCore: Calling delegate to provide SID listCdsWorkerAgent:8:5:CheckAccessCore: entered, have 1 trusted DDCsCdsWorkerAgent:8:5:UserAllowed: found no matching Controllers, access not allowed for user GET\KB-XD5-SP1-2$ S-1-5-21-1123877020-465626563-3648135752-3604

After worker Sin-Bin timeout:CdsWorkerAgent:1:1:Heartbeat to http://KB-XD5-03.get.services.citrite.net:80/Citrix/CdsController/IRegistrar rejected CdsWorkerAgent:2:1:EventLogManager decided to log event CDS_EVENT_WORKER_AGENT_HEARTBEAT_REJECTED of type Warning

Re-Registered (after timeout expires):CdsWorkerAgent:2:1:Succesfully registered with http://KB-XD5-03.get.services.citrite.net:80/Citrix/CdsController/IRegistrar; starting heartbeats

#CitrixSummit

Under the hood - VDA Session Launch explained

29

VDA

VDA

Controller #1

Desktop Service

Controller #2 SQL

User

WI

VDA registers to Controller #1 Worker flagged in DB as Ready

User launches session

WI Sends launch request to XML Broker

XML broker queries DB for a ready worker

XML sends PrepareSession ticket to VDA

XML Broker unregisters worker

ListOfDDCs=Controller #1

VDA checks ListOfDDCs to authorize PrepareSession

Controller #2 is not in ListOfDDCs, VDA invalidates session launch request XML Returns Error

to WI

WI Error returned to user

Broker Service

XML Broker

Worker is placed in SinBin

#CitrixSummit

Root Cause analysis

30

• The customer added a second Controller to handle XML requests for

redundancy

• As soon as the new Controller was added to the WI XML failover list it was

available to broker session launches by design

• Since the new Controller was not added as an authorized trusted agent

XenDesktop rejects the session logons

• Workstation agent de-registers temporarily and then attempts to re-register

#CitrixSummit

Resolution

31

• DDCs that handle authentication must be authorized agents and added to

“ListOfDDCs” registry value

• CTX132536 outlines the registry key and how to define broker groups

• Adding DDCs to WI XML failover list enables the ability for DDCs to handle

session logons

Resources discussed

#CitrixSummit

Optimal deployment recommendations

33

• CTX124087 - XenDesktop Modular Reference Architecture

• CTX127939 - XenDesktop 5 Database Sizing and Mirroring Best Practices

• CTX123244 - High Availability for Desktop Virtualization - Reference

Architecture

• CTX120760 - XenDesktop - Design Handbook

• CTX128700 - XenDesktop Planning Guide - XenDesktop Scalability

• Whitepaper - Benchmarking Citrix XenDesktop using Login Consultants VSI

#CitrixSummit

For More Information

34

• CTX132536 - Worker Unregisters at Session Launch

• CTX130147 - Citrix Scout

• CTX111961 - CDFControl

• CTX127492 - How to enable Controller Service Logging in XenDesktop 5

• CTX128075 - XDDBDiag: XenDesktop 5 Database Diagnostics

• CTX128909 - XenDesktop 5 Logon Process and Communication Flow

#CitrixSummit 35

Tools as a Servicehttp://Taas.Citrix.com/Beta

checkered racing shoes

Find out how to rev up environment maintenanceSee your Citrix pit crew in the expo hall with the

#CitrixSummit

We value your feedback!Take a survey of this session now in the mobile app

• Click 'Sessions' button

• Click on today's tab

• Find this session

• Click 'Surveys'

#CitrixSummit

Before you leave…

• Conference surveys are available online at www.citrixsummit.com starting Thursday, May 10○ Provide your feedback and pick up a complimentary gift at the registration desk

• Download presentations starting Monday, May 21, from your My Organizer tool located in your My Account