Horizon View Troubleshooting –Looking Under the Hood
Jack McMichael, VMware, IncMatt Coppinger, VMware, Inc
EUC4437
#EUC4437
• This presentation may contain product features that are currently under development.
• This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.
• Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.
• Technical feasibility and market demand will affect final delivery.
• Pricing and packaging for any new technologies or features discussed or presented have not been determined.
Disclaimer
CONFIDENTIAL 2
Identity Manager
ITUser
Horizon
Desktop
AirWatch
Mobile
Content
Collaboration
One Cloud
Workspace Suite
VMware Workspace Suite: Enabling Business Mobility
CONFIDENTIAL 3
CONFIDENTIAL 4
AirWatch Mobile:• A Leader for 5 Consecutive Years
• Placed Highest on Ability to
Execute Axis 3 Consecutive Years
Magic Quadrant
Figure. Magic Quadrant for Enterprise Mobility Management Suites
Source: Gartner, Inc., Magic Quadrant for Enterprise Mobility Management Suites, Terrence Cosgrove, et al, June 8 2015. &
Gartner, Inc., 2015 Critical Capabilities for Enterprise Mobility Management Suites, Terrence Cosgrove, et al, June 9 2015.
This graphic was published by Gartner, Inc. as part of a larger research document and should be evaluated in the context of the
entire document. The Gartner document is available upon request from AirWatch.
Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology
users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the
opinions of Gartner¹s research organization and should not be construed as statements of fact. Gartner disclaims all warranties,
expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.
Horizon Desktop:VMware leaps past the competition
“VMware's position reflects the company's market
position and commitment to providing resources to
expand its EUC product portfolio and infrastructure.”
CONFIDENTIAL 5
Agenda
CONFIDENTIAL 6
1 Common Issues
2 What can go wrong?
3 Domain 1: Horizon Client Connectivity
4 Domain 2: Desktop Availability
5 Domain 3: Application Management
6 Domain 4: Broken Broker
7 Domain 5: Performance
8 Resources
Top VMware Global Support Services Tickets
• SSL and Domains
• Connectivity – Understand Horizon network requirements!
• Persona and UEM
• Parent VM issues – Do NOT P2V! Clean VM please…
• PCoIP sizing
• Logs
CONFIDENTIAL 7
Working with SSL Certificates
Internal Certificate
Authorities are the easiest to setup, but not
the most trusted
Intermediate Certificate Authorities
require chaining and more work
to setup
Top level Certificate
Authorities are the most
expensive
CONFIDENTIAL 8
• https://pubs.vmware.com/horizon-view-60/topic/com.vmware.ICbase/PDF/horizon-view-60-scenarios-ssl-certificates.pdf
Horizon 6 SSL Guide
CONFIDENTIAL
Generating an SSL Certificate Signing Request
CONFIDENTIAL
Read the Guide!
Create a config file
Generate cert signing request
(CSR)
Validate CSR and
Private Key
Send CSR to CA
Receive Signed Cert
from CA
Import Cert
Configure Horizon
Server to use Cert
Test!
CONFIDENTIAL 11
Troubleshooting Keys
Check Horizon Administrator Dashboard
Understand the client connection process (where most problems lie)
Set the Logging Level on Broker
Check Connection Broker Logs
Check Horizon Agent Logs
Check Horizon Desktop PCoIP Logs
Use kb.vmware.com!
CONFIDENTIAL 12
You’ve Deployed Horizon. What Could Go Wrong?
You’ve got a problem:
• I can’t even connect to Horizon
• I get disconnected randomly!
• Why is the display so bad?
• Why is my desktop not available?
• I’m seeing an error in Horizon, what does it mean?
• vCenter is reporting an error
• My desktop is slow…
CONFIDENTIAL 13
Identifying the Problem Domain
CONFIDENTIAL 14
Horizon Client
Network
View Connection Server
View Composer
vCenter Server
Compute
Storage
Domain 1: Horizon Client Connectivity Issues
Common challenges
• Horizon Client can’t connect
• Logon failures
• Black screens
• Poor quality display
• Randomly disconnected session
CONFIDENTIAL 15
Domain 1: Horizon Client Connectivity Issues
Where to look
• Connection Broker logs
• Windows 2008 - <DriveLetter>:ProgramData\Application Data\VMware\VDM\logs
• Event Database
What to look for
• (Client connects) [SimpleAJPService] (ajp:broker:Request9) Request from /192.168.2.1: POST /broker/xml
• (Broker authentication) [WinAuthFilter] (SESSION:7072-***-a79c mattc) Attempting to authenticate user 'mattc' in domain 'FUTUREOFFICE’
• (User has authenticated to Broker) [AuthorizationFilter] (SESSION:7072-***-a79c) User FUTUREOFFICE\mattc has successfully authenticated to VDM
• (Audit Entry) [Audit] (SESSION:7072-***-a79c) BROKER_LOGON:USER:FUTUREOFFICE\mattc;USERSID:S-1-5-21-326850759-2560684469-1780228732-1113;USERDN:CN=S-1-5-21-326850759-2560684469-1780228732-1113,CN=ForeignSecurityPrincipals,DC=vdi,DC=vmware,DC=int;
• Event Database: BROKER_USERLOGGEDIN
CONFIDENTIAL 16
Domain 1: Horizon Client Connectivity Issues
CONFIDENTIAL 17
Track sessions using Tail/Grep utilities
CONFIDENTIAL 18
Domain 1: Horizon Client Connectivity Issues
Black Screen of Death
• PCoIP port blocked (TCP and UDP 4172) or SVGA Driver issue
• pcoip_server/client logs - C:\Users\All Users\VMware\VDM\logs
• Error attaching to SVGADevTap, error 4000: EscapeFailed
• MGMT_SCHAN :scnet_client_open: tera_sock_connectreturned error 10060 - Connection timed out!
• Incorrect PCoIP External URL configured for Security/Connection Servers
CONFIDENTIAL 19
Domain 1: Horizon Client Connectivity Issues
Poor quality display
• Bandwidth, latency, or QoS
• Pcoip_server logs report
• VGMAC :Stat frms: Loss=0.45%/0.21% (R/T)
• MGMT_PCOIP_DATA :BW: Decrease (loss) old = 234.9982 new = 176.8438
Randomly disconnected session?
• 15 min after established - wssm process hasn't started on desktop
• Horizon Agent logs C:\ProgramData\VMware\VDM\logs)
• PENDING_EXPIRED
• Sometimes caused by daisy-chaining the GINA
CONFIDENTIAL 20
Domain 2: Desktop Availability
• Common Issues
– No Desktop Available
– Pool provisioning issues –customization
– Agent not communicatingwith broker
– Stuck at desktop loginscreen (SSO)
• Where to look
– Connection Broker/Agent logs
– Event Database
• What to look for
• Broker returns list of desktops available to client
– [DesktopsHandler] (SESSION:7072-***-a79c) For user [S-1-5-21-326850759-2560684469-1780228732-1113] and pool [cn=gold-np,ou=server groups,dc=vdi,dc=vmware,dc=int] DesktopTracker returned 2 guest DNs
CONFIDENTIAL 21
Domain 2: Desktop Availability
Successful Connection Walkthrough
• Client requests desktop
• Event Database: BROKER_DESKTOP_REQUEST
• Broker allocates session to user
• [FarmImp] (SESSION:7072-***-a79c) cn=3f974017-409f-4912-83bc-2ee794f22fab,ou=servers,dc=vdi,dc=vmware,dc=int, total session count: 0
• [FarmImp] (SESSION:7072-***-a79c) allocateNewSession - identified server for application CN=GOLD-NP,OU=Applications,DC=vdi,DC=vmware,DC=int
• Event Database: BROKER_MACHINE_ALLOCATED
• Broker attempts SSO
• [FarmImp] (SESSION:7072-***-a79c) Using domain for SSO: FUTUREOFFICE**
• User won’t be logged on to the VM without this!
CONFIDENTIAL 22
Domain 2: Desktop Availability
Successful Connection Walkthrough
• Broker starts session on VM
• [DesktopSessionImp] (SESSION:7072-***-a79c) startSession – sending StartSession message
• Agent responds…
• "DesktopManager got a StartSession message”
• Client Info should be in Agent Log along with PCoIP launch
• Event Database: AGENT_PENDING
• [DesktopSessionImp] (SESSION:7072-***-a79c) startSession completed:
• DesktopTracker] User FUTUREOFFICE\mattc connected to machine gold-np-2 for desktop gold-np
• Client connects to VM (Agent)
• “PCoIPCnx::OnConnectionComplete Begin (PCOIP)”
• “WTS_SESSION_LOGON”
• Event Database: AGENT_CONNECTED
CONFIDENTIAL 23
Domain 2: Desktop Availability
Check the desktop’s status with ADSI
• Internal Horizon Database
• Installed on every broker
CONFIDENTIAL
24
Domain 2: Desktop Availability
Common ADSI Attributes:
• pae-DisplayName
• VM name as displayed in Horizon Admin
• pae-SVIVMSnapshot
• Indicates the current Snapshot that is in use
• pae-VmPath
• Indicates the full Path to the VM in vCenter
• pae-VmState
• Indicates the current state of the Desktop – some states are a combination of this value and other values
CONFIDENTIAL 25
Domain 2: Desktop Availability
• Find VMs with a Snapshot:
– (&(objectClass=pae-VM)(pae-SVIVmSnapshot=/Baseline/Snapshot1/Snapshot2))
• Find VMs with a Name:
– (&(objectClass=pae-VM)(pae-DisplayName=Desktop-234))
CONFIDENTIAL 26
Domain 2: Desktop Availability
Events Database
• Query for broker events or use Horizon Admin UI
• Horizon Event Notifierhttps://labs.vmware.com/flings/horizon-view-event-notifier
CONFIDENTIAL 27
Domain 2: Desktop Availability
Desktops not available due to provisioning error?
• Check View Administrator for Pool status, check datastore capacity
• Check Event Database -BROKER_PROVISIONING_ERROR_*
• Check View Composer has network access to ESX hosts
Desktop not available due to customization?
• Check Desktop status – AGENT UNAVAILABLE
• Check View Dashboard
• Desktop Status > Preparing Desktops OR Problem Desktops
• Check Desktop connectivity to DNS/AD/Connection Server
CONFIDENTIAL 28
Domain 2: Desktop Availability
Desktop not available due to VM reset/crash?
• Check Desktop status –ALREADY USED
• Typical on refresh-on-logoff or delete-on-use desktops
• Broker never received an explicit logout message from the agent
• Missing AGENT_ENDED event in DB for VM
View Composer Issues associated with incorrect domain credentials
• C:\ProgramData\VMware\View Composer\Logs\
• FATAL CSvmGaService -[svmGaService.cpp, 116] Domain join failed Error 5 (0x5): Access is denied.
CONFIDENTIAL 29
Domain 3: Application Management
• RDS and Group Policy Objects (computer and user policies)
– RD Licensing
– RD Connection and Client Connection
– Application Compatibility
– Device and Resource Redirection
– Session Environment and Time Limits
– Profiles and Temporary Folders
• More info: http://technet.microsoft.com/en-us/library/ee791756(v=ws.10).aspx
• FYI – we will be shipping custom ADM templates that are relevant to RDS integration with Horizon View
• We can and will honor existing RDS GPOs that are in use
CONFIDENTIAL 30
Domain 3: Application Management
RDS Applications
• Depending on Farm, can be one or multiple host sessions.
• Session limits per host (150 default)
• Licensing Diagnostics (user or device)
Best Practices
• No CPU Overcommitment
• Enable Hyperthreading
• Enable Transparent Page Sharing
• Enable Fixed Memory Allocation
• Disable Host Swapping
CONFIDENTIAL 31
Domain 4: Broken Broker
Common Issues
• Cannot connect to vCenter
• View Composer errors/issues
• JMS connectivity
• ADAM replication failure
Where to look
• View Administrator
• Event Database
• Windows Event Logs
• View Composer Logs
• Connection Server Logs
CONFIDENTIAL 32
Domain 4: Broken Broker
Broker Sizing Guidelines
• Common issues resulting from undersized brokers:
• Memory Heap Issues (In 5.x versions)
• Threading issues
• Latency Issues
Recommendations:
• Always size your broker right from the beginning, avoid downtime or issues later.
• Minimum recommended specifications:
• 2 vCPU / 10 GB RAM for normal sizing (50-500 VMs)
• 4 vCPU / 16+ GB RAM for large sizing (500+ VMs)
CONFIDENTIAL 33
Domain 4: Broken Broker
ADS Replication
• Check the Connection Broker Window Event Logs
– VMwareVDMDS log (Error: ADAM Replication)
• Check ADAM replication status on Connection Server:%systemroot%\adam\repadmin.exe /showrepl localhost:389 DC=vdi,DC=vmware,DC=int
CONFIDENTIAL 34
Domain 4: Broken Broker
vCenter Server Connectivity
• Admin UI will show RED status
• Check Event Database
• VC_DOWN events
• Impacts provisioning and power operations ONLY
• Check Connectivity from Connection Server to vCenter Server
• Check credentials used to connect to vCenter Server
• Attempt to login in directly to vCenter using vSphere Client
CONFIDENTIAL 35
Domain 4: Broken Broker
What to look for…
• View Composer
– VMs stuck in DELETING status
– VMs have been manually deleted – then pool/desktop deleted
• Causes Composer DB and VC DB to get out of sync
• Composer thinks VM already exists
• Orphaned VMs – KB-2015112 (kb.vmware.com)
• Desktop Composer Fault: 'Virtual Machine with Input Specification already exists‘
• JMS Connectivity
– Split site architecture / firewall causes “split brain”
– View Dashboard shows RED status
– Connection Server Logs
• tracker REJOIN messages – JMS connectivity
• tracker RESYNC messages – messages being delayed
CONFIDENTIAL 36
Cleaning Up Broken / Orphaned Pools
• BACKUP
• Disable Provisioning on Broker
• Stop View Composer
– Remove Composer Database Objects
– Remove ADLDS Servers, Server Groups, and Applications
– Remove AD Computer Entries
– Remove vCenter objects
• Unprotect replicas with sviconfig:
SviConfig -operation=RemoveSviClone -VmName=replica-<guid> -
AdminUser=administrator -AdminPassword=passowrd -
ServerUrl=https://localhost:18443/SviService/v2_0
Active Directory View Composer
AD LDS
CONFIDENTIAL 37
One Query To Rule Them All (Proceed with Extreme Caution!)
• Delete One VM
DELETE FROM dbo.SVI_VM_NAME WHERE NAME='replaceMe'
DELETE FROM dbo.SVI_COMPUTER_NAME WHERE NAME='replaceMe'
DELETE FROM dbo.SVI_SC_PDISK_INFO WHERE PARENT_ID=(SELECT ID FROM dbo.SVI_SIM_CLONE WHERE VM_NAME='replaceMe')
DELETE FROM dbo.SVI_SC_BASE_DISK_KEYS WHERE PARENT_ID=(SELECT ID FROM dbo.SVI_SIM_CLONE WHERE VM_NAME='replaceMe')
DELETE FROM dbo.SVI_TASK_STATE WHERE SIM_CLONE_ID=(SELECT ID FROM dbo.SVI_SIM_CLONE WHERE VM_NAME='replaceMe')
DELETE FROM dbo.SVI_REQUEST WHERE ID=(SELECT REQUEST_ID FROM dbo.SVI_TASK_STATE WHERE SIM_CLONE_ID=(SELECT ID FROM dbo.SVI_SIM_CLONE WHERE VM_NAME='replaceMe'))
DELETE FROM dbo.SVI_SIM_CLONE WHERE VM_NAME='replaceMe'
• Delete ALL VMs and Pools
DELETE FROM dbo.SVI_VM_NAME
DELETE FROM dbo.SVI_COMPUTER_NAME
DELETE FROM dbo.SVI_SC_PDISK_INFO
DELETE FROM dbo.SVI_SC_BASE_DISK_KEYS
DELETE FROM dbo.SVI_TASK_STATE
DELETE FROM dbo.SVI_REQUEST
DELETE FROM dbo.SVI_SIM_CLONE
DELETE FROM dbo.SVI_REPLICA
DELETE FROM dbo.SVI_DG_CUST_PROP
DELETE FROM dbo.SVI_DEPLOYMENT_GROUP
Note: Composer will auto clean replicas when no dependent SIM_CLONE objects are available. CONFIDENTIAL 38
Domain 4: Broker Broker
• Entitlements: Local vs Global
• Global always takes precedence
• Global Data Layer Replication
• Limits
• 2 Sites
• 4 Pods
• 20,000 Users
Cloud Pod Architecture
CONFIDENTIAL 39
Domain 5: Performance
• Common Issues
– Storage IO bottleneck
– Memory contention
– CPU contention
– Network issues
• Where to look
– vCenter Server
– ESXTOP
– 3rd Party Tools?
CONFIDENTIAL 40
Domain 5: Performance
What to look for
• CPU
– Cluster/Host utilization < 90%
– VM utilization - %USED (ESXTOP)
– VM %RDY Time (ESXTOP) < 10
• Memory
– Host utilization < 85%
– VM utilization
– Swapping / Ballooning SWCUR > 1 / MCTLSZ > 1 (ESXTOP)
• Storage
– Disk Read Latency < 25ms
– ESXTOP DAVG or KAVG < 25ms (ESXTOP)
CONFIDENTIAL 41
Domain 4: Performance
Optimize your Desktop Parent images!
CONFIDENTIAL 42
Domain 5: Performance
vRealize Operations Manager
• Aggregates metrics into workload, capacity and health scores
• Relies on dynamic thresholds
CONFIDENTIAL 43
vRealize Operations for Horizon
CONFIDENTIAL 44
Capturing ESX Performance Snapshots
• Use the following command to collect performance metrics for 8 hours on a host:
for i in `seq 8`;do esxtop -a -b -d 5 -n 720 > $i.<hostname>.csv;done
• The above command will create eight 100mb files consisting of 1 hour's worth of ESXTOP snapshots.
– -d = delay in seconds
– -n = iterations
– (-d 5 x -n 720 = 3600 seconds or 1 hour).
• Useful for replaying performance data over wide time periods for support to analyze!
CONFIDENTIAL 45
Summary
• Understand where the issue may lie
– Client? Agent? Server? Composer? vCenter? ESX?
• Know the problem domains
• Check the Horizon Dashboard and Event Database
• Identify the issue – know what a successful connections looks like
• Check the logs
• Use Performance Tools – ESXTOP, vCenter, vRealize Operations for Horizon
• Get Help
CONFIDENTIAL 46
Getting Help
• Read the manual! It’s searchable!
• Double check your configuration!
• Check kb.vmware.com for your issue
• http://communities.vmware.com
• Run Horizon “DCT Support” to extract the logs
– http://kb.vmware.com/kb/1017939
– Check the product documentation for using VDMADMIN command for creating various Data Collection Tool bundles
• Submit a Support Request
CONFIDENTIAL 47
QuestionsTwitter: @jackwmc4 / @mcopping
EUC VMworld 2015 Session Recommendations
CONFIDENTIAL 49
TechnicalApplication Lifecycle Management with
VMware Horizon EUC4561
Why Everybody Needs VMware User
Environment Manager (UEM) EUC5430
Beyond the Marketing:
Horizon 6 Technical Deep Dive EUC5052
Your Desktops Secured:
What Can NSX Do for You? EUC5062
Beyond the Marketing: VMware App
Volumes Technical Overview EUC5434
High Performance 30 Workloads on
Horizon 6 and NVIDIA GRID vGPUEUC5481
Access Point -EUC Common Gateway EUC6101
What's New with Horizon 6 with VOl and
Hosted Applications EUC6129
What's New in VMware Identity Manager EUC6105
Mobile Device Management or
Container: Do We Have to Choose?EUC5622
End-to-end Security with AirWatch. NSX
and Intelligent Networking EUC5762
Business/StrategyShow Me the Money !! Finding
Value in EUC – Why Identifying
Benefits Beyond Cost Matter Most
EUC5662
VMware’s End User Computing
(EUC) Strategy into 2015 and
Beyond
EUC5909
The Real Story of Customers
Delivering 3D Workstations with
VMware Horizon and NVIDIA GRID
EUC6621
Horizon Air: How to Provide an
Uncompromised User Experience at
a Cost Less than Physical
EUC6082
AirWatch 101: Enterprise Mobility
Management SimplifiedEUC5523
Peek Into the Future: Our Vision
for Business MobilityEUC5645-QT
Bring Your Own Content, We’ll Keep
it SecureEUC5764
Driving Business Mobility with
AirWatch and VMware IdentityEUC6098
The Future of End User Computing EUC6225-QT
VerticalHorizon at Point of Sales–We Did it
so Can You!EUC4919
Higher IT Pressure in Higher
Education: VMware End User
Computing Provides Instant Relief
EUC5664
Embracing Mobility in the
Federal SectorEUC5687
Mobilizing Healthcare: From Shared
to Personal Clinical Workspaces ...
And Beyond!
EUC5965
Financial Services Transformation in
Branch ComputingEUC6018
Horizon View Troubleshooting –Looking Under the Hood
Jack McMichael, VMware, IncMatt Coppinger, VMware, Inc
EUC4437
#EUC4437
Top Related