VMworld 2015: Horizon View Troubleshooting - Looking Under the Hood

Horizon View Troubleshooting –Looking Under the Hood

Jack McMichael, VMware, IncMatt Coppinger, VMware, Inc

EUC4437

#EUC4437

• This presentation may contain product features that are currently under development.

• This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.

• Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

• Technical feasibility and market demand will affect final delivery.

• Pricing and packaging for any new technologies or features discussed or presented have not been determined.

Disclaimer

CONFIDENTIAL 2

Identity Manager

ITUser

Horizon

Desktop

AirWatch

Mobile

Content

Collaboration

One Cloud

Workspace Suite

VMware Workspace Suite: Enabling Business Mobility

CONFIDENTIAL 3

CONFIDENTIAL 4

AirWatch Mobile:• A Leader for 5 Consecutive Years

• Placed Highest on Ability to

Execute Axis 3 Consecutive Years

Magic Quadrant

Figure. Magic Quadrant for Enterprise Mobility Management Suites

Source: Gartner, Inc., Magic Quadrant for Enterprise Mobility Management Suites, Terrence Cosgrove, et al, June 8 2015. &

Gartner, Inc., 2015 Critical Capabilities for Enterprise Mobility Management Suites, Terrence Cosgrove, et al, June 9 2015.

This graphic was published by Gartner, Inc. as part of a larger research document and should be evaluated in the context of the

entire document. The Gartner document is available upon request from AirWatch.

Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology

users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the

opinions of Gartner¹s research organization and should not be construed as statements of fact. Gartner disclaims all warranties,

expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

Horizon Desktop:VMware leaps past the competition

“VMware's position reflects the company's market

position and commitment to providing resources to

expand its EUC product portfolio and infrastructure.”

CONFIDENTIAL 5

Agenda

CONFIDENTIAL 6

1 Common Issues

2 What can go wrong?

3 Domain 1: Horizon Client Connectivity

4 Domain 2: Desktop Availability

5 Domain 3: Application Management

6 Domain 4: Broken Broker

7 Domain 5: Performance

8 Resources

Top VMware Global Support Services Tickets

• SSL and Domains

• Connectivity – Understand Horizon network requirements!

• Persona and UEM

• Parent VM issues – Do NOT P2V! Clean VM please…

• PCoIP sizing

• Logs

CONFIDENTIAL 7

Working with SSL Certificates

Internal Certificate

Authorities are the easiest to setup, but not

the most trusted

Intermediate Certificate Authorities

require chaining and more work

to setup

Top level Certificate

Authorities are the most

expensive

CONFIDENTIAL 8

• https://pubs.vmware.com/horizon-view-60/topic/com.vmware.ICbase/PDF/horizon-view-60-scenarios-ssl-certificates.pdf

Horizon 6 SSL Guide

CONFIDENTIAL

Generating an SSL Certificate Signing Request

CONFIDENTIAL

Read the Guide!

Create a config file

Generate cert signing request

(CSR)

Validate CSR and

Private Key

Send CSR to CA

Receive Signed Cert

from CA

Import Cert

Configure Horizon

Server to use Cert

Test!

CONFIDENTIAL 11

Troubleshooting Keys

Check Horizon Administrator Dashboard

Understand the client connection process (where most problems lie)

Set the Logging Level on Broker

Check Connection Broker Logs

Check Horizon Agent Logs

Check Horizon Desktop PCoIP Logs

Use kb.vmware.com!

CONFIDENTIAL 12

You’ve Deployed Horizon. What Could Go Wrong?

You’ve got a problem:

• I can’t even connect to Horizon

• I get disconnected randomly!

• Why is the display so bad?

• Why is my desktop not available?

• I’m seeing an error in Horizon, what does it mean?

• vCenter is reporting an error

• My desktop is slow…

CONFIDENTIAL 13

Identifying the Problem Domain

CONFIDENTIAL 14

Horizon Client

Network

View Connection Server

View Composer

vCenter Server

Compute

Storage

Domain 1: Horizon Client Connectivity Issues

Common challenges

• Horizon Client can’t connect

• Logon failures

• Black screens

• Poor quality display

• Randomly disconnected session

CONFIDENTIAL 15


Where to look

• Connection Broker logs

• Windows 2008 - <DriveLetter>:ProgramData\Application Data\VMware\VDM\logs

• Event Database

What to look for

• (Client connects) [SimpleAJPService] (ajp:broker:Request9) Request from /192.168.2.1: POST /broker/xml

• (Broker authentication) [WinAuthFilter] (SESSION:7072-***-a79c mattc) Attempting to authenticate user 'mattc' in domain 'FUTUREOFFICE’

• (User has authenticated to Broker) [AuthorizationFilter] (SESSION:7072-***-a79c) User FUTUREOFFICE\mattc has successfully authenticated to VDM

• (Audit Entry) [Audit] (SESSION:7072-***-a79c) BROKER_LOGON:USER:FUTUREOFFICE\mattc;USERSID:S-1-5-21-326850759-2560684469-1780228732-1113;USERDN:CN=S-1-5-21-326850759-2560684469-1780228732-1113,CN=ForeignSecurityPrincipals,DC=vdi,DC=vmware,DC=int;

• Event Database: BROKER_USERLOGGEDIN

CONFIDENTIAL 16


CONFIDENTIAL 17

Track sessions using Tail/Grep utilities

CONFIDENTIAL 18


Black Screen of Death

• PCoIP port blocked (TCP and UDP 4172) or SVGA Driver issue

• pcoip_server/client logs - C:\Users\All Users\VMware\VDM\logs

• Error attaching to SVGADevTap, error 4000: EscapeFailed

• MGMT_SCHAN :scnet_client_open: tera_sock_connectreturned error 10060 - Connection timed out!

• Incorrect PCoIP External URL configured for Security/Connection Servers

CONFIDENTIAL 19


Poor quality display

• Bandwidth, latency, or QoS

• Pcoip_server logs report

• VGMAC :Stat frms: Loss=0.45%/0.21% (R/T)

• MGMT_PCOIP_DATA :BW: Decrease (loss) old = 234.9982 new = 176.8438

Randomly disconnected session?

• 15 min after established - wssm process hasn't started on desktop

• Horizon Agent logs C:\ProgramData\VMware\VDM\logs)

• PENDING_EXPIRED

• Sometimes caused by daisy-chaining the GINA

CONFIDENTIAL 20

Domain 2: Desktop Availability

• Common Issues

– No Desktop Available

– Pool provisioning issues –customization

– Agent not communicatingwith broker

– Stuck at desktop loginscreen (SSO)

• Where to look

– Connection Broker/Agent logs

– Event Database

• What to look for

• Broker returns list of desktops available to client

– [DesktopsHandler] (SESSION:7072-***-a79c) For user [S-1-5-21-326850759-2560684469-1780228732-1113] and pool [cn=gold-np,ou=server groups,dc=vdi,dc=vmware,dc=int] DesktopTracker returned 2 guest DNs

CONFIDENTIAL 21


Successful Connection Walkthrough

• Client requests desktop

• Event Database: BROKER_DESKTOP_REQUEST

• Broker allocates session to user

• [FarmImp] (SESSION:7072-***-a79c) cn=3f974017-409f-4912-83bc-2ee794f22fab,ou=servers,dc=vdi,dc=vmware,dc=int, total session count: 0

• [FarmImp] (SESSION:7072-***-a79c) allocateNewSession - identified server for application CN=GOLD-NP,OU=Applications,DC=vdi,DC=vmware,DC=int

• Event Database: BROKER_MACHINE_ALLOCATED

• Broker attempts SSO

• [FarmImp] (SESSION:7072-***-a79c) Using domain for SSO: FUTUREOFFICE**

• User won’t be logged on to the VM without this!

CONFIDENTIAL 22


Successful Connection Walkthrough

• Broker starts session on VM

• [DesktopSessionImp] (SESSION:7072-***-a79c) startSession – sending StartSession message

• Agent responds…

• "DesktopManager got a StartSession message”

• Client Info should be in Agent Log along with PCoIP launch

• Event Database: AGENT_PENDING

• [DesktopSessionImp] (SESSION:7072-***-a79c) startSession completed:

• DesktopTracker] User FUTUREOFFICE\mattc connected to machine gold-np-2 for desktop gold-np

• Client connects to VM (Agent)

• “PCoIPCnx::OnConnectionComplete Begin (PCOIP)”

• “WTS_SESSION_LOGON”

• Event Database: AGENT_CONNECTED

CONFIDENTIAL 23


Check the desktop’s status with ADSI

• Internal Horizon Database

• Installed on every broker

CONFIDENTIAL

24


Common ADSI Attributes:

• pae-DisplayName

• VM name as displayed in Horizon Admin

• pae-SVIVMSnapshot

• Indicates the current Snapshot that is in use

• pae-VmPath

• Indicates the full Path to the VM in vCenter

• pae-VmState

• Indicates the current state of the Desktop – some states are a combination of this value and other values

CONFIDENTIAL 25


• Find VMs with a Snapshot:

– (&(objectClass=pae-VM)(pae-SVIVmSnapshot=/Baseline/Snapshot1/Snapshot2))

• Find VMs with a Name:

– (&(objectClass=pae-VM)(pae-DisplayName=Desktop-234))

CONFIDENTIAL 26


Events Database

• Query for broker events or use Horizon Admin UI

• Horizon Event Notifierhttps://labs.vmware.com/flings/horizon-view-event-notifier

CONFIDENTIAL 27


Desktops not available due to provisioning error?

• Check View Administrator for Pool status, check datastore capacity

• Check Event Database -BROKER_PROVISIONING_ERROR_*

• Check View Composer has network access to ESX hosts

Desktop not available due to customization?

• Check Desktop status – AGENT UNAVAILABLE

• Check View Dashboard

• Desktop Status > Preparing Desktops OR Problem Desktops

• Check Desktop connectivity to DNS/AD/Connection Server

CONFIDENTIAL 28


Desktop not available due to VM reset/crash?

• Check Desktop status –ALREADY USED

• Typical on refresh-on-logoff or delete-on-use desktops

• Broker never received an explicit logout message from the agent

• Missing AGENT_ENDED event in DB for VM

View Composer Issues associated with incorrect domain credentials

• C:\ProgramData\VMware\View Composer\Logs\

• FATAL CSvmGaService -[svmGaService.cpp, 116] Domain join failed Error 5 (0x5): Access is denied.

CONFIDENTIAL 29

Domain 3: Application Management

• RDS and Group Policy Objects (computer and user policies)

– RD Licensing

– RD Connection and Client Connection

– Application Compatibility

– Device and Resource Redirection

– Session Environment and Time Limits

– Profiles and Temporary Folders

• More info: http://technet.microsoft.com/en-us/library/ee791756(v=ws.10).aspx

• FYI – we will be shipping custom ADM templates that are relevant to RDS integration with Horizon View

• We can and will honor existing RDS GPOs that are in use

CONFIDENTIAL 30

http://technet.microsoft.com/en-us/library/ee791756(v=ws.10).aspx

Domain 3: Application Management

RDS Applications

• Depending on Farm, can be one or multiple host sessions.

• Session limits per host (150 default)

• Licensing Diagnostics (user or device)

Best Practices

• No CPU Overcommitment

• Enable Hyperthreading

• Enable Transparent Page Sharing

• Enable Fixed Memory Allocation

• Disable Host Swapping

CONFIDENTIAL 31

Domain 4: Broken Broker

Common Issues

• Cannot connect to vCenter

• View Composer errors/issues

• JMS connectivity

• ADAM replication failure

Where to look

• View Administrator

• Event Database

• Windows Event Logs

• View Composer Logs

• Connection Server Logs

CONFIDENTIAL 32


Broker Sizing Guidelines

• Common issues resulting from undersized brokers:

• Memory Heap Issues (In 5.x versions)

• Threading issues

• Latency Issues

Recommendations:

• Always size your broker right from the beginning, avoid downtime or issues later.

• Minimum recommended specifications:

• 2 vCPU / 10 GB RAM for normal sizing (50-500 VMs)

• 4 vCPU / 16+ GB RAM for large sizing (500+ VMs)

CONFIDENTIAL 33


ADS Replication

• Check the Connection Broker Window Event Logs

– VMwareVDMDS log (Error: ADAM Replication)

• Check ADAM replication status on Connection Server:%systemroot%\adam\repadmin.exe /showrepl localhost:389 DC=vdi,DC=vmware,DC=int

CONFIDENTIAL 34


vCenter Server Connectivity

• Admin UI will show RED status

• Check Event Database

• VC_DOWN events

• Impacts provisioning and power operations ONLY

• Check Connectivity from Connection Server to vCenter Server

• Check credentials used to connect to vCenter Server

• Attempt to login in directly to vCenter using vSphere Client

CONFIDENTIAL 35


What to look for…

• View Composer

– VMs stuck in DELETING status

– VMs have been manually deleted – then pool/desktop deleted

• Causes Composer DB and VC DB to get out of sync

• Composer thinks VM already exists

• Orphaned VMs – KB-2015112 (kb.vmware.com)

• Desktop Composer Fault: 'Virtual Machine with Input Specification already exists‘

• JMS Connectivity

– Split site architecture / firewall causes “split brain”

– View Dashboard shows RED status

– Connection Server Logs

• tracker REJOIN messages – JMS connectivity

• tracker RESYNC messages – messages being delayed

CONFIDENTIAL 36

Cleaning Up Broken / Orphaned Pools

• BACKUP

• Disable Provisioning on Broker

• Stop View Composer

– Remove Composer Database Objects

– Remove ADLDS Servers, Server Groups, and Applications

– Remove AD Computer Entries

– Remove vCenter objects

• Unprotect replicas with sviconfig:

SviConfig -operation=RemoveSviClone -VmName=replica-<guid> -

AdminUser=administrator -AdminPassword=passowrd -

ServerUrl=https://localhost:18443/SviService/v2_0

Active Directory View Composer

AD LDS

CONFIDENTIAL 37

One Query To Rule Them All (Proceed with Extreme Caution!)

• Delete One VM

DELETE FROM dbo.SVI_VM_NAME WHERE NAME='replaceMe'

DELETE FROM dbo.SVI_COMPUTER_NAME WHERE NAME='replaceMe'

DELETE FROM dbo.SVI_SC_PDISK_INFO WHERE PARENT_ID=(SELECT ID FROM dbo.SVI_SIM_CLONE WHERE VM_NAME='replaceMe')

DELETE FROM dbo.SVI_SC_BASE_DISK_KEYS WHERE PARENT_ID=(SELECT ID FROM dbo.SVI_SIM_CLONE WHERE VM_NAME='replaceMe')

DELETE FROM dbo.SVI_TASK_STATE WHERE SIM_CLONE_ID=(SELECT ID FROM dbo.SVI_SIM_CLONE WHERE VM_NAME='replaceMe')

DELETE FROM dbo.SVI_REQUEST WHERE ID=(SELECT REQUEST_ID FROM dbo.SVI_TASK_STATE WHERE SIM_CLONE_ID=(SELECT ID FROM dbo.SVI_SIM_CLONE WHERE VM_NAME='replaceMe'))

DELETE FROM dbo.SVI_SIM_CLONE WHERE VM_NAME='replaceMe'

• Delete ALL VMs and Pools

DELETE FROM dbo.SVI_VM_NAME

DELETE FROM dbo.SVI_COMPUTER_NAME

DELETE FROM dbo.SVI_SC_PDISK_INFO

DELETE FROM dbo.SVI_SC_BASE_DISK_KEYS

DELETE FROM dbo.SVI_TASK_STATE

DELETE FROM dbo.SVI_REQUEST

DELETE FROM dbo.SVI_SIM_CLONE

DELETE FROM dbo.SVI_REPLICA

DELETE FROM dbo.SVI_DG_CUST_PROP

DELETE FROM dbo.SVI_DEPLOYMENT_GROUP

Note: Composer will auto clean replicas when no dependent SIM_CLONE objects are available. CONFIDENTIAL 38

Domain 4: Broker Broker

• Entitlements: Local vs Global

• Global always takes precedence

• Global Data Layer Replication

• Limits

• 2 Sites

• 4 Pods

• 20,000 Users

Cloud Pod Architecture

CONFIDENTIAL 39

Domain 5: Performance

• Common Issues

– Storage IO bottleneck

– Memory contention

– CPU contention

– Network issues

• Where to look

– vCenter Server

– ESXTOP

– 3rd Party Tools?

CONFIDENTIAL 40


What to look for

• CPU

– Cluster/Host utilization < 90%

– VM utilization - %USED (ESXTOP)

– VM %RDY Time (ESXTOP) < 10

• Memory

– Host utilization < 85%

– VM utilization

– Swapping / Ballooning SWCUR > 1 / MCTLSZ > 1 (ESXTOP)

• Storage

– Disk Read Latency < 25ms

– ESXTOP DAVG or KAVG < 25ms (ESXTOP)

CONFIDENTIAL 41


Optimize your Desktop Parent images!

CONFIDENTIAL 42


vRealize Operations Manager

• Aggregates metrics into workload, capacity and health scores

• Relies on dynamic thresholds

CONFIDENTIAL 43

vRealize Operations for Horizon

CONFIDENTIAL 44

Capturing ESX Performance Snapshots

• Use the following command to collect performance metrics for 8 hours on a host:

for i in `seq 8`;do esxtop -a -b -d 5 -n 720 > $i.<hostname>.csv;done

• The above command will create eight 100mb files consisting of 1 hour's worth of ESXTOP snapshots.

– -d = delay in seconds

– -n = iterations

– (-d 5 x -n 720 = 3600 seconds or 1 hour).

• Useful for replaying performance data over wide time periods for support to analyze!

CONFIDENTIAL 45

Summary

• Understand where the issue may lie

– Client? Agent? Server? Composer? vCenter? ESX?

• Know the problem domains

• Check the Horizon Dashboard and Event Database

• Identify the issue – know what a successful connections looks like

• Check the logs

• Use Performance Tools – ESXTOP, vCenter, vRealize Operations for Horizon

• Get Help

CONFIDENTIAL 46

Getting Help

• Read the manual! It’s searchable!

• Double check your configuration!

• Check kb.vmware.com for your issue

• http://communities.vmware.com

• Run Horizon “DCT Support” to extract the logs

– http://kb.vmware.com/kb/1017939

– Check the product documentation for using VDMADMIN command for creating various Data Collection Tool bundles

• Submit a Support Request

CONFIDENTIAL 47

http://kb.vmware.com/kb/1017939

QuestionsTwitter: @jackwmc4 / @mcopping

EUC VMworld 2015 Session Recommendations

CONFIDENTIAL 49

TechnicalApplication Lifecycle Management with

VMware Horizon EUC4561

Why Everybody Needs VMware User

Environment Manager (UEM) EUC5430

Beyond the Marketing:

Horizon 6 Technical Deep Dive EUC5052

Your Desktops Secured:

What Can NSX Do for You? EUC5062

Beyond the Marketing: VMware App

Volumes Technical Overview EUC5434

High Performance 30 Workloads on

Horizon 6 and NVIDIA GRID vGPUEUC5481

Access Point -EUC Common Gateway EUC6101

What's New with Horizon 6 with VOl and

Hosted Applications EUC6129

What's New in VMware Identity Manager EUC6105

Mobile Device Management or

Container: Do We Have to Choose?EUC5622

End-to-end Security with AirWatch. NSX

and Intelligent Networking EUC5762

Business/StrategyShow Me the Money !! Finding

Value in EUC – Why Identifying

Benefits Beyond Cost Matter Most

EUC5662

VMware’s End User Computing

(EUC) Strategy into 2015 and

Beyond

EUC5909

The Real Story of Customers

Delivering 3D Workstations with

VMware Horizon and NVIDIA GRID

EUC6621

Horizon Air: How to Provide an

Uncompromised User Experience at

a Cost Less than Physical

EUC6082

AirWatch 101: Enterprise Mobility

Management SimplifiedEUC5523

Peek Into the Future: Our Vision

for Business MobilityEUC5645-QT

Bring Your Own Content, We’ll Keep

it SecureEUC5764

Driving Business Mobility with

AirWatch and VMware IdentityEUC6098

The Future of End User Computing EUC6225-QT

VerticalHorizon at Point of Sales–We Did it

so Can You!EUC4919

Higher IT Pressure in Higher

Education: VMware End User

Computing Provides Instant Relief

EUC5664

Embracing Mobility in the

Federal SectorEUC5687

Mobilizing Healthcare: From Shared

to Personal Clinical Workspaces ...

And Beyond!

EUC5965

Financial Services Transformation in

Branch ComputingEUC6018

Horizon View Troubleshooting –Looking Under the Hood

Jack McMichael, VMware, IncMatt Coppinger, VMware, Inc

EUC4437

#EUC4437

VMworld 2015: Horizon View Troubleshooting - Looking Under the Hood

Technology

Transcript of VMworld 2015: Horizon View Troubleshooting - Looking Under the Hood