Citrix XenServer 5.5 Troubleshooting

Citrix® XenServer

Webcast 26. März 2010

Thomas KrampeCitrix Technology Professional

Advanced Troubleshooting

204/19/10 (c) 2009 Thomas Krampe

Agenda

Architecture

Troubleshooting

Birth and Death of a VM

Object modell & CLI

Unattended Setup & Configuration

Question & Answers


Architecture

Architecture

XenServer Architecture

Xen Domain0

Guest OS Internals

XAPI

Storage Manager

CLI

GUI (XenCenter)


XenServer Architecture

Virtualized HardwareVirtualized Hardware

Hardware

VT/AMD-V

XAPIXAPIDrivers

Storage

CLI

SMSM

NFS iSCSI FC


Xen Domain0

Dom0 kernel and initrd passed on start up

Dom0 has direct access to HW

Bootloader

/boot/grub/grub.conf

/boot/extlinux.conf

[Ctrl-A] 3 times will trap into the Xen debug console


Guest OS internals

PV Guest

Modified Kernel (“Xenified”)

Kernel name will have “xs” somewhere in the filename

Guest agent reports VM stats

Use pygrub bootloader

HVM Guest (Windows)

Unmodified Kernel

Need PV drivers

Event Channel, Network and Block

Guest agent uses WMI


XAPI /opt/xensource/bin/xapi

Service, autostarted on boot

Implements the Xen API spec

Talks to Storage Manager (sm) for SR control operations

Stores configuration in sqlite DB

/var/xapi/state.db

sqlite3 /var/xapi/state.db

DB is replicated on all hosts in a pool

Sets up SR defaults on firstboot

Failing firstboot scripts might cause issues

/var/xapi/generate-* and /var/xapi/firstboot-*

Defaults from /etc/xensource-inventory


Storage Manager

Manages SR control operations to various backends

/opt/xensource/sm/sm.py

Plugin model

One plugin for each SR type

/opt/xensource/sm/<Type>SR.py

Error codes listed in

/opt/xensource/sm/XE_SR_ERRORCODES.xml

xe sm-list lists all supported drivers


CLI

Local or Remote

Talks to XAPI

Bash auto completion

xe.exe (for Windows)

xe help or xe help --all

1004/19/10 (c) 2009 Thomas Krampe

GUI

Uses Microsoft .NET framework

Uses https/xml-rpc to communicate with XAPI

VNC Client for guest frame buffer

VNC traffic is tunneled over https

RDP client for Windows guests

1104/19/10 (c) 2009 Thomas Krampe

Troubleshooting

TroubleshootingServer Install

Useful Linux Commands

Server Diagnostics & Inventory

Saving System State

XS Server Log Files

XenSource Agent

XenCenter Logging

Summery

1204/19/10 (c) 2009 Thomas Krampe

Server Install After Product CD Boots

Alt-F2 displays Linux login prompt

Login: root (no password required)

Alt-F3 displays install event log

Alt-F1 displays original install menu

Support Tarballsupport.sh (creates /tmp/support.tar.bz2)

Support tarball is created in /tmp on install error

Support tarball is saved in /root after successful install

1304/19/10 (c) 2009 Thomas Krampe


Storage

List Disks detected

fdisk –l

Network

Show NIC configuration

ifconfig eth0 [eth1 ..]

ifup eth0

TCP/IP Tools ping, route -n

dhclient eth0

1404/19/10 (c) 2009 Thomas Krampe


Process Management

ps –Af

top

xentop

1504/19/10 (c) 2009 Thomas Krampe

Server Diagnostics and Inventory

Server Diagnostics

xen-bugtool

Run on the XS Console

Records all log files

Writes compressed file for submission to L2 support

/var/opt/xen/bug-report/bug-report-xxx.tar.bz2

1604/19/10 (c) 2009 Thomas Krampe


Server Diagnostics

Automatic Upload for Support

xe host-bugreport-upload

Can be run from admin system

Runs xen-bugtool and uploads to XS Support

Note unique ID of report for support

1704/19/10 (c) 2009 Thomas Krampe


Server Inventory

system-info.sh | less

Interactive script that captures all non-log state

Hardware devices & Device drivers

Storage repository and VM configurations

1804/19/10 (c) 2009 Thomas Krampe

Saving System State

xen-bugtool archives state and configuration files

Network config

Install options

Install logs

Storage Repository metadata

VM configs

Xenstore

1904/19/10 (c) 2009 Thomas Krampe

XS Server Log Files

XenSource XAPI Agent

/var/log/messages

Contains other system logging

/var/log/xensource.log

Xen/Domain 0 Xen Boot Messages

xe host-dmesg

Domain 0 Boot Messages

dmesg

2004/19/10 (c) 2009 Thomas Krampe

Debug Logging for XAPI Agent

Enable debug logging for all subsystems

xe log-set-output level=debug output=file:/tmp/debug-all

Debug for only a certain subsystem

xe log-set-output level=debug key=storage output=file:/tmp/storage-

debug

List Available Subsystems

xe log-get-keys

Restore normal logging xe log-set-output level=debug output=nil

xe log-set-output level=debug key=storage output=nil

2104/19/10 (c) 2009 Thomas Krampe

XenSource Agent

XS Agent runs on top Xen Hypervisor API and XenStore

Can be restarted without harming VMs or other members

of the pool

service xapi restart

If the UI seems to get stuck or goes into a weird state, restart

the agent and reconnect the UI

Another simular command is xe-toolstack-restart

2204/19/10 (c) 2009 Thomas Krampe

XenCenter Logging

Location

{user home}\Application Data\XenSource\XenCenter\logs\

XenCenter.log

Rolling

Log files will get rolled

XenCenter.log.#

Increase Log Level C:\Program Files\XenSource\XenCenter directory

<level value=“DEBUG” />

2304/19/10 (c) 2009 Thomas Krampe

Summary

xen-bugtoolgenerates XE Server diagnostic file for submission to Technical Support Representative.

xe host-bugreport-upload

runs xen-bugtool and uploads the result to XenSource support.

xe host-bugtool

can be used to save Domain 0 state, to assist during emergency recovery

Use Linux commands (network, storage, modules) to troubleshoot server install.

2404/19/10 (c) 2009 Thomas Krampe



VM Installation Internals

Behind the Scenes

Debugging VMs

2504/19/10 (c) 2009 Thomas Krampe

VM Installation Internals

Debian Sarge/Etch

Root file system is extracted into the VDI

Debian Xen kernel

RHEL (3.x,4.x,5) / SLES (9.x, 10.x)

Uses regular repositories

RHEL / SLES Xen kernel

Windows (all) Installs from CD/ISO bits

Unmodified Windows kernel

2604/19/10 (c) 2009 Thomas Krampe

Behind the Scenes

Paravirtualized VMs use pygrub

Simple python based bootloader for Xen

Finds kernel and initrd from VM’s boot device

CPU then jumps to this kernel to start this VM

HVM VMs use hvmloader

hvmloader creates a virtual BIOS and jumps to it. This is like a

BIOS in a physical machine

qemu-dm does device emulation, provides disks, NICs, serial

ports, USB devices etc to this VM

Follows standard boot process to boot from disk

2704/19/10 (c) 2009 Thomas Krampe

Debugging Windows VMs

Have client Windows machine ready, install WinDbg

Add /DEBUGPORT=com1 to boot.ini in VM, shut down VM

xe vm-param-add uuid=<VM UUID> param-name=other-config:hvm_serial=tcp:<client IP>:7001

sockpipe my_pipe 7001 on client machine

windbg -k com:pipe,port=\\.\pipe\my_pipe,resets=0

Start VM

Refer to WinDbg manual for commands

http://www.microsoft.com/whdc/DevTools/Debugging/default.mspx

http://www.microsoft.com/whdc/DevTools/Debugging/default.mspx

2804/19/10 (c) 2009 Thomas Krampe

Object Model and CLI

Object Model and CLI

Object Model

CLI Command Format

Maps

Constarining Information

Interesting Commands

Looking inside a log file

2904/19/10 (c) 2009 Thomas Krampe

Object Model

3004/19/10 (c) 2009 Thomas Krampe

CLI Command Format

Object commands

xe <class>-<TAB>

Object Param commands xe <class>-param-<TAB>

Some fields are marked <expensive

Data Types Single Valued/Multi-Valued

Read Only/Read-Write

Set/Map

3104/19/10 (c) 2009 Thomas Krampe

Maps

Setting Map values[root@localhost] xe vm-param-set uuid=VM uuid other-config:foo=baa

Getting Map values

[root@localhost] xe vm-param-get uuid=VM-uuid param-name=other-config mac_seed: 6fjafhahffhadfbadfad

[root@localhost] xe vm-param-get uuid=VM-uuid param-name=other-config param-key=mac_seed 6fjafhahffhadfbadfad

3204/19/10 (c) 2009 Thomas Krampe

Constraining Information

Filtering

vm-list HVM-boot-policy="BIOS order" power-state=halted

Parameterization xe vm-list params=name-label

Minimal Output xe vm-list --minimal

Parameterization and filtering can be combined

3304/19/10 (c) 2009 Thomas Krampe

Interesting Commands

Wait for a VM to run

xe event-wait class=vm name-label=myvm power-state=running

List of CPUs on a host xe host-cpu-list

CPU Flags must match across all hosts in pool

List of subsystems xe log-get-keys

Can be used to isolate subsystem logs

3404/19/10 (c) 2009 Thomas Krampe


Each Entry looks like

[timestamp][hostname][level] [ref-1] [(num-1) unix_rpc] text-1 text2 text3….

[timestamp] Timestamp of log entry

[hostname] Host name (in a pool)

[level] Debug level

[ref-1] ID of the task on whose behalf this is created

num-1 Thread ID, local to the current process

unix_rpc Unix Domain Socket (as opposed to inet_rpc=TCP)

text-1 Subsystem

text2/text3 more information on action performed

3504/19/10 (c) 2009 Thomas Krampe


xensource.log[20100209 16:58:28.688|debug|localhost|3078703|SR scanner D:dfd9ef0b708d|helpers] login done[20100209 16:58:28.921|debug|localhost|3078706 unix-RPC|SR.scan R:1ac783270499|taskhelper] task destroyed[root@localhost log]#

Xenstored-access.log[20100209 16:49:05.381] D35 write data/meminfo_free 551040[20100209 16:49:05.381] D35 write data/meminfo_total 1035760[20100209 16:49:05.381] D35 write data/updated 1

messagesFeb 9 17:04:58 fragmgrid2001 xapi: [ info|localhost|3078916|SR scanner D:dfd9ef0b708d|dispatcher] spawning a new thread to handle the current taskFeb 9 17:04:58 fragmgrid2001 xapi: [ info|localhost|3078917 unix-RPC|session.slave_login D:58b671575b1e|xapi] Session.create trackid=ea252f3e59cc6f0d2d3de1345967d54b pool=true uname= is_local_superuser=true auth_user_sid=Feb 9 17:04:58 fragmgrid2001 xapi: [ info|localhost|3078919 unix-RPC|dispatch:SR.scan D:64e76e160ebc|taskhelper] task SR.scan R:98637b78c34f (uuid:6e8469dc-724d-0260-a584-44d36bca2953) created (trackid=ea252f3e59cc6f0d2d3de1345967d54b) by task D:dfd9ef0b708d

3604/19/10 (c) 2009 Thomas Krampe

Unattend Installation

Possible over TFTP - PXE

use a single xml configuration file

http://wiki.xenmaster.de/doku.php/xenserver/xspxeunattend

3704/19/10 (c) 2009 Thomas Krampe

Questions & Answers

Citrix XenServer 5.5 Troubleshooting

Technology

Transcript of Citrix XenServer 5.5 Troubleshooting