Project MedEvac 1.0
-
Upload
abdul-kader-s -
Category
Documents
-
view
116 -
download
0
Transcript of Project MedEvac 1.0
![Page 1: Project MedEvac 1.0](https://reader036.fdocuments.us/reader036/viewer/2022062512/553da5ea550346b9308b476f/html5/thumbnails/1.jpg)
CITRIX MEDEVAC
GLOBAL ESCALATION TEAM
![Page 2: Project MedEvac 1.0](https://reader036.fdocuments.us/reader036/viewer/2022062512/553da5ea550346b9308b476f/html5/thumbnails/2.jpg)
2 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
Introducing Citrix Medevac 1.0
• What is Citrix Medevac 1.0?
• How does it work?
• Architecture
• How can Citrix Medevac help me?
• Case Study
• When can I use it?
• Limitations
![Page 3: Project MedEvac 1.0](https://reader036.fdocuments.us/reader036/viewer/2022062512/553da5ea550346b9308b476f/html5/thumbnails/3.jpg)
3 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
What is Citrix Medevac 1.0?
• Citrix Medevac 1.0 is a troubleshooting Utility developed by the Citrix Escalation Team to aid in troubleshooting and resolving complex and time consuming issues that are XML, Terminal Services, IMA and RPC related
• It is a standalone Utility that does not require a Windows Service or Server Side components to be installed for its use
• It can be run from any machine on the network
![Page 4: Project MedEvac 1.0](https://reader036.fdocuments.us/reader036/viewer/2022062512/553da5ea550346b9308b476f/html5/thumbnails/4.jpg)
4 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
What is the Health Check Agent?
• The Citrix Health Check Agent runs as a service on each Citrix Server
• This Agent has the ability to perform actions based on test results
• The tests monitor for the following: XML ticketing issues IMA Service application resolution issues Terminal Services health
• This tool is extensible by administrator to be configurable for monitoring other custom events.
![Page 5: Project MedEvac 1.0](https://reader036.fdocuments.us/reader036/viewer/2022062512/553da5ea550346b9308b476f/html5/thumbnails/5.jpg)
5 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
ARCHITECTURE
How does it work?
A Windows 32-bit Utility used to simulate the ICA client browsing requests to test XML,Terminal Services,IMA & RPC functionality
Uses method of Isolation based on Citrix Presentation Server Farm architecture to isolate Farm functionality
![Page 6: Project MedEvac 1.0](https://reader036.fdocuments.us/reader036/viewer/2022062512/553da5ea550346b9308b476f/html5/thumbnails/6.jpg)
6 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
“Medevac”
ARCHITECTURE
XML BrokerXML Broker
IMA Service
TermSrv
XML Service
Server Farm
Request the Least Loaded Server
DirectLL ServerQueries
RPC
Zone Data Collector
LHCDS
IMA
![Page 7: Project MedEvac 1.0](https://reader036.fdocuments.us/reader036/viewer/2022062512/553da5ea550346b9308b476f/html5/thumbnails/7.jpg)
7 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
ARCHITECTURE…
XML TICKETING TEST:
By simulating a client connection request we are able to test XML ticketing using Medevac as if it were an actual ICA client.
RPC TEST:
Based on input provided by the user, RPC tests can be verified by making RPC calls to the specified Server testing its functionality and returning values based on the result.
TERMINAL SERVICES TEST:
By querying information from Terminal Services remotely using Medevac, we are able to test the responsiveness of Terminal Services to ensure that it is functioning properly.
![Page 8: Project MedEvac 1.0](https://reader036.fdocuments.us/reader036/viewer/2022062512/553da5ea550346b9308b476f/html5/thumbnails/8.jpg)
8 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
IMA (Local DataCollector)
XML Service
IMA (Remote DataCollector)
Web InterfaceWeb Server
XML Broker
Or
XML Least Loaded Server Request
XML – Launching an Application.
![Page 9: Project MedEvac 1.0](https://reader036.fdocuments.us/reader036/viewer/2022062512/553da5ea550346b9308b476f/html5/thumbnails/9.jpg)
9 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
The Least Loaded Server
Termsrv
wsxica.dll
CCTicket.dll
CCTicket.dll
XML Service
CCTicket.dll
XML Ticket
Request
IMA Service
CCTicket.dll
or
XML IMA Service
XML Broker
Web InterfaceServer
XML Ticket
Request
credentialsticket
credentialsticket
credentialsticket
XML – Launching an Application.
![Page 10: Project MedEvac 1.0](https://reader036.fdocuments.us/reader036/viewer/2022062512/553da5ea550346b9308b476f/html5/thumbnails/10.jpg)
10 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
ARCHITECTURE
The Least Loaded Server Tests:
Performs XML, RPC & Terminal Services tests• It uses the XML Broker’s XML service to request the least loaded
farm server for a specified published application• It uses that least loaded server IP address to make health checks
directly against the least loaded server out of the servers that have that published application available
• This is a more efficient method for checking the availability of critical environment applications rather than a “shotgun” approach
Example: If your environment has a couple of hundred servers it becomes increasingly difficult to isolate a problem server
• More importantly, this utility can proactively find a problem least loaded server before users do, reducing user outages and help desk calls
![Page 11: Project MedEvac 1.0](https://reader036.fdocuments.us/reader036/viewer/2022062512/553da5ea550346b9308b476f/html5/thumbnails/11.jpg)
11 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
ARCHITECTURE
What is actually verified on the listed server farm:
The XML service
The IMA service
The Termsrv Service
The RPC service
![Page 12: Project MedEvac 1.0](https://reader036.fdocuments.us/reader036/viewer/2022062512/553da5ea550346b9308b476f/html5/thumbnails/12.jpg)
12 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
Enhancements for Medevac 2.0
• Save test configuration to a configuration file.
• Save and import a server lists.
• Command line parameter implementation.
• Application name browsing.
• E-Mail alerts (SMTP).
• Full ICA Connection test.
• Multi-threaded.
• Allow administrators to restart certain services. (Maybe)
![Page 13: Project MedEvac 1.0](https://reader036.fdocuments.us/reader036/viewer/2022062512/553da5ea550346b9308b476f/html5/thumbnails/13.jpg)
13 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
How can Citrix MedEvac help me?
• Imagine a situation where you have an outage in your entire Citrix Farm during peak production time. None of your users can launch their published applications
• Everything else seems to check out fine
• Where would you begin?
• Any time wasted has severe business impact
Your Solution:
CITRIX MEDEVAC 1.0
![Page 14: Project MedEvac 1.0](https://reader036.fdocuments.us/reader036/viewer/2022062512/553da5ea550346b9308b476f/html5/thumbnails/14.jpg)
14 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
Typical “Black Hole” Problem
Symptoms
• Server cannot accept user connections
• Server still responding to Data Collector’s heart beats
• Data Collector views server as “least loaded”
• Data Collector directs incoming user connections to this “least loaded” server
• Denial of Service occurs to users connecting to these problematic servers.
![Page 15: Project MedEvac 1.0](https://reader036.fdocuments.us/reader036/viewer/2022062512/553da5ea550346b9308b476f/html5/thumbnails/15.jpg)
15 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
“Black Hole” Problem(s)
DataCollector
Problematic & “Least Loaded”Presentation Server
IMA TermSrv
XML
“Heart Beat”
IMA
Healthy Presentation Server
IMA TermSrvXML
Server Load Levels
“Heart Beat”
XML
User Connections
![Page 16: Project MedEvac 1.0](https://reader036.fdocuments.us/reader036/viewer/2022062512/553da5ea550346b9308b476f/html5/thumbnails/16.jpg)
16 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
Medevac 1.5 Demo
Demo
![Page 17: Project MedEvac 1.0](https://reader036.fdocuments.us/reader036/viewer/2022062512/553da5ea550346b9308b476f/html5/thumbnails/17.jpg)
17 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
CASE STUDY
USERS UNABLE TO CONNECT TO THEIR PUBLISHED RESOURCES
![Page 18: Project MedEvac 1.0](https://reader036.fdocuments.us/reader036/viewer/2022062512/553da5ea550346b9308b476f/html5/thumbnails/18.jpg)
18 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
CASE STUDY
PROBLEM DEFINITION:
• Users are unable to connect to published resources in the Farm
ENVIRONMENT:
• CPS Version: Citrix Presentation Server 4.0
• CPS Hotfixes: PSE400W2K3R01
• OS Version: Windows Server 2003 Enterprise/wSP1
• Data Store: Microsoft SQL 2000/wSP3a
• CPS Servers: 50
• Zones: 1
• Connection Method: Web Interface 4.0
• ICA client: 9.x
![Page 19: Project MedEvac 1.0](https://reader036.fdocuments.us/reader036/viewer/2022062512/553da5ea550346b9308b476f/html5/thumbnails/19.jpg)
19 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
CASE STUDY
TROUBLESHOOTING METHODOLOGY:
THE APPROACH:
Description of issue:
User attempt to launch a published application and receives an error - "Server - unavailable" or "Cannot connect to the Citrix MetaFrame Server. I/O error"
Some things we know:• Issue is affecting all users• Issue has never occurred before in this Environment• Nothing has changed in the Environment• Users are connecting through Web Interface 4.0• Web Interface does enumerate the published resources• Connecting directly to a Server works
Now Lets take a closer look at an ICA connection….
![Page 20: Project MedEvac 1.0](https://reader036.fdocuments.us/reader036/viewer/2022062512/553da5ea550346b9308b476f/html5/thumbnails/20.jpg)
20 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
IMA (Local DataCollector)
XML Service
Problems Launching An Application (Step 1)
Problems Launching An Application (Step 1)
IMA (Remote DataCollector)
Web InterfaceWeb Server
XML Broker
Or
XML Least Loaded Server Request
Local Host Cache
Dynamic Store
![Page 21: Project MedEvac 1.0](https://reader036.fdocuments.us/reader036/viewer/2022062512/553da5ea550346b9308b476f/html5/thumbnails/21.jpg)
21 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
The Least Loaded Server
Termsrv
wsxica.dll
XML IMA Service
CCTicket.dll
credentialsticketcredentialsticket
credentialsticket
CCTicket.dll
Web InterfaceWeb Server
XML Ticket Request
Problems Launching AnApplication (Step 2)
Problems Launching AnApplication (Step 2)
XML Broker
End User Client
Connection / Logon To The Citrix Server’s
Published Application
![Page 22: Project MedEvac 1.0](https://reader036.fdocuments.us/reader036/viewer/2022062512/553da5ea550346b9308b476f/html5/thumbnails/22.jpg)
22 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
CASE STUDY
TRADITIONAL APPROACH:
• QFARM
• Zone Isolation
• Isolation based on application hosting (which servers host certain apps)
• Direct server connections
• Manual testing using custom ICA connections
• Forced ZDC elections…
• Etc.
![Page 23: Project MedEvac 1.0](https://reader036.fdocuments.us/reader036/viewer/2022062512/553da5ea550346b9308b476f/html5/thumbnails/23.jpg)
23 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
CASE STUDY
INSTEAD…
Let’s use MedEvac to assist with troubleshooting
• Using MedEvac on a workstation from the same Domain (with Admin privileges) we can begin
![Page 24: Project MedEvac 1.0](https://reader036.fdocuments.us/reader036/viewer/2022062512/553da5ea550346b9308b476f/html5/thumbnails/24.jpg)
24 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
CASE STUDY
• Due to the architecture of Load Balancing, we can query the XML Broker to determine the Least Loaded Server....and run some basic tests against the Least Loaded Server that is returned….
XML Broker
Least Loaded Server
Tests run against the
LLS
![Page 25: Project MedEvac 1.0](https://reader036.fdocuments.us/reader036/viewer/2022062512/553da5ea550346b9308b476f/html5/thumbnails/25.jpg)
25 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
CASE STUDY
Now we can run further health checks against the Least Loaded Server that was returned…
• So now let us run a XML Ticketing Test, as we have already tested Terminal Services and RPC
![Page 26: Project MedEvac 1.0](https://reader036.fdocuments.us/reader036/viewer/2022062512/553da5ea550346b9308b476f/html5/thumbnails/26.jpg)
26 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
CASE STUDY
• XML TICKETING TEST
Least Loaded Server specified
Test results
XML Ticketing
Test specified
![Page 27: Project MedEvac 1.0](https://reader036.fdocuments.us/reader036/viewer/2022062512/553da5ea550346b9308b476f/html5/thumbnails/27.jpg)
27 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
CASE STUDY
• MedEvac identified the “Least Loaded Server” so we know which server the ZDC is routing connections to…
• So how do we verify the IMA service is functioning properly on the “Least Loaded Server?”
• By running the LLS test and specifying the Least Loaded Server in MedEvac, we are making the LLS use it’s IMA service to communicate to the Zone Data Collector to return itself as the Least Loaded Server…
• If this test returns success, we know that the IMA service on the Least Loaded Server is working properly…
![Page 28: Project MedEvac 1.0](https://reader036.fdocuments.us/reader036/viewer/2022062512/553da5ea550346b9308b476f/html5/thumbnails/28.jpg)
28 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
CASE STUDY
LLS Test against the Least Loaded Server (IMA TEST)
FAILURE
![Page 29: Project MedEvac 1.0](https://reader036.fdocuments.us/reader036/viewer/2022062512/553da5ea550346b9308b476f/html5/thumbnails/29.jpg)
29 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
CASE STUDY
• In this scenario, the Least Loaded Server test failed when run against itself.
• So now we can then focus all troubleshooting towards that server’s IMA service…
• Investigation indicated that the IMA service on that server was “hung” since it could not be stopped or re-started
• Although IMA still responded to the Data Collector’s “heartbeat” messages, it could not accept any connections due to its “hung” state
![Page 30: Project MedEvac 1.0](https://reader036.fdocuments.us/reader036/viewer/2022062512/553da5ea550346b9308b476f/html5/thumbnails/30.jpg)
30 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
CASE STUDY
RESOLUTION:
• Using PSKill from http://www.sysinternals.com we manually forced termination the IMA service
• Recreated the local host cache
• Restarted IMA - started successfully
• Users able to launch their published resources as normal now
![Page 31: Project MedEvac 1.0](https://reader036.fdocuments.us/reader036/viewer/2022062512/553da5ea550346b9308b476f/html5/thumbnails/31.jpg)
31 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
CASE STUDY
ADDITIONAL INFORMATION:
• Detailed information on Citrix Medevac and its usage can be found at the following locations:
http://support.citrix.com/article/CTX107935
![Page 32: Project MedEvac 1.0](https://reader036.fdocuments.us/reader036/viewer/2022062512/553da5ea550346b9308b476f/html5/thumbnails/32.jpg)
32 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
When can I use it?
• The Citrix Medevac Utility can be used anytime where isolating a problem Server is difficult (with respect to load balancing, XML, Terminal Services and RPC)
• To troubleshoot connectivity issues with published resources, such as the mysterious ‘black hole’ issue
• To run direct health checks against Citrix Presentation Servers
![Page 33: Project MedEvac 1.0](https://reader036.fdocuments.us/reader036/viewer/2022062512/553da5ea550346b9308b476f/html5/thumbnails/33.jpg)
33 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
KNOWN LIMITATIONS
• Please see the ‘Readme’ and requirements posted on the Citrix Online Support Knowledgebase
http://support.citrix.com/article/CTX107935
• Recommended to register to the Citrix Developers Network for access to Tools and Utilities
Disclaimer• These software applications are provided to you as is with no representations, warranties or conditions of
any kind. You may use and distribute it at your own risk. CITRIX DISCLAIMS ALL WARRANTIES WHATSOEVER, EXPRESS, IMPLIED, WRITTEN, ORAL OR STATUTORY, INCLUDING WITHOUT LIMITATION WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NONINFRINGEMENT. Without limiting the generality of the foregoing, you acknowledge and agree that (a) the software application may exhibit errors, design flaws or other problems, possibly resulting in loss of data or damage to property; (b) it may not be possible to make the software application fully functional; and (c) Citrix may, without notice or liability to you, cease to make available the current version and/or any future versions of the software application. In no event should the code be used to support of ultra-hazardous activities, including but not limited to life support or blasting activities. NEITHER CITRIX NOR ITS AFFILIATES OR AGENTS WILL BE LIABLE, UNDER BREACH OF CONTRACT OR ANY OTHER THEORY OF LIABILITY, FOR ANY DAMAGES WHATSOEVER ARISING FROM USE OF THE SOFTWARE APPLICATION, INCLUDING WITHOUT LIMITATION DIRECT, SPECIAL, INCIDENTAL, PUNITIVE, CONSEQUENTIAL OR OTHER DAMAGES, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. You agree to indemnify and defend Citrix against any and all claims arising from your use, modification or distribution of the code.
![Page 34: Project MedEvac 1.0](https://reader036.fdocuments.us/reader036/viewer/2022062512/553da5ea550346b9308b476f/html5/thumbnails/34.jpg)
34 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.
Questions?
![Page 35: Project MedEvac 1.0](https://reader036.fdocuments.us/reader036/viewer/2022062512/553da5ea550346b9308b476f/html5/thumbnails/35.jpg)
35 Citrix Company Confidential © 2003 Citrix Systems, Inc.—All rights reserved.