A New Paradigm in Linux Debug

September 2008

By

Art Lee

Vice President of Business Development

Viosoft Corporation

© 2008 Viosoft CorporationAll rights reserved.

The Current Paradigm

No one today would argue against the fact that Linux has taken the embedded RealTime Operating System (RTOS) space by storm. More and more applications thathistorically required either a commercially available RTOS, or one that was internallycreated and maintained, are being replaced with a Linux-based platform.

The reasons for this movement are varied from one company to the next, but some ofthe most common factors are:

1. The availability of source code to the Operating System

2. A wealth of device drivers and communication stacks

3. An increasingly available pool of software engineers that are Linux proficient

4. A perceived cost advantage achieved with removal of the OS royalty componentfrom the products’ bill of materials

5. Semiconductor suppliers now provide a Linux port to their SoC based hardwarereference platform, along with tool chains and a reference distribution

To take full advantage of the Linux operating system, Original Equipment Manufacturers(OEMs) have a choice of engaging with a commercial Linux vendor or adding additionalengineering capability in-house. Both models have proven successful, but each carriesits own specific costs.

Regardless of the direction the OEM chooses, the typical debug model available fortheir engineers is the same…a command line-based, client server environment basedon GDB (GNU Debugger). This model is illustrated in Figure 1, which depicts aninstantiation of GDBSERVER attached and running on each Linux process under debugon the target. Each GDBSERVER is communicating with the host through an Ethernetport.

In addition, it is important to understand that in this approach to Linux debug, thestandard Linux kernel is replaced with a “static” version specifically built with debuggingcode instrumented throughout. In addition and with only a few exceptions, all debugcommunication to the target via KGDB is limited to an RS232 serial link.

This approach provides an additional challenge to the developer that is using theinstrumented version of the Linux kernel by actually altering the performance of thetarget under debug from the “released version” that will eventually ship with the product.

Figure 1: Standard Linux Debug Model

While this is by default the accepted Linux debug environment, there are some wellunderstood limitations to this approach. For example, applications that consist ofmultiple processes will require multiple copies of GDBSERVER running in the oftenlimited target memory. This can affect the performance of the target under debug.There have been cases of a 50%+ degradation of target performance.

Even in the best case scenario where all kernel instrumentations and communicationchannels are available, there are still areas of the code that are inherently inaccessibleunder this debugging paradigm. The illustrated “problem” areas in Figure 2 havepresented multiple challenges to kernel and application developers. These areasinclude the large amount of threads under each process and kernel loadable modulesthat are code and data position independent. While it is possible for skilled developersto scaffold an environment based on existing technologies to address the debuggingneeds in these areas, such an environment has been shown to be very user-unfriendlyand non-scaleable under load.

Consider the case of Linux kernel loadable modules, which consist of an initializationroutine to be invoked at module loading time. Current debug paradigms suggest thatsuch modules be loaded, and their code and data offsets then be adjusted (manuallyand automatically) within the debugger. However, by this time, the initialization code ofthe module has already been executed and there is no possible way to debug aproblem in this area of the code. Another use scenario involves shared libraries, whichare often not handled well by GDBSERVER or equivalents.

Given these obstacles, it is not uncommon that many engineers to resort to printf (userspace) and printk (kernel space) as their primary debugging aids. Not withstanding the“ugliness” issues and time overhead of recompiling and linking in these messages, it isvery common for such debug “instrumentations” to sufficiently skew the behavior of thecode in a manner that leads to different behaviors in the program, with or without suchcode in place.

Figure 2: The “Problem” Areas

The Arriba Debugger: A Holistic Approach to Debugging Linux

The Arriba Debugger is designed from the ground up to provide a holistic approach todebugging embedded Linux and to be completely agnostic to the hardware platformarchitecture or version of Linux under development. In place of GDBSERVER andKGDB, VMON (a dynamically loadable, demand-based debug agent) runs on theembedded Linux target. Communicating with the Arriba Debugger on the host, VMONprovides total visibility of the Linux target, from user-level threads to the static kernel.

The VMON module has a very small memory footprint and even when active, has analmost immeasurable performance impact on the running system. At less than 250Kbin size, VMON is able to provide end-to-end debugging of the target over a singleEthernet connection.

ProblemAreas

Figure 3: The Arriba Solution

Arriba = “No Problem” Areas

“No Problem” 1 - Loadable Modules

Through the Arriba Debugger, VMON can be configured to signal the host when akernel module of a given property is loaded on the target. Upon the reception of thissignal, the Arriba Debugger will automatically and correctly load the symbol informationof the respective module, and place control at the entry point to the module initializationfunction. The user can now have full debug control of the module in question over ahigh-speed Ethernet link.

Traditionally, debugging of the Linux kernel or module (when it’s possible) isaccomplished with KGDB or JTAG, which completely halts the target under debug. Incontrast, an important feature of VMON is its ability to provide the same level ofdebugging non-preemptively. In other words, the Linux kernel on the target continues tohandle inbound and outbound network traffic, multimedia data, and other time-criticalactivities that are crucial in maintaining the appearance of normal execution to theoutside world. This ability is critical to many data and media-centric applications suchas set-top boxes, digital media appliances, and high-speed networking switches androuters.

“No Problem” 2: Debugging of Multiple Processes; Parent/Child Processes

In many instances, Linux application programmers need to create applications thatinvolve multiple processes. Such processes are spawned from a single parent process

earlier in the application initialization sequence. A frequent challenge revolves aroundthe need to set breakpoint(s) in the child process and eventually hit such breakpointswhen the child process is created and running. Straightforward as it may sound; this isan unsupported use-case with any debugging tools for Linux today, embedded orotherwise. As a workaround, developers often find themselves manually insertinginstrumented code in the child process with an infinite loop that is gated by a variableinitially set to ‘true’. This enables debugging tools such as GDBSERVER to attach tothe child process in question, change the value of the gating variable to ‘false’ tounblock to loop, and resume debugging.

Because VMON has ultimate visibility into the Linux target, events such as processcreations result in a notifying signal being sent to the Arriba Debugger on the host. TheArriba Debugger, upon determining that a breakpoint is pending for the child process,transmits the proper run-control sequence to ensure that such a breakpoint is set in thechild process code space.

“No Problem” 3: Debugging Kernel Drivers and Shared Libraries… ProductionReleased Kernel

Depending on the scope and breadth of the application, the list of Linux debug “problemareas” can range anywhere from the inability of the programmer to use the debug toolson his or her deployment platforms due to footprint and performance constraintsimposed by debugging techniques, to the tedious and error-prone workarounds thatresult in much wasted time and increased frustrations. The Arriba Debugger providesan in-depth solution to these problems, and beyond.

As a final example, consider the need for programmers and field application engineersto diagnose and fix bugs manifested in deployment. Under such conditions, the targetplatform is subject to severely limited debugging and communication access. VMON asa dynamically loadable module can be configured to be launched on already-deployedsystems. Thus, VMON, with its ability to effectively debug and diagnose such systemswith minimal intrusion, has time and time again proven to be an indispensable toolthrough all stages of the product lifecycle.

Eclipse Integrated Development Environment (IDE) Plug-in Support:

Viosoft Corporation recently announced an enhancement to the Arriba Debugger tosupport the latest open source Eclipse environment. The powerful Eclipse basedIntegrated Development Environment (IDE) is the cockpit for existing and future tools fordeveloping your Linux based design. This seamless integration with Eclipse providesthe developer with a complete Linux development and debug environment.

In addition to the Arriba Linux Debugger, there is also a new profiling tool called theArriba Linux Event Analyzer (LEA) that is also a plug in to an Eclipse IDE. This new toolprovides the ability to see all Linux events occurring on the target by capturing theinformation and displaying them in a time domain format. The Arriba LEA collects andprovides a significant amount of information about the Linux system including contextswitches among processes and threads, signals, and elapsed execution time.Simultaneously, LEA incurs a minimal impact on the overall CPU cycles and memoryfootprint. LEA light-weighted-ness, coupled with its ability to dynamically add and

remove instrumentation points on a production-ready running Linux system make LEAan ideal performance analysis and debugging tool, both for in-house development andfor field deployment.

An example result of this format is shown in Figure 4 below. Within this view, the usercan zoom in and out to gain a detailed understanding of how their code behaves and thetasks execute in both the kernel and user space areas. Providing the ability to measurelatencies, response times to external events and even the load each event representson the running system. This information is also available in a “raw” format that caneasily be imported to Microsoft Excel for additional post processing and analysis.

Because no two end-user applications are alike, each developer or team of developerswithin organization is likely to be interested in collecting and visualizing different aspectsof the system with LEA. The needs for having an open-ended analysis tools leads to adesign of LEA that is highly customizable. By creating and deploying their own kernelmodule plug-in to LEA, developers can easily and rapidly gain a level of visibility intotheir applications and system that not possible with other close-ended tools.

LEA uses the same instrumentation technology employed by VMON in the Arriba LinuxDebugger, which means that no debug patches or special compilation of the Linuxkernel is required. This capability makes LEA an ideal choice for deployment onproduction systems.

The combination of the Arriba Linux Debugger, Arriba LEA and the Eclipse IDE providesour customers with a very comprehensive and powerful Linux development environment.This solution was designed to shorten our customers’ time to market while providingdevelopers the ability to create a level of quality in their code that is unavailable fromany other solution.

Figure 4The Linux Event Analyzer (LEA)

Eclipse View

Seeing is believing!

As with any new technology solution to a well known challenge, it is reasonable forprudent readers to cast doubts and suspicions on the claims put forth in this article.

Viosoft Corporation welcomes you to contact us for a demonstration, discussion, andeven a test drive of our products. Regardless of the processor architecture you areusing or the version/source of the Linux under development, the Arriba Debugger isyour best solution. Whether you are using a uni-processor version of Linux or a SMPversion, you can take full advantage of Arriba.

So, put aside the “Hello World” examples that others demonstrate their tools on andbring on your million-line plus applications and dynamically loadable driver code and putus to the test. You will be glad that you did!

Copyright © 2008 Viosoft Corporation All rights reserved.

Unpublished rights (if any) reserved under the copyright laws of the United States of America and othercountries.

This document contains information that is proprietary to Viosoft Corporation (“Viosoft”). Any copying,reproducing, modifying or use of this information (in whole or in part) that is not expressly permitted inwriting by Viosoft or an authorized third party is strictly prohibited. At a minimum, this information isprotected under unfair competition and copyright laws. Violations thereof may result in criminal penaltiesand fines.

Any document provided in source format (i.e., in a modifiable form such as in FrameMaker or MicrosoftWord format) is subject to use and distribution restrictions that are independent of and supplemental toany and all confidentiality restrictions. UNDER NO CIRCUMSTANCES MAY A DOCUMENT PROVIDEDIN SOURCE FORMAT BE DISTRIBUTED TO A THIRD PARTY IN SOURCE FORMAT WITHOUT THEEXPRESS WRITTEN PERMISSION OF VIOSOFT CORPORATION

Viosoft reserves the right to change the information contained in this document to improve function,design or otherwise. Viosoft does not assume any liability arising out of the application or use of thisinformation, or of any error or omission in such information.

Any warranties, whether express, statutory, implied or otherwise, including but not limited to the impliedwarranties of merchantability or fitness for a particular purpose, are excluded. Except as expresslyprovided in any written license agreement from Viosoft or an authorized third party, the furnishing of thisdocument does not give recipient any license to any intellectual property rights, including any patentrights that cover the information in this document.

The information contained in this document shall not be exported, re-exported, transferred, or released,directly or indirectly, in violation of the law of any country or international law, regulation, treaty, ExecutiveOrder, statute, amendments or supplements thereto. Should a conflict arise regarding the export re-export, transfer, or release of the information contained in this document, the laws of the United States ofAmerica shall be the governing law.

The information contained in this document constitutes one or more of the following: commercialcomputer software, commercial computer software documentation or other commercial items. If the userof this information, or any related documentation of any kind, including related technical data or manuals,is an agency, department, or other entity of the United States government (“Government”), the use,duplication, reproduction, release, modification, disclosure, or transfer of this information, or any relateddocumentation of any kind, is restricted in accordance with Federal Acquisition Regulation 12.212 forcivilian agencies and Defense Federal Acquisition Regulation Supplement 227.7202 for military agencies.The use of this information by the Government is further restricted in accordance with the terms of thelicense agreement(s) and/or applicable contract terms and conditions covering this information fromViosoft or an authorized third party.

Arriba Linux Debugger, Linux Event Analyzer, LEA, VMON, VMON are trademarks or registeredtrademarks of Viosoft Corporation in the United States and other countries. All other trademarks referredto herein are the property of their respective owners.

Viosoft to insert trademarks