Post on 21-Jul-2018
Chapter 4
IDE SELECTION AND EV ALUA TION
4.1 Introduction
In order to meet several requirements, embedded systems need to. be developed globally
in collaboration with different people and companies. Embedded devices cannot operate
without the embedded software. A wide range of S/W tools to develop these software's
are available. Since a wide range of applications incorporate embedded systems,
embedded software development requires an even wider range of tools like editors,
translators, debuggers, memory analyzers, trace analyzers, profilers, version controllers,
code coverage tools etc to facilitate greater productivity [59]. Such tools are available
from different domains which are commercial, public or proprietary. Most of these tools
are integrated into a single package called Integrated Development Environment (IDE). It
allows the use of different individual tools to from a single development platfonn [80],
which helps the embedded software developer to deliver the products on time.
The success of a project is highly dependent on the chosen IDE. Choosing an IDE is
difficult for many reasons. The first is that the capabilities of these IDEs differ from one
vendor to another and. further these are available for low-end applications to high-end
applications. Secondly, in order to meet the ever-increasing complexity of the embedded
system's applications such as: medical electronics, defence, robotics, consumer
51
electronics etc, IDE's are incorporating more and more features. And also, no two
commercially available IDEs provide similar functionality, and choosing the right IDE
with desired features becomes a further complex task [23, 77]. In addition to this,
choosing a suitable IDE is often most important issue while selecting a particular
processor to an assigned project [60, 61]. If the existing IDE is not supporting the chosen
processor, then it may lead to the extra cost to the firm for purchasing the new IDE.
Secondly, the developer needs to spare extra time to understand the IDE and get familiar
with it. Therefore, in order to choose an IDE, it is necessary to understand the ever
increasing number of features. In this chapter, we propose a framework for selection and
evaluation ofIDEs. It may be noted that the envirOlID1ent used to develop applications for
embedded systems is different from the traditional development environment as in the
case of embedded systems the target processor of the application is different from the
processor of the host system.
4.1.1 Software Development Tool Chain (IDE) for Normal Applications
In normal software development environment, application software is developed al1d
used in the same system known as the host system. In this development environment, the
host system or workstation has standard peripheral devices. The tool chain used for
application software development [Figure 4.1] includes an Editor, Preprocessor,
Compiler, Assembler,. Linker, and Loader etc., which are explained bellow:
Editor: An editor is a software application used for editing plain text. Editors are often
provided with operating systems or software development packages, and can be used to
52
change configuration files and programmmg language source code. Editor generates
source code which can be further compiled and then executed for proper functioning.
Preprocessor: Preprocessor directives are lines included in the code of programs that are
not program statements but directives for the preprocessor. The preprocessor is executed
before the actual compilation of code begins; therefore the preprocessor understands all
these directives and translated into extended source code.
Executable Program
Preprocessor
Figure 4.1: Normal Applications Development Tool
Expanded Source
Program
Assembler: Assembler is used when source code is in assembly language. A program
written in assembly language consists of a series of instructions (mnemonics) that
correspond to a stream of executable instructions. Assembler creates object code by
translating mnemonics into machine language.
Compiler: Compiler is used when source code is in high-level language. A compiler is a
computer program (or set of programs) that translates text written in a high level
language into assembly language or object code.
53
Linker: A linker is a program that takes one or more objects generated by compilers and
assemblers and combine them into a single executable program. Many programming
languages allow writing different pieces of code, called modules, separately. This
simplifies the programming task because you can break a large program into small, more
l11anageable pieces. In addition to combiliing modules, a linker also replaces symbolic
addresses with real addresses.
Loader: A Loader is an operating system utility that copies programs from a storage
device to main memory, where they can be executed. In addition to copying a program
into main memory, the loader can also replace virtual addresses with physical addresses.
Most loaders are transparent, i.e, they cannot be executed directly, but the operating
system uses them when necessary.
We can use the traditional IDE for building embedded applications, provided the
applications need to work on the same or similar processor on which applications are
being developed. Machine codes are processor dependent and traditional IDEs support
for couple of processors; hence we need a separate IDE for ES development.
4.1.2 Software Development Tool Chain (IDE) for Embedded Applications
The generic tool chain for developrl1ent of embedded applications [Figure 4.2] involves
the integration of many tools. A brief discretion of these tools is explained as follows:
Cross Compilers: It takes an expanded source program written in the high level language
as an input and translates into equivalent assembly language program of a target
processor.
54
Cross Assemblers: It converts source program written in the assembly language into the
machine language of the target processor.
Locator: It does the linking and loading of the object programs. It is responsible for
generating the executable program as needed by the target programmers. Locators take
inputs that describe the target environment such as: memory layout of
RAM/ROM/FLASH, system data structures, global descriptor Tables, interrupt
structures, etc. and generate absolute addresses. The generated image from a locator is
then directly downloadable to the target hardware. The locator also produces a MAP file
that describes name, size and absolute and relative locations of the program segments and
global symbols.
Embedded system development tool chain is shown in Figure 4.3, which describes the
need of overall components of IDE of an embedded system development and shows that
it is different from normal s/w development IDE. With this, we understand the need of
separate selection and evaluation approach for IDEs used for embedded applications
development. We further can classify these tools of IDE into different groups in the next
section.
4.1.3 Classification of Development Tools for Embedded System De~ign
Embedded software development tools can be broadly categorized based on the literature
into three types viz. This classification will help the designers and developers to choose
the suitable IDE's based on the requirement and evaluation.
1. Computer-aided Software Engineering (CASE) tools: These tools are basically used for
requirements analysis, specification and design.
55
2. Compilation tools: Editors, Cross-compilers, assemblers, linkers, loaders and locators.
3. Debugging tools: These tools are used for targeting, validating and performance
monitoring. Debugging tools can be categorized into software debuggers (debug kernels
and source-code debuggers), low level ROM monitor kernels that have direct hardware
links into the target system, and hardware debuggers such as logic analyzers and in-
circuit emulators. Software debuggers are used to trace and monitor program execution.
Project Management Te}..1:
Editor Compiler! Cross Compiler!
-~l
Processing ) Cross j..ssembler Debugger .-~ --i. ___ ---
Linker/Locator I. ~ IDE ') J----~~. / ..
[ RTOS Y/~I~ ~ '--------" "-. /-------.
Simulator and S/W Emulati on .
r _." -"'---.-" "\
I Configuration J ~ Management _
Module Descriptor
Target I Prograrnmers ' J
Figure 4.2: Tools of a Generic IDE for ES Development.
56
.. Devekper Tool
, I r Editor Input! Output
+ Source Program
~ Pre Processor
Exp. Source Program J Exp. Source Program (FILL) (ALL)
---"-~-""-"'l"'---".!
Cross Compiler Cross Assembler
I I .~ ~
[ ,
Object Program
I ---. Linker! Locator .. DLL's
~ ,
Exe Program Debuggerl Simulator
'_«~_'_oJ(*,w.>«~"""""_~''"'''''_''' ~,,*_'m,_' '''''''''.Y,.'~o(_'' r
• Target Programmers .v~".,." .. Target System
Figure 4.3: Tool Chain for ES Development.
57
4.2 Related Work
Various traditional scoring techniques have been proposed for evaluation and selection of
the software's such as: IDEs, DBMS, CASE tools and OS based on Weighted Sum
Approach [55], Multicriteria [62] and LSP methods [48, 49]. For example SpecInt95
benchmark suite is used to evaluate the performance of programming tools, including
those used for embedded systems programming [76] and in [78], a general cost-benefit
decision model has been presented for the evaluation, comparison, and selection of
alternative products with a multiplicity of features such as:' complex computer system.
Let us consider the approaches given in [48, 58] for the IDE evaluation for general
purpose applications development.
D. Budgell, M. Thomson and Lutz Kirchner et aJ. [48] have proposed a quantitative
model for IDE selection and evaluation for Java enterprise applications. It is based on the
Logic Scoring of Preference (LSP) method for system evaluation to determine which
major IDEs satisfy typical software developer requirements. Lutz Kirchner and Jurgen
Jung [58] gave the framework for the evaluation of Meta-Modeling tools. Here, various
parameters like simulation, model transformation, metrics etc are considered. The authors
have demonstrated the applicability of the approach with two commercially available
tools such as: MetaEdit+ and Cubetto.
As far as safety critical systems are concerned, two approaches are addressed for IDE
selection [45], one is based on the S/W engineering viewpoint and other is based on
controlling engineering paradigm. However, for avionics, static source analyzer
evaluation have been proposed [74] to determine their adequacy for use in developing
real-time embedded software where the use of development tools and methods is
58
controlled by a federal regulatory agency. It is based on the weights and ranking of the
parameters.
The IDE evaluation approaches for general-purpose applications development described
above are not suitable for evaluation of IDEs used for the development of embedded
applications. These issues have been addressed by several researchers which are
discussed below, as its evaluation should include the following parameters:
1. Target Processor Support
2. RTOS suppOli
3. On-chip debugging
4. Simulation of on-chip peripherals
5. Optimization for code density, speed or both etc.
Several researchers have suggested different parameters that may be considered, while
selecting the IDE for embedded systems. As per the recommendations of IEEE software
engineering body of knowledge (SWEBOK) Carrington (2004) different parameters such
as: reduction in the connectivity load on the developer, and to allow the developer to
concentrate on the creative aspect of the process for systematic development. S.Kapur
[56] et al. has proposed the IDE selection based on the systems' requirements and then
the IDE is tested with the help of application. Based on test results, the selection is made.
R.B. Kline, A. Seffah [59] has suggested that the IDE should support the reusability. Thi~
is based on the results of three empirical studies ofIDE usability. In [60, 61] the authon
considered a different set of parameters such as: OS support, processor support
assemblers, compilers, debuggers and editors, and presented the review results of variow
IDEs that are commercially available with respect to these parameters in a tabula1
59
manner. A new set of parameters such as: portability, multiprocessing support, design
visualization, and rich content library are considered in [63, 64] and information sharing
and communication [65] are two parameters considered for selection of the IDE .
.Tim Turley [70] considered debugging tools and its importance in selecting the IDE. The
evaluation criterion of the IDE proposed by Berger [13] is based on the features of
compiler tools, hardware, debugging tools and performance measuring tools. He has .
outlined the set of parameter for selection and evaluation of these tools. In [79], a
checklist based evaluation methodology has been presented for the selection of the RTOS
in which IDE support is one of the parameter considerations.
As outlined in [10, 11, 20], the tool selection criteria must includes the parameters like
compiler, debugger, locator and RTOS. Among several parameters, RTOS support, GUI
features, availability of I/O connections, hardware-assisted debugging features, processor
support are most important parameter considerations in the IDE selection [68]. Several
parameters such· as: ease code creation, speed up in the development process, code
navigation, automatic error fixing, automatic indentation, graphical debugging, online
help, remote development etc are considered in the selection of the IDE [71].
In [73, 75], the authors demonstrated the evaluation and selection criteria of the
development tools especially the compilers for embedded systems. For the evaluation of
compiler several parameters such as: parameter passing, interrupts, memory allocation,
exceptions handling etc., are considered by Phillip A. Laplante [73]. Almost a different
set of parameters such as: compiler operation, linker features, support of match processor,
memory requirements [22], other language support, optimization features, nll1time
systems, intermpts, debugger, profilers, libraries, support, and cost are considered· in
60
approach given by A. Tetewsky et al. [75]. Further, they demonstrated the selection and
evaluation based on the tabular approach. In [77] several parameters such as: debuggers,
editors, compilers, EDA tools, emulation tools are considered while selecting the IDE.
From the above, the drawbacks in the selection and evaluation of IDE's can be
summarized as follows:
1. Even though several researchers and developers have suggested parameters which
may be considered for selection, there is no evaluation of the effectiveness of the
parameters on selection ofIDEs.
2. Since an integrated framework for evaluation is not available it is likely that the
developer may spend more time in selection and evaluation.
3. There exist several sub parameters for every parameter that are not considered.
For example compiler support with sub parameters such as memory models,
memory types, optimization etc. are important that are not considered while
selecting the IDE.
4. Existing approaches do not have a quantitative model for selection and evaluation.
5. These approaches do not carry out the case studies of the IDE s to evaluate and
select the proposed techniques.
Hence there is a need for a framework that helps the designers and programmers 1\1
selection and evaluation of the IDEs for embedded applications.
'61
In this chapter, we propose a framework to evaluate and select the IDEs for efficient
embedded system's development that involves the following steps:
1. Identify the IDE Evaluation criteria.
2. Identify the parameters that are to be considered for selection and evaluation of
IDEs based on the literature.
3. Perform the case studies oflDEs based on the identified parameters in step2.
4. Evaluation ofIDEs based on the parameters identified in step2 by lIsing step 1.
5. Rank the IDEs based on the different criteria.
6. Analyze the theoretical results with the practical results to find the COITectness of
the proposed framework.
7. Analyze the feedback results of the proposed method collected from different
designers.
These steps are presented one by one in the following sections.
4.3 Identification of Evaluation Criteria
As is outlined in [49], LSP method is used for complex systems evaluation and selection.
However, traditional scoring techniques can also be lIsed in the case where the systems
are not complex. In our system, the IDEs for the low to medium range applications are
considered. Thus we have decided to use the scoring method [58] for evaluation and
selection. In this framework, we have considered the weighted sum approach for
evaluation of IDE in which the weights (Wi) are assigned to the parameters (Pi) of the
IDE. These weights are chosen on the basis of domain knowledge and the requirements
62
of the application. For each IDE, the final .score (Sj) IS calculated by USll1g the
equation4.1.
n
Sj=I(WiPi)---·- (4.1)
i=O
Where j= 1 to 8 (total eight IDEs) and i= 1 to n where n is the no of parameters.
There exist· several internal parameters for each of the main parameters which are
considered for IDE selection. For example, compiler support is one of the main
parameters exists several internal parameters within the compiler [2] need to be
considered for the evaluation. The score is calculated individually for all the main
parameters of the IDE. These individual scores are added to get the final score for each of
these IDEs. This approach helps in two ways:
1. IDE selection based on the required parameter.
2. IDE selection based on the consideration of some or all the parameters.
To select the IDE for a given application(s) one needs to consider one of the mall1
parameters rather than all depending on the application. In such cases, approach one
helps to choose the right IDE. However, for general applications, some or all the
parameters of the IDE may need to be considered. In such cases, approach 2 helps to
choose the IDE. The implementation of these approaches is discussed in the section6.
63
4.4. Identification of Parameters
In this section, the various parameters and their sub parameters that need to be considered
. in the selection and evaluation are identified. The selection of the suitable IDE for a given
embedded project involves many tools and several parameters within each tool.
Based on the need of the particular project, the selection criterion of the IDE involves the
follows parameters:
I. Target processor support: If one needs to choose the IDE for a given application,
processor support is one of the main parameter considerations and there is no single IDE
that supports all the target processors. Each one supports a class of architectures and a set
of processors from each of the architecture. Hence, the developer or designer needs to
consider the processor support into account while selecting the IDE such that the IDE
must support different architectures and different vendors of the processors. This helps
the designers and developers' in the easy and fast development of applications because
the same IDE can be reused for future projects due to wide support of target processors.
2. Compiler features: Compiler is one of the most important tool in the IDE and plays a
vital role in the development of applications efficiently in tem1S of the execution time and
. code size. The efficiency of the generated code depends on several parameters such as:
optimization techniques, memory models, memory types, and data types that are
supported by the compiler. The compiler should support optimization for speed or code
or both to meet the requirements of the application. This is because, for some applications
code size is important where as for some other applications, speed is important.
Moreover, most of the embedded applications are battery operated and need to run
throughout the clock. Compiler should be chosen in such a way that it should support
64
required features. The parameters such as: code size, platform support, documentation,
strength of IDE, price, speed, optimization, and libraries support help in the selection of
the compiler for embedded systems.
3. Tools of IDE: An IDE should have an integration of several tools like Source Code
Editor, Cross Assembler, Cross Compiler, Locator, Simulator etc and it should have the
provision to suppOli of third party tools. Tool support help the designer to select the IDE
based on the tools that are required.
4. Simulation and Debugging: Simulation and debugging feature of the IDE are
important in the selection of the IDE. It saves a lot of resources such as time and money
and improves the confidence level of the designers and developers. It also provides
observeability and controllability of the application during the development. The IDE
should support various features of simulation and debugging of application, peripherals,
processors etc.
5. RTOS: RTOS provides the different features such as: IPC, real time scheduling, task
management, and memory management. Some of the IDEs support RTOS and some do
not. The designer needs to choose the IDE whether RTOS support required or not. The
use of RTOS helps to develop the applications faster and provides various features.
6. Host development: The host development environment may be either Windows or
Linux or any other desktop OS. Each IDE may support one or more such OSs. Based on
the need of the developer or application the IDE should be selected in terms of the host
environment.
7. Evaluation version: Several vendors are providing the evaluation versions of IDEs to
develop the applications. But, there is a limitation on either features or duration or code
65
size of these IDEs. For trail purpose, these tools can be used to verify the requirements
either meeting or not. For educational purpose and low-end applications development,
many of these evaluation versions will help in the development, understanding and
getting the familiarity.
8. Cost: One of the main parameters to choose the IDE in terms of the business prospects
is the cost. The top-level managers mainly involved the final decision of the IDE based
on the cost. The cost of the IDE depends on various parameters such as: number of
licenses, host environment, support, and target processor support etc. Hence, in this work,
cost of the IDEs is not taken into account because it is a variable parameter.
9. Network support: Many embedded systems may need to work under the network
environment and they are generally sub systems in a large system. Hence to interconnect
with each embedded system with the other it needs different communication protocols
like TCPIIP, Serial communication, Ethemet, USB etc. The IDE should support some of
these modes of the communication protocols for interconnecting the devices.
10. Evaluation board support: Evaluation boards that are supported by the IDEs will
help the developers to meet the time to market constraints because it speed up the
applications development with the help of evaluation boards.
Table 4.1 shows the summary of parameters proposed by different researchers and
designers.
66
No
2
3
4
5
6
7
8
Parameter
Target processor support
Compiler features
Tools of IDE
Simulation and Debugging
RTOS
Host development
Evaluation version
Cost
Considered by
[60,61,63,64,68]
[10,13,60,61,63,64,71,73,75,77]:
[10,11,60,61,63,64,71,73,75,77]
[10,13,60,61,68,70,71,75,77]
[10,60,61,63,64,68,75,79]
[56,60,61]
[75]
[65, 73]
Table 4.1: Parameters for IDE Selection and Evaluation.
4.5 Case Studies of IDEs based on the Identified Parameters
In this work, eight IDEs [82-87] were chosen which are widely used for commercial
applications. They are studied based on parameters described in the above section. The
compiled results are shown with respect to each selection criterion that was not available
earlier in the literature. The results given are in the form of Tables and help the designers
to choose the one required. The availability of the specified feature in the IDE is shown
by the symbol (--J). This section outlines the compiled results based on the selection
criteria of the IDE and the limitations of the existing approaches and its elimination in the
proposed framework are described below.
Target processor support: Table 4.2 depicts the results of the IDEs based on the target
processor support. It shows the various families of processors that are supported by the
IDE in terms of architecture and data bus size. The IDE that supports more number of
target processors is the best one because the processor may change from one project to
another project. In such cases the same IDE can be reused instead of going for the new
IDE which further adds the cost and time to the designers. The existing approaches
67
considered a sub class of target processors support. In the proposed scheme, we have
considered the detail support of the processors which helps in the selection of the IDE
efficiently. More over, it helps to the developer to choose the IDE while considering the
support of processors for the feature projects.
Target processors (Arch) Kcil HT IAR Raisonancc SP J Systems BiPOM AVOCET Wickcn- hacuser
8051
80251
ARM
STI0/XCI6X
MPS 430
AVR
Motorola
PIC
Power PC
8 bit
16 bit
32 bit
../
../
/
../
../
../
../
../
../
../
../
../
../ ../
../ ../
../ ../
../
./
No o{bit o{the processors
../
../
../
../
../
../
../
../
../
../
../
../ ../
Table 4.2: Evaluation of the IDEs based on the Target Processors Support.
Compiler Features: Table 4.3 depicts the results of the IDE based on the compiler
features. It shows the features of compilers in terms of memory models, memory types,
supported data types, optimization languages support and evaluation version. These
features guide the developer to choose the IDE based on the compiler support. The
number of data types supported by each IDE is shown in numbers within the brackets.
Several existing approaches discussed earlier are considered compiler support in the
selection. In the proposed approach, several other sub parameters of the compiler are
considered which helps in efficient selection of the IDE.
68
COl1lpiler features Keil . HT IAR Raisonancc S r J S ystcl1ls BirOM AVOCET Wickcn-hacuscr
No. of Data type ~ (17) ~( II) ~ (II) ~ (14) 7 (X) ~ (II) 7 (8) v (10)
optimization for code ./ ./ ./
optimization for speed ./ ./ ./ ./ ./
Interrupts ./ ./ ./ ./ ./ ./ ./
Memory Model
Small ./ ./ ./ ./ ./
Medium ',/ ./ .'/
COlllpact ./ ./ ./
Largc ./ ./ ./ ./
Huge ./
Generic ./
Far ./
Internal and External
Tiny ./ ./
Memory Types
Code ./ ./ ./ ./ ./ ./
Data ./ ./ ./ ./ ./ ./
Idata ./ ./ ./
Pdata ./ ./ ./
Xdata ./ ./ ./
Bdata ./ ./ ./
Near ./ ./
For ./ ./ ./
Huge ./
Incar
Evaluation Version or free version
Lilllitation in eode 2kb 4kb 8kb 2kb 2kb 04kb 8kb
Limitation in time No limit 45 days 30 days No limit No limit No limit No limit No limit
Limitation in features No limit No limit No limit No limit No limit No limit No limit
Language support
Assembly ./ ./ ./ ./ ./ ./ ./ ./
C ./ ./ ./ ./ ./ ./ ./ ./
C++ ./
Table 4.3: Evaluation of the IDEs based on the Compiler Features.
69
Tools of IDE: An IDE should have the integration of many tools and provIsIon to
support third party tools as mentioned in section 4. The compiled results of the mEs
based on the tools support are depicted in the Table 4.4. Several existing methods
consider the sub set of the tools in the evaluation and selection. In the proposed approach
several other tools also considered for efficient selection of the IDE.
Tools
Source code editor
Cross Assembler
Cross Compiler
Locator
Simulator
Source code Library
ROM Monitor
Flash programmer(isp)
Debugger
Remote debugger
Project Manager
RTOS
Version controller
Code wizard
Third party Tools
I<eil HT JAR Raisonancc
./
./
./
./
./
SPJ BiPOM
./
./
./
AVOCET
Table 4.4: Evaluation of the IDEs based on the Tools.
Wickcn
haeuser
Simulation and Debugging: To meet the time to market constraints, the IDE should
suppOli the various features of simulation and debugging as mentioned in section 4. The
compiled features of simulation and debugging are shown in the Table 4.5.
70
Simulation and debugging Keil HT IAR Raisonancc SPJ BiPOM AVOCET Wickcn-
features Systcms hacuscr
High Level debugging v' v' v' v' v' v' v'
Interrupts v' v' v' v' v' v' v'
Rcmote dcbugging v' v' v' v'
Uscr debugg functions v' v' v' v'
Basic debugging v' v' v' v'
CAN Controllcr v' v' v' v'
GPIO v' v' v' v' v'
Timcrs/ Counters v' v' v' v' v'
Watch dog timer v' v' v' v'
ADC v' v' v'
DAC v' v' v'
PWM v' v'
ICE v' v' v' v'
UART v' v' v'
12C v' v'
HAG v' v' v' v' v'
Ethernct v' v'
CPU rcgisters v' v' v' v' v' v'
Pcrformance analyzer v'
Logic Analyzer v'
Serial window v' v' v' v' v' v' v'
ROM Monitor v' v' v' v' v' v' v'
Memories v' v' v' v' v' v'
Extra features ++ ** # +
Table 4.5: Evaluation of the IDEs based on the Simulation and Debugging Features.
+ Automatic error detection of stack ovelflow. undeljlows (lnd out of range memOlY writes (lnd reads are supported. * PKD- Peripheral Development Kit is a software intel/ace by which the user can adopt the simulation of newlycreated peripherals. ++ C- Wizard is a graphical tool to generate C code ji'o171 graphical representation of the application # LCD, 8255, 8155-intel/ace simulation is available. ** C-SPY debugger
71
The IDEs that support several features which helps the designer in debugging the
application can be chosen. Several existing methods discussed in the section 4 do not
support these features that are important. In the prospered framework, several parameters
are considered to evaluate the IDE based on the simulation and debugging that helps in
right selection of the IDE.
RTOS: RTOS support is one of the important features of the IDE and the different
features of RTOS are mentioned in section 5 that are to be considered in the selection of
the IDE. Table 4.6 depicts the results of the IDE based on the RTOS suppOli.
Keil HT JAR Raisonance SPJ BiPOM AVOCET Wicken-Systems haeuser
RTX51 Salvo Lite CMX- RTX KRXA Tiny AvSYS
RTX tiny Salvo LE ThreadX KRXA
Salvo pro mCOS-1i
SMX
MiSPO
Fusion
----------------------------------------------------~-----Table 4.6: Evaluation of the IDEs based on RTOS Support.
Host development: Table 4.7 depicts the results of the IDE based on the host environment
which intern helps the developer to choose the IDE. Very few researchers have
considered the host development in the selection of the IDE.
Host OS Keil HT JAR Raisonance SPJ Systems BiPOM AVOCET Wicken-haeuser
Windows,/, ,/ ,/
Linux
Mac OS ,/
Table 4.7: Results of the IDE based on the Host Environment.
72
Evaluation version: Many IDEs provide the evaluation versIOns to develop the
applications. Table 4.3 depicts the results of the IDE based on limitation in code, time
and features. Most of the existing approaches do not consider the evaluation version of
the IDEs in the selection. This is an important feature that needs to be considered in the
selection of IDE for educational, R&D and prototyping the designs.
Language support: An IDE should support one or more languages so that the developer
can choose the language for implementation of the applications for efficient code
generation. Table 4.3 depicts the results of the IDE based on the language support.
Benchmarking: There exists a set of benchmarks for IDEs of embedded systems. Some
IDES are evaluated with standard benchmarks and their results are shown in the Table
4.8. This provision makes task of evaluation easy and efficient. Due to the evaluation
versions that are considered in the system, many of the bench marking results is not
depicted because of the code size limitation. Table 4.8 depicts the results of the IDE
based on the various benchmarks.
Keil HT JAR Raisonance SP.l Systems BiPOM A YOCET Wickcn-haeuser
--------------£EMBC Allto mobile suite
NXP LPC 3180
8051 (12 )
1240 b
6.012 s
8051 (12 )
5269 b
1.096 s
805 J (12 )
9236 b
4 .. 5 s
STR91xFA and
NxP LPC 212x
AYR
91 ms
Sieve Benchmark
989 b
Dhrystonc Benchmark
5059 b
Whetstone Benchmark
9960 b
Table 4.8: Evaluation of the IDEs based on the Results of the Benchmarks.
73
Cost: The cost of the IDE depends on various parameters such as: of number of licenses,
host environment, support, and target processor support etc. Hence, in this work, cost of
the IDEs is not taken into account. Very few methods have proposed the evaluation based
on the cost.
Network Support: Generally, embedded systems are subsystems in a large system and
work under the networked environments. Hence to interconnect with each subsystem, we
have to use different communication protocols like TCP/JP, Ethernet, Serial
communication, CAN bus, I2C etc. which is shown in the Table 4.5. The existing
approaches do not considered this feature though it is very important. However in the
proposed framework, the IDE are evaluated based on this feature, which in tern helps the
designers to select the IDE efficiently.
Evaluation board support: In addition to many parameters that are described above,
evaluation boards that are available for specific IDE is shown in the Table 4.9. In the
initial stage of applications development, the evaluation boards playa vital role in the
entire development process. Table 4.9 helps the designers to choose the IDE based on
the evaluation board support. This is an important feature needs to consider especially for
educational, R&D and prototyping the initial designs and existing approaches do not
considered this feature in the selection ofIDE.
S.No. Keil HT IAR Raisonance SPJ Systems BiPOM AVOCET Wicken-haeuser
MCBx51 I.XA- I. XEVA I. Mini -51 I.MINI- I.FlexGate -III
2 MCB 900 G30 2. XEVA- 2. SBC -51 MA75
3 MCB 950 Fast 3 .. Mini ARM 2.8051 Kit
4 MCBXC 866
5 MCBXC88x
Table4.9: Evaluation of the IDEs based on the Evaluation Boards.
74
4.6 Evaluation of IDEs based on the Identified Parameters
IDE Evaluation methodology as described in section 4.3 is two ways that are described
below.
1. Based on the specific parameter: In order to select the IDE based on specific
parameter, first we need to evaluate IDEs based on different parameters like language
support, evaluation version, target processors support, compiler features, tools support,
simulation and debuggers, development platform, RTOS and benchmarking. Secondly,.
based on these results, IDE can be selected with respect to the required parameter as
given below.
Evaluation results based on the Target processor support: Table 4.2 shows the features
of the IDEs based on the target processors support.
Parameter Weight Keil HT JAR Raisonance SP] BiPOM AVOCET Wicken
8051/251
ARM
STIO/XCI6X
MPS 430
AVR
PIC
Power PC
8 bit
16 bit
32 bit
Total Score
5555334
2 2 0 0
o O· I
o 0 0 0 o
o 0 0 0 0
o 0 0 0 0
o 0 000 0 0
4
1.
o
o
o
o
No of bit pl'Ocess(}J:I'
II 11 9 9 10 10 9 9
2 . 2 22 2
o
00000
00000
o 0
00000
00000
00000
55665
2
o
o
o
o
o
o
5
Table 4.10: Evaluation of the IDEs based on the Target Processor Support.
75
3
o
o
o
o
o
o
6
3
o
o
o
o
o
o
6
These results are further converted into quantitative parameters and the respective
weights as shown in Table 4.10. The result shows that Keil IDE has scored the top
followed by IAR and HT respectively.
Evaluation Results based on the Compiler Features: Table 4.3 shows the compiler
features, language support and evaluation version of the IDEs. These features are
translated into quantitative parameters and the weights are assigned for each of these
parameters as shown in Table 4.11 based on the relative importance of each parameter.
Language support and evaluation version are also considered here. The evaluation result
shows that Keil IDE has got the highest score followed by TAR and Raisonance, and
Wicken-haeuser respectively.
Parameter Weight Kei1 HT 1AR Raisonancc
No of Data
types
Optimization
l'or code
Optimization
for speed
Inlen'upts
Memory
Model
3
4
5
4
2
Memory Types 6
Evaluation
Version
Language
support
Total Score
5
2
3 9 262 6 2 6
4 4 4 0 o
5 5 5 5
4 4 4 4
3 6 4 8 5 10 5 10
3 18 2 12 3 18 2 12
4 20 2 10 3 15 5 25
2 4 2 2 2
18 70 14 51 17 64 17 64
SP.I BiPOM AVOCET Wickcn
2 6 2 6 2 CJ
00000 o o o
o 0 5 0 o 5
4 4 4 4
245 10 2 4
3 18 12
4 20 4 20 (, 30 5 25
2 2 2
13 54 15 53 12 50 15 61
Table 4.11: Evaluation of the IDEs based on the Compiler Features,
76
Evaluation results based on the Tool support: Table 4.4 shows the features of the IDEs
based on the tool support. These results are further converted into quantitative parameters
as shown in Table 4.12. The result shows that Keil IDE, IAR and HT have got the same
score followed by Raisonance and SPJ respectively.
Parameter Keil HT IAR Raisonance SP] BiPOM AVOCET Wicken-haeuser
Tools 3 3. 3 2 2
Table 4.12: Evaluation of the IDEs based on the Tools.
Evaluation results based Oil Simulator and Debugging: Table 4.5 shows the features of
each IDE based on the simulation and debugging. These results are further converted into
quantitative parameters as shown in Table 4.13. The result shows that Keil IDE, IAR and
HT have the same perfomlance followed by Raisonance.
Parameter Keil HT IAR Raisonance SP] BiPOM AVOCET Wicken-haellser
Simulation 5 5 5 4 222
Table 4.13: Evaluation of the IDEs based on the Simulation and Debugging.
Evaluation results based on the RTOS and Host development: The features such as:
RTOS support and Host development support of the IDEs are shown in the Table 4.6 and
Table 4.7 respectively and the quantitative parameters of these results in the Table 4.14.
It shows that HT perfomling the best followed by IAR and Keil respectively.
Parameter Keil HT IAR Raisonance SP] BiPOM AVOCET Wicken-haellser
Host 3
RTOS 2 o
2
o o
Table 4.14: Evaluation of the IDEs based on the RTOS and Host Development.
77
2. Based on Complete Evaluation ~esults: In this approach, IDEs are evaluated by
considering all the parameters along with their respective weights as shown in the Table
4.15. The score for each parameter is calculated by using the equation I given in section
3. The complete evaluation result of each IDE is shown in the Table 4.16. It shows that
the cumulative score is highest for Keil IDE followed by JAR, HT and Raisonance.
No Name of the parameter
Target processors Support
Compiler features
Simulation or debuggers Features
Development platform
Tools
Weight
6
5
4
1
3
2
3
4
5
6
7
9
10
Evaluation version or free tools support
Language Support
Is consider in compilers
RTOS supported and it's features
Language support
11 Benchmarking
" 3
2
Table 4.15: Relevance of Selected Criteria.
Parameter Keil HT IAR Raisonance SPJ BiPOM AVOCET
Compiler 35 26 32 32 27 27 25
Target Processor 66 54 60 54 30 36 30
Simulation 20 20 20 16 8 8 8
Tools 21 21 21 14 14 ·7 7
Plot form 2 6 '2 2 2 2 4
RTOS 3 3 6 3 3 0 0
Score 147 130 141 121 84 80 74
Table 4.16: Complete Evaluation Results of IDEs.
78
W icken-haeuser
31
36
4
7
2
0
80
It is because target processor support, compiler features are favorable with these tools
although development platfoJ1l1 and RTOS support are not. Table 4.17 shows the
complete evaluation results ofIDEs with out considering weights. Keil, JAR, and HT are
depicting almost similar results with a variation of sore of 1. However, the performance
of Keil IDE is at the top followed by JAR and HT. The results shown in the Table 4.17
helps the designers to choose the IDE based on the specific parameter and Table 4.18
shows the complete evaluation results of IDEs with weights and Table 4.19 shows the
results in both the cases. We have observed that the results in both the cases i.e. with
weights and without weights are almost the same.
Wicken-Parameter Keil HT IAR Raisonance SPJ BiPOM AVOCET haeuser
Compiler 7 5.1 6.4 6.4 5.4 5.3 5 6.1 Target Processor 11 9 10 9 5 6 5 6
Simulation 5 5 5 4 2 2 2
Tools 3 3 3 2 2 1
Plot form 3 1 2 1
RTOS 2 0 0 0 Score with out weights 28 26.1 27.4 23.4 16.4 15.3 15 15.1
Table 4.17: Evaluation Results without Weights.
Wicken-
Parameter Keil HT IAR Raisonarice SPJ l3iPOM AVOCET haeuser
Score 147 130 141 121 84 80 74 80
Score with out
weights 28 26.1 27.4 23.4 16.4 15.3 15 15.1
Table 4.18: Evaluation Results with Weights
79
Name of the Weight Kcil I-IT IAR Raisonancc SP.I BiPOM ;\ VOCET Wickcn-
parameter haellser
Target 6 II 66 9 54 10 60 9 54 5 30 6 36 5 30 6 36
Compiler 5 7 35 5.1 26 6.4 32 6.4 32 5.4 27 5.3 27 5 25 6.1 31
Simulation 4. 5 20 5 20 5 20 4 16 2 8 2 8 2 8 4
or debuggers
Development 2 2 3 6 2 2 2 2 2 4 2
platform
Tools 7 3 21 3 21 3 21 2 14 2 14 7 7 7
RTOS 3 3 3 2 6 3 3 0 0 0 0 0 0
Total Score 147 130 141 121 84 80 74 80
Table 4.19: Final Evaluation Results.
4.7 Ranking of ~he IDEs based on the different Criteria
Table 4.20 shows the ranking of the IDEs based on different criteria. This will help the
designer to arrive the decision easily. Figure 4.4 shows the result of the evaluation of
IDEs based on various aspects and Figure 4.5 shows the final score of each IDE. It shows
that Keil IDE has scored the highest followed by IAR and HT among other IDEs.
Parameter 1st Rank 2nd Rank 3,d Rank Remarks Rank
Compiler Keil JAR Raisonance JAR and Raisonance are in second position
Target Keil JAR HT& Processor Raisonance Simulation & Keil HT JAR All are at the same position Debugging Tools Keil HT JAR All are at the same position
Development HT AVOCET Remaining all are at the same position Environment. RTOS JAR Keil, HT, Raisonance and SPJ are in
second position Overall Keil JAR HT
Table 4.20:Ranking of the IDEs.
80
Wicken-haeuser ~ a RTOS
AVOCET • Platform OTools
8iPOM ~ o Sim ulation - • Target Proce ssor SPJ [J Compiler
iIiiiI Raisonance
IAR '--
HT
IilIIi1IIiI Keil
0 2 4 6 8 10 12
Figure 4.4: Evaluation Results with actual cores.
Wicken-haeuser ~-~EE' *C:::::::::::::::::::J
AVOCET ~-~, ~~~!:3
BiPOM ~-~!:!!:,!:::!:~::!,!,!:!a SPJ .....
• Score 'tVith out weights
[J Score
RaisonanceJll! ...... ~~:E3!5l!!im:a~!El1==:::;:]
o ~ ~ 00 00 100 1~ 1~ lffi
Figure 4.5: Ranking of the IDEs.
81
4.8 Analysis of Theoretical Results with the Practical Results
To verify the actual performance of the top three evaluated IDEs, we have considered
four different applications. Each application is executed on these IDEs and the extracted
results are shown in Table 4.21 to Table 4.24. The resulls show that the code size and
object file size is small with Keil compared with other two IDEs. The similar behavior
was observed in the evaluation of IDEs. Hence, the evaluation framework given in this
work is converging with the actual results. Similarly, it has also been observed that as
regards the simulation time for these applications, KeiI IDE has taken less execution time
as compared with other two IDEs. It shows that the applications are optimized for code
density and speed as well with KeiIIDE. We present the feedback results of the proposed
framework in the next section to corroborate the proposed work.
Code Size Idata (in Source file size Object file size List fi Ie size Simulation (in bytes) bytes) (in kB) (in kB) (in kB) time (in sec)
---~-.-.----.---------- .. ~------.-.-~---------.-.---KEIL 82 I I 2 3 2 HT 86 3 17 3 TAR 176 2 8 5 4
Table 4.21 : Application 1- Dancing LEDs.
Code size idata Source file size Object file size List file size Simulation time
KEIL 48 9 2 2
HT 51 0 ~ 10 ,)
IAR 112 0 5 4 2
Table 4.22: Application 2- Timer Programming.
82
Code size idata Source file
KEIL 82
HT
IAR
704
160
9
6
12
6
Object file
4
9
4
List file Simulation (in sec)
3
59
4
4
14
12
Table 4.23: Application 3- Display a Word through Serial Window.
Code size idata(in bytes) Source file Object file List file Simulation (in sec)
KEIL 1096 30 2 2 3 2
HT 3677 28 2 3 14 4
IAR 2016 0 2 6 5 3
Table 4.24: Application 4: LCD Interfacing.
Further, the advantages and disadvantages of this work are given below.
Advantages:
1. This helps the designers or pro.:ect managers to choose the IDE i.e. suitable for a
given project quickly and efficiently.
2. The various parameters that need to be considered to select the IDE are illustrated
along with the results.
3. Ranking of the IDE considered in this work helps the designer in selecting the IDE.
4. It helps the designers in selecting the IDE based on one or more parameters or their
combinations depending on the requirements.
5. The IDEs which are evaluated in [60, 61] have not been considered for analysis.
6. Many features of IDEs such as: compiler features and processor support are compared
in detail which was not done earlier.
7. Features such as: RTOS support, evaluation board's support, and N/W features are
also considered.
83
Limitations: , 1. The graphical IDEs and the IDEs that are supporting the high-end applications are not
considered.
2. Due to the evaluation versions of the IDEs, many of the benchmarking results have
not been depicted.
4.9 Summary
Many applications incorporate embedded systems and each application demands
specialized requirements. To meet these requirements, the IDE may provide different
features. There exist several IDEs for different applications of Embedded systems like
low-end applications, medium-end applications and high-end applications. To support
these applications, no single IDE exists due to versatile nature of these applications. In
this chapter, frameworks for the selection and evaluation of the commercially available
IDEs for embedded systems' development have been proposed. Based on the evaluation
methods defined within the framework, we have applied the proposed approach on these
IDEs and ranked based on the different selection parameters. Finally, evaluation results
of these IDEs are analyzed with practical applications and observed that the practical
results are correlating with the evaluation results of the proposed framework.
Here, we have considered IDEs for low-end and medium-end applications. This may be
further extended to high-end applications of embedded systems in feature work.
Performance benchmarks may be considered for various applications to find the
suitability of the IDE further.
84