Selecting and Implementing An Embedded Database

System

Presented by Jeff Webb

March 2005

Article written by Michael Olson

IEEE Software, 2000

OverviewKey Strategy – Focus on application

requirementsEmbedded DB products vary

Some do less than what you need Some will do more

Choose the tool that most closely matches your needs

OverviewAfter choosing OS, HW platform and DB

you must design for reliabilityDesign for performance up frontEvaluate performance once the

application is built

Evaluate Database Services Available High end RDM systems provide:

Concurrency Transaction processing Disaster recovery

Although these features may be needed, enterprise systems are seldom a good choice due to: platform differences packaging differences

Evaluate Database Services AvailableSeveral embedded DB systems use the

same techniques as enterprise systems but in smaller packages.

Often, full blown disaster recovery is not needed

Many embedded databases are configurable allowing inclusion or exclusion of services

Consider Services RequiredNow… consider which services you

need. For example:Locking?

Will run faster without lockingRecovery from failures matter?

Lack of, or disabling will increase performance

Operating Systems For Embedded systems Hundreds of OS for variety of processor

hardware DB developers have an enormous job porting

their software to the variety of platforms Few OS dominate the market

MS Win CE Neutrino VxWorks Wind Rivers

Databases For Embedded systemsClassification

Client Server relational systems Client Server OO systems Embedded libraries

ClassificationClient Server relational systems

+ Programmers already know SQL - Extra run-time cost of client server

communications

Classification Client Server OO systems

Appear to be a good choice however Most are designed for Unix systems and

their deeply engrained assumptions about memory management and interprocess communications are difficult to port to embedded OS.

ClassificationEmbedded libraries

Created specifically for embedded systems Provide simple language API that does not

require SQL + Faster execution, increased reliability - Require developers to master

nonstandard programming interfaces

Platform Support Consider combination of:

OS Processor Storage system Exotic processor boards

Rare combinations can be difficult to fit Often embedded OS vendors maintain lists of

partner companies whose products run on their systems

Performance ConsiderationsHigh concurrency?Size of databaseMultiple control threads?Can not rely on standard benchmark

measuring systemsEvaluation of actual performance is a

must

Designing the ApplicationDesign for performanceConsider:

Speed Predictability & Reliability

Designing the Application Speed Predictability & Reliability

Data representationAccess patternsConfiguration

Designing for SpeedData representation/Access patterns/Configuration

Most embedded DB tools operate on a fixed set of data types

Every fetch and store may require translation

A few systems, mostly library types allow storage in program-native format

Designing for Speed Data representation/Access patterns/Configuration

Consider the queries that the application will need to make

Data should be laid out accordinglyCan keys be used that will allow related

records to be physically stored together?

B+tree storage typically performs faster than hash table algorithms

Designing for Speed Data representation/Access patterns/Configuration

Must understand the chosen systems configuration options Memory use for secondary cache Write data to disk or store in memory? Locking system granularity Entire locking system on/off Vendors often choose the wrong defaults

Designing the ApplicationDesign for performanceConsider:

Speed

Predictability & Reliability

Predictability & ReliabilityMay need to run with no humans

presentNot easyFanatically check return values and

error indicatorsResource exhaustion

More common in embedded systems Track and release resources yourself

Predictability & ReliabilityAre transactions required so that

changes won’t be lost after a crash?

The recovery system must be callable from the application program on start up?

Performance tuningCommon causes for poor performance

Contention – 2 or more threads contending for same data

Disk-to-memory transfers Deadlocks

Contention/Disk-to-memory transfers/ Deadlocks

When several threads are waiting for the same resources Use record level locking if possible Use smaller pages if possible to make

page level locks more like record level locks

Touch the Hot data last and hold it for short periods of time

Contention/Disk-to-memory transfers/ Deadlocks

Disk latency – mechanicalFlash RAMMore memory for buffer cache Indexes

Contention/Disk-to-memory transfers/ Deadlocks

Obj O1 locked

T1

Obj O2 locked

T2

T1 waits for a lock on O2

T2 waits for a lock on O1

Deadlocks Turn off locking if it is not needed and if the

application permits Break large transactions into several smaller

transactions Write transactions so that they all acquire the

same resources in the same order Consider Optimistic Concurrency Control

Price Helps make final decision Some are available at no cost Licensing methods vary

Per developer Per application using it Per deployment platform Per user is less common in embedded systems

During development / deployment

Questions