Tuning and Scalability for Sonic

Tuning and Scalability for Sonic

Analyzing, testing and tuning JMS/ESB performance

Jiri De JagereSolution Engineer EMEA

© 2007 Progress Software Corporation2 Session ID: Session Title

D I S C L A I M E R

Setting Performance Expectations

System performance is highly dependent on machine, application and product version. Performance levels described here may or may not be achievable in a specific deployment.

Tuning techniques often interact with specific features, operating environment characteristics and load factors. You should test every option carefully and make your own decision regarding the relative advantages of each approach.

D I S C L A I M E R


Agenda

Methodology Analysis Testing Tuning



Performance Engineering Terms

“System Under Test”

“Test Harness”

“Load” = “Sessions” * “Delivery Rate”

“Latency” = ReceiveTime – SendTime“Test Components”

“ExternalComponents”

“Platform” “System Metric”

R

V

“Variable” =client param,app param,system param

V

V

Expert Tip: Limit scope to those test components thatare critical to performance and under your control


Concepts: Partitioning Resource Usage

% C

PU

Partitionable resources can be broken down as the sum of the contributions of each test component on the system

• Total resource usage is limited by system capacity• Becomes the bottleneck as it nears 100%• Goal is linear scalability as additional resource is added• Vertical versus Horizontal scalability

• Total latency is the sum across all resource latencies, i.e.:Latency = CPU_time + Disk_time + Socket_time + wait_sleep

X

Free

Z

Y

( Overhead X Message rate ) = Load∑(Writes/sec)(msg/sec)(writes/msg)svcs

Bottom-Up Rule: Test and tune each component before youtest and tune the aggregate.


Concepts: Computer Platform Resources

Favorite tools: task mgr, perfmon, top, ping –s, traceroute

Disk I/O(read/write)

Network I/O(send/receive)

CPU time

Memory(in use, swap)

# Threads

• Use level of detail appropriate to the question being asked• Machine resource (such as %CPU) artifacts:

• side effects from uncontrolled applications• timing of refresh interval• correlation with test intervals

• Scaling across different platforms and resource types


The Performance Engineering Project

The Project is Goal Driven

Analyze

Test

Tune

For each iteration1. Test performance vs goal2. Identify likeliest area for gain

Startup tasks1. Define project completion goals2. Staff benchmarking skills3. Acquire test harness

Expert Tip: Schedule daily meetings to share results andreprioritize test plan.


Performance Project Skills

Requirements Expert• SLA/QoS levels – minimal & optimal• Predicted load – current & future• Distribution topology

Integration Expert• Allowable design options• Cost to deploy• Cost to maintain

Testing Expert• Load generation tool• Bottleneck diagnosis• Tuning and optimization

SOLUTION

I.E.T.E.

R.E.

Cos

t/Ben

efit

Load/Distribution

Design Options


Tools for a Messaging Benchmark

Configurator – creates conditions to bring system under test into known state

Harness – the platforms and components whose performance response is being measured

Analyzer – tools and procedures to make meaningful conclusions based on result data.

Platform

Component

System Under Test

TestHarness

TestConfigurator

TestAnalyzer


Performance Project Timeline

1 2 3 4 5 6 7 8

DevelopmentProject

Week

Process Dev

Service Dev Sizing

System Test

Deployment Plan

Launch

PerformanceProjectPerf Prototype

Design Application Tuning


Agenda




Performance Analysis

Performance scenarios• Requirements

• Goals

System characterization• Platforms

• Architecture

Test cases


Performance Scenario Specification

First, triage performance-sensitive processes• substantial messaging load and latency requirements

• impact of resource-intensive services

Document only process messaging & services • leave out Application specific logic – this is a prototype

Set specific messaging performance goals• Message rate and size

• Minimum and average latency required

Try to quantify actual value and risk• Why this use case matters

Rule of Thumb: Focus on broker loads over 10 msg/sec or 1 MByte/sec,and service loads over 10,000 per hour.


Example: Performance scenario specification

Overall project scope:• Project goals and process• Deployment environment• System architecture

For each Scenario:• Description of business process• Operational constraints (QoS, recovery, availability)• Performance goals, including business benefits

validate CreditCard Event

make a paymentEvent

Midas BillingSystem

Make Payment with Credit Card

make_payment (Progress Adapter)

CC Validate

Payementech(Mocked Object)

Make_payement API

Client Apps

is Billing System Up? Queue

yes

no

Credit Card SyntaxValidation

1

2

3

4

5

7-n

6

8

9

10

XML validation

7


System Characterization

Scope current and future hardware options available to the project

Identify geographical locations, firewalls and predefined service hosting restrictions

Define predefined Endpoints and Services Define data models and identify required

conversions and transformations.


DMZField DMZ

Platform configuration specification

CPUnumbertypespeed

Networkbandwidthlatencyspeed

Disktypespeed

Firewallcryptoslatency

Memorysizespeed

Rule of Thumb:LAN 15 – 150 MBytes / secondDisk: 2 – 10 MBytes / secondXSLT: 200 – 300 KBytes / second


Architecture Spec: Service distribution

Identify services performance characteristics Identify high-bandwidth message channels Decide which components can be modified Annotate with message load estimates

Back Office PartnerField Front Office

CRM

Fin

an

ce

SF

ACRM

SC

M

ESBESB

Tra

ckin

g

Ser

vice

ESB

Po

S

ERP

Adapter

SCM

Adapter

Partner ESB

Integration Broker


Architecture Spec: Data Integration

Approximate the complexity of data schemas Identify performance critical transformation events Estimate message size Identify acceptable potential services

Data Schemas 1…n Data Schemas 2…m

?

Expert Tip: Transform tools vary in efficiency:XSLT – slowest (but most standard)Semantic modeler – generally faster (e.g. Sonic DXSI)Custom text service – fastest, but not as flexible


Network

Memory

Disk

CPU

Platform Profile: Real-time messaging

% capacity

System resource limitations

90%

70%

20%

5%

Rule of Thumb: Real-time, 1 KB messages, broker performance is about 1,000 to 10,000 msg/sec for each 1 gHz cpu power.


Network

Memory

Disk

CPU

Platform Profile: Queued requests

% capacity

System resource limitations

50%

20%

40%

85%

Rule of Thumb: Queued msgs, 1 KB messages, broker performance is about 100 to 1,000 msg/sec for each 1 gHz cpu power.


Specifying Test Cases

Factors to include:• Load, sizes, complexity of messaging• Range of scalability to try (e.g. min/max msg size)• Basic ESB Process model• Basic distribution architecture

Details to avoid:• Application code (unless readily available)• Detailed transformation maps

Define relevant variables:• Fixed factors• Test Variables• Dependent measures


Typical test variables

JMS Client variables:• Connection / session usage

• Quality of Service

• Interaction mode:

• Message size and shape

ESB container variables• Service provisioning and parameters

• Endpoint / Connection parameters

• Process implementation and design

Expert Tip: External JMS client variables are easily managedwith the Test Harness.


Example: Test Case Specification

For each identified Test Case there is a section specifying the following: Overview of test

• How this use case relates to the scenario• Key decision points being examined

Functional definition• How to simulate the performance impact• Description of ESB processes and services• Samples messages• Design alternatives that will be compared

Test definition• Variables manipulated • Variables measured

Completion criteria• Throughput and latency goals• Issues and options that may merit investigation


Agenda




Testing

Think about the approach you want to take• Bottom-up vs Top-down

Create a controllable environment • Simulate non-essential services

• Use a testing tool for simulating clients

Evaluate the results


Simulating clients with Test Harness

File or JNDI connection configuration Producer / Consumer parameters Message generation

System Under Test

TestHarness

Connection

Broker

JNDI

MsgPool

Reply

Request

MessageGeneration


Evaluating the results

Test Harness output:• Throughput (msg/sec)

• Latency (msecs per round trip)

• Message size (bytes)

System measures:• CPU usage (%usr, %sys)

• Disk usage (writes/sec)

Broker metrics:• Messaging rate (bytes/second)

• Peak queue size (bytes)

Expert Tip: Spreadsheets are excellent for documenting and interpreting the results


Evaluating the results

Evaluate against the goals

Perform a bottleneck analysis

Compare across test runs

Expert Tip: Spreadsheets are excellent for documenting and interpreting the results


Agenda




The Art of Tuning

Art of tuning

• Change only 1 parameter at a time

• Take your time for it

• Always check the same things (even reboot/restart after each run)

• Know what you are measuring

Pimp my ride!


Java Tuning Options

‘Fastest’ JVM depends a little on the application and a lot on the platform

VM heap Garbage Collection:

• default settings good for optimal throughput

• use advanced (jdk4 or later) GC options to maximize latency

Rule of Thumb: On windows platforms, the Sun 1.5.0 JVM is10% to 50% slower than the default IBM 1.4.2 JVM.


Broker Tuning Options

Sources of broker CPU cost

• CPU cost of i/o– Network sockets– Log/Data disks

• Web Service protocol– Web Service security

• Security– Authorization– Message or channel encryption


Broker Tuning Options

Sources of broker disk overhead

• Message recovery log– Persistent message store– Might not be used if other guarantee mechanisms work

• Message data store– Disconnected consumer– Queue save threshold overflow

• Flow to disk overflow• Message storage overhead depends on disk sync behavior

– Explicit file synchronization ensures data retained on crash– Tuning options at disk, filesystem, JVM and Broker levels


Broker Tuning Parameters

Core Resources:• JVM heap size• JVM thread, stack limits• DRA, HTTP and Transaction threads• TCP settings

Message storage:• Queue size and save threshold• Pub/sub buffers• Message log and store

Message management• Encryption• Flow control and flow-to-disk• Dead message queue management

Connections• Mapping to NIC’s• Timeouts• Load balancing

Rule of Thumb: For non-trivial queues, multiply defaultsettings by 10 to 100.


Messaging Tuning Options

Implement optimal QoS for speed versus precision

ESB

QoS

MQ

SettingMessage Loss Events

Duplicate

Msg Events

N/A DiscardableBuffer overflow,

Any crashNever

Best EffortNonPersistent,

DupsOK ack

Broker crash,

Client crashNever

At Least OncePersistent,

DupsOK ackNever Never

Exactly Once Transacted Never Never

Expert Tip: With CAA configured, Best Effort service is equivalentto At Least Once, with substantially lower overhead.

(Based on CAA brokers and fault-tolerant connections)


Messaging Tuning Options

Message transfer overhead is generally fixed Hidden ack messages amenable to tuning:

• AsyncNonPersistent mode decouples ack latency• Transaction Commit allows 1 ack per N messages• DupsOK ack mode allows ‘lazy’ ack from consumer• Pre-Fetch Count allows batched transmit to consumer

ESB Design option: send one multi-part message instead of N individual messages

Producer Broker Consumer

Msg

Msg

…Ack

Msg

Msg

Ack

…

Use message batching to accelerate message streams


Flow Control

Point to Point

SenderSenderSenderSenderPotentialPotentialReceiverReceiver

PotentialPotentialReceiverReceiver



QueueQueueQueueQueue

Flow Control in PtP:• If the Maximum Size of a Queue is reached, Flow Control

is kicking in

• Sender seems to be hanging but is only waiting for space in the Queue

• Flow Control can be switched off and replaced by an Exception in your code.


Flow Control

In Pub/Sub, the working of Flow Control is more difficult to predict• Each Subscriber gets a Buffer (Outgoing Buffer Size)• Sonic will try to only have 1 copy of the message in memory• When the buffer is reaching a threshold, Flow Control is issued.• Threshold depends on message priority

Publish and Subscribe

PublisherPublisherPublisherPublisher TopicTopicTopicTopic

SubscriberSubscriberSubscriberSubscriber



Outgoing Buffer

Flow Control

Publish and Subscribe

PublisherPublisherPublisherPublisher TopicTopicTopicTopic



Pub/Sub

TEXT

Outgoing Buffer

Only a pointer is inserted in the Buffer but the calculated size is the message size

Ack

Ack

Danger !!!Slow Subscriber may result in the following situation

TEXT

TEXTTEXTTEXT

TEXTTEXTTEXT

TEXT TEXT

FLOW CONTROL


Client Tuning Options

Reuse JMS objects to reduce setup cost• Objects with client and broker footprint

– Connection– Session– Sender / Receiver / Publisher / Subscriber

• Coding best practices– Share connections across sessions– Share sessions across producers / consumers

Not across threads!!

Rule of Thumb: Up to 500 queues per Broker and10,000 topics per broker.


ESB Tuning Options

X-scaling: Multiple ListenersY-scaling: Multiple JVMsZ-scaling: Multiple Machines


ESB Tuning Options

BrokerDispatch Outbox

Marshall Msg

Send Msg

…

…Dispatch Outbox Unmarshall Msg

Marshall Msg Call onMessage

Send Msg …

Receive Msg

Unmarshall Msg

Call onMessage

Instantiate Proc

…

Inter-Container Messaging

Intra-Container Messaging

v7.5: better!faster!


ESB Tuning Options

Transformations XML Server BPEL Server Database Service DXSI Service …


In Summary

Know what you’re doing!

If you don’t, get someone in who does!

Use good old plain common sense


For More Information, go to…

PSDN• White paper: Benchmarking

Enterprise Messaging Systems

• Sonic Test Harness

Documentation:• Progress Sonic MQ Performance Tuning

Guide


Questions?


Thank you foryour time

Tuning and Scalability for Sonic

Documents

Transcript of Tuning and Scalability for Sonic