SOA-36: Tuning and Scalability for Your Sonic ™ Enterprise Messaging Andreas Gies Principal...
-
Upload
gillian-amber-benson -
Category
Documents
-
view
217 -
download
2
Transcript of SOA-36: Tuning and Scalability for Your Sonic ™ Enterprise Messaging Andreas Gies Principal...
SOA-36: Tuning and Scalability for Your Sonic™ Enterprise Messaging
Andreas GiesPrincipal Architect
© 2008 Progress Software Corporation2 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
D I S C L A I M E R
Setting Performance Expectations
System performance is highly dependent on machine, application and product version. Performance levels described here may or may not be achievable in a specific deployment.
Tuning techniques often interact with specific features, operating environment characteristics and load factors. You should test every option carefully and make your own decision regarding the relative advantages of each approach.
D I S C L A I M E R
© 2008 Progress Software Corporation3 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
Agenda
Overview – Typical performance projects Discovery – Defining performance goals Benchmark – Testing performance Tuning – Improving results
© 2008 Progress Software Corporation4 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
Performance Engineering Terms
“System Under Test”
“Test Harness”
“Load” = “Sessions” * “Delivery Rate”
“Latency” = ReceiveTime – SendTime“Test Components”
“ExternalComponents”
“Platform” “System Metric”
R
V
“Variable” =client param,app param,system param
V
V
Tip: Limit scope to those test components thatare critical to performance and under your control
© 2008 Progress Software Corporation5 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
Performance project documents
Proposal• Goals and Scope• Architecture• Scenario definition
Project plan• Details on architecture and scenarios• Test cases• Schedule & logistics
Performance report• Executive summary• Test results• Conclusions
© 2008 Progress Software Corporation6 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
Common Pitfalls
Testing functionality, not performance No model for scalability Tuning decisions not applicable to production Insufficient time to test and tune Lack of Performance Engineering skills Inflexibility in exploring high-performance
architecture options
Rule of Thumb: Schedule a two hour discovery call to clarify customer’s understanding of performance issues.
© 2008 Progress Software Corporation7 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
The Performance Engineering Project
The Project is Goal Driven
Analyze
Test
Tune
For each iteration1. performance vs goal2. identify likeliest area for gain
Startup tasks1. success criteria2. resources & skills3. tool selection
Tip: Schedule daily meetings to share results andreprioritize test plan.
© 2008 Progress Software Corporation8 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
Performance Project Skills
SOLUTION
IntegrationExpert
TestingExpert
RequirementsExpert
Cos
t/Ben
efit
Load/Distribution
Design Options
© 2008 Progress Software Corporation9 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
Tools for a Messaging Benchmark
Platform
Component
System Under Test
TestHarness
TestConfigurator
TestAnalyzer
© 2008 Progress Software Corporation10 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
Agenda
Overview – Typical performance projects Discovery – Defining performance goals Benchmark – Testing performance Tuning – Improving results
© 2008 Progress Software Corporation11 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
Performance Project Specification
Rule of Thumb: Focus on message loads over 10 msg/sec,and process loads over 10,000 per hour.
Project spec outline: Project scope Goals and procedures Architecture Business scenarios Test cases
Business view• Value• Relevance• Risk
ESB goals• Throughput• Latency• Scalability
Architecture• Messaging• ESB Process• SOA Services
© 2008 Progress Software Corporation12 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
Specification: Platform Architecture
Back OfficeField DMZ
Platform architecture spec For each Machine:
• CPU number, type and speed• Disk / Memory
Network topology & specs
© 2008 Progress Software Corporation13 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
Specification: SOA architecture
PartnerField HQF
ina
nc
e
SF
A
CRM
ESBESB
Tra
ckin
g
Po
S
Partner ESB
SOA architecture spec Brokers and service containers Geographical distribution Scaling model
© 2008 Progress Software Corporation14 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
Specification: Data Architecture
Data Schemas 1…n Data Schemas 2…m
?
Tip: Transform tools vary in efficiency:XSLT – slowest (but most standard)DataXtend® SI much faster and most flexibleCustom service – fastest, but less flexibleData architecture spec
Key transform events Schema complexity Opportunities for caching Hi-load databases
© 2008 Progress Software Corporation15 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
Business Scenario patterns
Async Process
Realtime Feed
Request/Reply
Tip: The easiest way to manage decoupled SOA processes isthe Sonic ESB Itinerary.
© 2008 Progress Software Corporation16 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
Scenario Specification
Scenario spec details: Business process description QoS, availability, dependencies Goals and expected benefits
validate CreditCard Event
make a paymentEvent
Midas BillingSystem
Make Payment with Credit Card
make_payment (Progress Adapter)
CC Validate
Payementech(Mocked Object)
Make_payement API
Client Apps
is Billing System Up? Queue
yes
no
Credit Card SyntaxValidation
1
2
3
4
5
7-n
6
8
9
10
XML validation
7
Tip: The User’s language,The User’s diagram,
The User’s needs
© 2008 Progress Software Corporation17 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
Specifying Test Cases
Test case spec details: Scenario being simulated Load characteristics Basic ESB process model Services to deploy or simulate Variables manipulated Data measured
Inputs• Message• Protocol• QoS
ESB process• Service impl• Distribution• Params
Results• Throughput• Latency• Validation
Tip: Avoid application details; focus on performance.
© 2008 Progress Software Corporation18 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
Test Variables
Tip: External JMS client variables are easily managedwith the Test Harness.
Sonic ESB Service config Endpoints / Connections Process design XML operations
SonicMQ®
Connection / session Quality of Service Interaction mode Message def
Load Client sessions Messages / second Message size
Application Interaction pattern Service overhead Database interaction
© 2008 Progress Software Corporation19 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
Agenda
Overview – Typical performance projects Discovery – Defining performance goals Benchmark – Testing performance Tuning – Improving results
© 2008 Progress Software Corporation20 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
Configuring the ESB
Upload services• prototype services
• simulation services
Upload processes• test cases
• simulated external services
Other resources• external web services
• databases
© 2008 Progress Software Corporation21 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
Provisioning Brokers and Containers
Message brokers CAA Cluster / DRA ESB Containers & Connections
Tip: For each client connection scenario define a namedJNDI Connection Factory and document its characteristics.
© 2008 Progress Software Corporation22 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
Client Session Configuration
Rule of Thumb: Create one connection for every 10 to 20 sessions
Test HarnessQoS
Msg genreports
ESB
Reply
© 2008 Progress Software Corporation23 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
Configuring Client Quality of Service (QoS)
HTTP and Web service clients JMS Client
• Send mode• Ack mode• Flow control
ESB service JMS options Broker disk overhead
Tip: Best compromise is at least once QoS based on CAA, Persistent, Async disk and DUPS_OK.
© 2008 Progress Software Corporation24 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
Generating Message Content
StatusRequest
priority=2
org=1
Cust Info
gen random int
gen sample xml
Addr svc lookup
transform rule
Tip: Collect sample messages early on, to ensure thatrouting rules and lookups run as expected.
© 2008 Progress Software Corporation25 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
Simulating Clients with Test Harness
System Under Test
TestHarness
Connection
Broker
JNDI
MsgPool
Reply
Request
MessageGeneration
© 2008 Progress Software Corporation26 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
Running Performance Test Iterations
Logistics• multi-client• test intervals• Warm-up / cool-down
Data collection Repeatability
efficiency
relevance
accuracy
Rule of Thumb: Start with 10-20 intervals of 30 secs; if steady state is soonreached, reduce to 10 intervals of 10 seconds, plus warmup.
© 2008 Progress Software Corporation27 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
The Zen of Benchmarks
Spiral Up Childish curiosity Focus on goals Ensure repeatability Deal with now Big picture Use every resource Prioritize
© 2008 Progress Software Corporation28 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
Agenda
Overview – Typical performance projects Discovery – Defining performance goals Benchmark – Testing performance Tuning – Improving results
© 2008 Progress Software Corporation29 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
Performance Tuning with Sonic ESB®
Platform factors Top Ten Tuning Tips Tuning options
• Broker
• ESB
© 2008 Progress Software Corporation30 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
ESB System Usage: CPU
Sources of CPU cost:• Application code
• XML Parsing
• CPU cost of i/o– Network sockets– Log/Data disks
• Web service protocol– Web Service security
• Security– Authorization– Message or channel encryption
© 2008 Progress Software Corporation31 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
ESB System Usage: Disk
Sources of Disk overhead:• Database services
• Large File / Batch services
• Message recovery log
• Message data store
• Flow to disk overflow
Tip: To estimate best-case message log performance, usethe DiskPerf utility bundled with Test Harness.
© 2008 Progress Software Corporation32 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
ESB System Usage: Network
Sources of Network overhead:• Client requests• ESB Service invocations• ESB Web service callouts• CAA broker replication messages• Admin messages
Network performance limit:• Network bandwidth:
– Network Interface Card (NIC)– Switches, routers and firewalls
• Network load
Tip: To estimate best-case network performance, usethe DiskPerf utility bundled with Test Harness.
© 2008 Progress Software Corporation33 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
Tip #1: Increase sender and listener threadsto make service & client scale
Increase # Listeners for key entry endpoints Add more Service/Process instances Warning: Intra-Container ignores endpoint
• split scalable services into separate containers• turn intra-container messaging off• note: sub-Process is always intra-container
Container1
ServiceX
…
Container2
ServiceX
… …
Tip: Stop increasing Listeners when CPU usagenears maximum acceptable.
© 2008 Progress Software Corporation34 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
Tip #2: Implement optimal QoSto balance speed versus precision
ESBQoS
MQ
SettingMessage Loss Events
Duplicate
Msg Events
N/A DiscardableBuffer overflow,
Any crashMissed Ack, Crash of UoW
Best EffortNonPersistent,
DupsOK ack
Broker crash,
Client crashMissed Ack, Crash of UoW
At Least OncePersistent,
DupsOK ackNever
Missed Ack, Crash of UoW
Exactly Once Transacted Never Never
Tip: With CAA configured, Best Effort service is equivalentto At Least Once, with substantially lower overhead.
(Based on CAA brokers and fault-tolerant connections)
© 2008 Progress Software Corporation35 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
Tip #3: Re-use JMS objectsto reduce setup costs
Objects with client and broker footprint:• Connection• Session• Sender/Receiver/Publisher/Subscriber• Temporary destination
Tuning strategies:• Reuse JMS objects in client code• Share each Connection across sessions• Share Sessions across Producers and Consumers
– but not across JVM Threads• For low-load topics/queues
– Use Anonymous Producer – Use wildcard or multi-topic subscription
Rule of Thumb: Up to 500 queues per Broker and10,000 topics per broker.
© 2008 Progress Software Corporation36 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
Tip #4: Use intra-container service callsto avoid broker hops
BrokerDispatch Outbox
Marshall Msg
Send Msg
…
…Dispatch Outbox Unmarshall Msg
Marshall Msg Call onMessage
Send Msg …
Receive Msg
Unmarshall Msg
Call onMessage
Instantiate Proc
…
Inter-Container Messaging
Intra-Container Messaging
Tip: If you save service resources in sonicfs, the ESBcontainer will dynamically load and cache them.
© 2008 Progress Software Corporation37 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
Tip #5: Use NonPersistentReplicated modeto reduce disk overhead
Normal broker mechanisms require disk sync• Contributes to latency across the board
• Interferes with batching of packets
• Limited bandwidth
Disabling disk sync eliminates this overhead• Send mode NonPersistentReplicated
• Optional broker params to disable entirely
• WARNING: Log-based recovery will lose recent messages
• BUT: CAA failover will not
Rule of Thumb: Network overhead for Replication channel isabout ½ the Persistent Msg load of the broker.
© 2008 Progress Software Corporation38 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
Tip #6: Use XCBR instead of CBRto eliminate Javascript overhead
CBR rules implemented via Javascript• Dynamic change with complex rules
• Very high overhead for runtime engine
XCBR rules extract data fields for comparison• Only simple comparisons supported
• No script engine overhead
• Use message property data key for best effect
Rule of Thumb: Invocation of the Rhino JavaScript engine requiresabout 100 milliseconds CPU time on a typical platform.
© 2008 Progress Software Corporation39 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
Tip #7: Use message batchingto accelerate message streams
Message transfer overhead is generally fixed Hidden ack messages amenable to tuning:
• AsyncNonPersistent mode decouples ack latency• Transaction Commit allows 1 ack per N messages• DupsOK ack mode allows ‘lazy’ ack from consumer• Pre-Fetch Count allows batched transmit to consumer
ESB Design option: send one multi-part message instead of N individual messages
XML transforms and other services handle multi-record data efficiently
Producer Broker Consumer
Msg
Msg
…Ack
Msg
Msg
Ack
…
© 2008 Progress Software Corporation40 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
Tip #8: Minimize XML/SOAP operationsto avoid parsing overhead
Bypass SOAP and Web services processing• Use HTTP Direct Basic instead of SOAP or WS• Risk of invalid XML if source is unreliable
Combine multiple XML parsing steps into one• Save target XPath results as Message props• Also relevant for BPEL correlation ID’s
InputMessage
XMLTransform
CustomJAXB
InputMessage
propY=9propX=1
InputMessage
JAXB ObjMsg part
XCBR
JAXBService
© 2008 Progress Software Corporation41 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
Tip #9: Use high-speed encryptionto reduce security overhead
Default SSL encryption uses old RCA stack• At least 2X slower than more modern options
Change to any JSSE compliant stack:• Modify client DSSL_PROVIDER_CLASS to:
progress.message.net.ssl.jsse.jsseSSLImpl
• Change broker SSL provider from RSA to JSSE
Use more efficient cipher suites:• RSA_With_Null_MD5 is the smallest and fastest
Reduce broker memory overhead by deleting any unused ciphers
© 2008 Progress Software Corporation42 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
Tip #10: Use stream API'sto improve large message performance
SonicMQ Recoverable File Channels• Uses JMS layer to manage large file transfer• Queue-based initiation of transfer• High-speed JMS pipeline for blocks of data• Recovery continues at point interrupted
Sonic ESB open-source Large Message Service• Provides dynamic provisioning• Interacts with ESB processes
SonicStream API (version 7.5 or later)• Topic-based, pipeline into Java Stream api• No recovery
© 2008 Progress Software Corporation43 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
Broker Tuning Parameters
Core Resources:• JVM heap size• JVM thread, stack limits• DRA, HTTP and
Transaction threads• TCP settings
Message storage:• Queue size and save
threshold• Pub/sub buffers• Message log and store
Message management• Encryption• Flow control and flow-
to-disk• Dead message queue
management
Connections• Mapping to NIC’s• Timeouts• Load balancing
Rule of Thumb: For non-trivial queues, multiply defaultsettings by 10 to 100.
© 2008 Progress Software Corporation44 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
ESB Tuning Options
Load balancing and scalability of services:• Number of distributed service instances
• Number of service listener threads
Container Java VM heap size Intra-Container messaging Endpoint and connection parameters
• Same principles as JMS client
Tip: Start with small Java heap and only increase -Xms size if it improves performance.
© 2008 Progress Software Corporation45 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
Progress Software Developers Network:• http://www.psdn.com
• Search: “performance”, “sonic test harness”, “tuning”
SonicMQ Performance Tuning Guide Benchmarking Enterprise Messaging Systems white
paper Sonic Test Harness documentation Progress Professional Services
• Developer training courses
• Sonic Performance Analysis package
Other Performance Engineering Resources
No “Magic Bullet”, but plenty of places you can go for info:
© 2008 Progress Software Corporation46 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
Questions?
© 2008 Progress Software Corporation47 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging
Thank You
© 2008 Progress Software Corporation48 SOA-36: Tuning and Scalability for Your Sonic Enterprise Messaging