Presented to: SOA Brown Bag #12 By: Paul Caron, SWIM T&E Lead Date: December 13, 2011 Federal...

Presented to: SOA Brown Bag #12

By: Paul Caron, SWIM T&E Lead

Date: December 13, 2011

Federal AviationAdministration

System Wide Information Management (SWIM)

Service-Oriented Architecture (SOA) Testing

2Federal AviationAdministration

SOA Testing

December 13, 2011

Agenda

•Traditional Testing

•Migration to SOA

•Conceptual SOA

•Traditional vs. SOA Testing

•SOA Testing Solutions

•Anticipated Challenges


SOA Testing

December 13, 2011

Characteristics of Traditional Testing• In most cases, non-agile but well established proven process• Single-system focused: not dependant on other systems

within your business domain• Tightly coupled• Well-defined interfaces specified in the Interface Control

Document (ICD) and Initial Requirements Document (IRD)• Limited security constraints, if any• Test program is based on the common Verification &

Validation (V&V) test model• Test team only needs to be technically savvy on the system

under test, and not the entire business domain• Test Tool Strategy is needed because systems are usually

furnished with custom tools• Test requirements only apply to the system/subsystem and

interfaces


SOA Testing

December 13, 2011

Stand Alone Server Architecture

• Applications share data and functions on an isolated single machine, no loose coupling

• Each application communicates with others in a point-to-point, one off interface design

• Difficult to upgrade or replace


SOA Testing

December 13, 2011

Client Server Architecture

• Server applications share data and functions on a single machine, no loose coupling

• Each server based application communicates with others in a point-to-point, one-off interface design

• Supports multiple users


SOA Testing

December 13, 2011

Distributed Server Architecture

• Server applications share data and functions across multiple server machines, possible loose coupling

• Each server based application communicates with others in a point-to-point, intranet, or internet design

• Supports multiple users and supplies composite data


SOA Testing

December 13, 2011

Web Services Server Architecture

• Server applications share data and functions across multiple server machines, possible loose coupling

• External data and/or function provided externally

• Each server based application communicates with others in a point-to-point, intranet, or internet design

• Supports multiple users, supplies composite data


SOA Testing

December 13, 2011

SOA

• Data collections and function hosted by internal and external servers loosely coupled

• Collections often communicate using a single interface

• Reuse and upgrades are less challenging

• Multiple remote users


SOA Testing

December 13, 2011

1980 2000 20101990

Test Complexity

Year

Software Complexity Timeline


SOA Testing

December 13, 2011

The Concept of SOAFrom Oasis (www.oasis-open.org), SOA is defined as – “An architectural style whose goal is to

achieve loose coupling among interacting software agents/services.”

SOA enables business flexibility in an interoperable, technology-agnostic manner. It consists of a composite set of business-aligned services that support a flexible and dynamically re-configurable end-to-end business processes realization using interface-based service description.

SOA is a commonly chosen implementation architecture process for today's largely distributed businesses or systems.


SOA Testing

December 13, 2011

Some Benefits of SOA are:

• Reducing integration expense: uses open standards, not expensive proprietary middleware

• Increasing asset reuse: legacy systems can be easily wrapped as services and exposed

• Increasing business agility: services and systems may be rapidly developed, tested, and delivered

• Reduction of business risk

• Loosely coupled: a modified service does not affect other service or client

• Interoperable: platform-neutral


SOA Testing

December 13, 2011

Components that make up SOA

SOA services are intended to be reusable and can be shared within the entire business domain

• Service provider

• Service consumer

• Service registry

• Governance Lead

• Administrator


SOA Testing

December 13, 2011

SOA Characteristics• SOA transport protocols:

– Hypertext Transfer Protocol (HTTP) – HTTP Secure (HTTPS) – Java Message Service (JMS)

• Messaging: – Extensible Markup Language (XML)– Web Services (WS)-Addressing– SOAP

• Interface: – Web Service Description Language

(WSDL)• Quality of Service (QoS):

– Security – Reliable messaging – AuthN / AuthZ – Policies– Etc.

• Governance:– National Airspace System (NAS) Service Registry/Repository

(NSRR)


SOA Testing

December 13, 2011

Common SOA Testing Challenges

• No visibility below the User Interface to isolate errors• Data driven business logic is service embedded• Inaccessible external services• Inability to test components that do not have an available

User Interface • Need for continuous validation of application functionality• Inability to test composite solutions due to limited access

or availability of dependent services and data needed for testing

• Inadequate or incomplete testing, resulting in costly problem identification and debugging once released into production


SOA Testing

December 13, 2011

Common SOA Testing Challenges (cont.)

• Poor collaboration between development and Quality Assurance (QA), with minimal to no reuse of test assets between unit, functional, regression and performance testing

• Security testing needs to intensify as threat grows

• Test team structure will require alignment to business domains (processes) and not to technologies to ensure that agility and speed on SOA testing do not compromise overall service quality

• Organizations will have to develop and maintain “Test Assets” for key services to guarantee performance and security throughout the entire test coverage cycles


SOA Testing

December 13, 2011

Traditional Testing vs. SOA Testing

Similarities:

• Both Traditional and SOA methodologies can utilize the success proven V-Model

• The V-Model enforces testing discipline throughout the project life cycle

• Both can utilize the top down or bottom up test approach


SOA Testing

December 13, 2011

Traditional Testing vs. SOA TestingDifferences:

Traditional SOA

Limited scope (system, subsystem, interfaces) Business-wide, based on client size or subscriptions

Black Box testing White Box testing

Non-agile: follows a rigid process Flexible, parallel development is a common industry practice

Expensive: utilizing proprietary middleware Less expensive: incorporating open source products and reusable services

Testing within the system boundaryTesting involve elements inside and outside of system boundary

A/B level requirement testing Service-component-level testing

Commonly available test tools Customizable, environmentally adaptable SOA based test tools are needed

Platform specific Universal platform

Traditional systems troubleshooting is straight forward and failures are easy to identify

SOA Troubleshooting is more complex: end-to-end testing involving multiple systems


SOA Testing

December 13, 2011

Traditional Testing vs. SOA Testing

Traditional SOA

Only requirements and regulations need to be validated and verified

Governance is involved and industry standards apply (Quality of Availability (QoA) on performance, regulatory policies, business policies, audit policies, and infrastructure policies)

Small scale security testing, usually reserved at the end of test schedule

SOA security testing process requires continuous validation throughout the project cycle

Test data are easily to control and configure. Test data generated and modified at different systems, therefore hard to control (data aggregation)

Creating a fall-back backup configuration for the traditional test bed is simple

Creating a fall-back configuration for entire SOA type distribution system is complex especially when open source products are involved (frequent version updates)

Larger portion of testing occurs after the development phase of the project

SOA testing occurs from the beginning alongside the project business and requirements development

Regression testing occurs only when major functional requirement changes occur or when bugs are fixed (intensive but less frequent)

Regression testing is needed when software updates occurs and only affects the updated service units – more frequent, but lightly scaled

Differences continued:


SOA Testing

December 13, 2011

SOA Testing Solutions

• Service Level Testing (SLT) emphasis

• Security Testing and the project lifecycle

• SOA and NAS experts leverage

• Test Tool Strategy implementation


SOA Testing

December 13, 2011

SLT Emphasis

• SLT Emphasis– Test emphasis at the SLT level over Governance testing,

service-component level testing, integration-level testing, process/orchestration-level testing, system-level testing, and Security testing

• Formal Code Peer reviews – Ensures standards and compliance are met– Identifies potential interoperability, performance, and security

defects and weaknesses

• Continuous Functional, Performance and Security tests against the Services– Requires the use of test tools and/or test harnesses

• Quality Entrance and Exit Criteria for SLT– Ensures service usability


SOA Testing

December 13, 2011

Security Testing and the Project Lifecycle• Security requirements should be established

• Security risk assessments should be performed during the technical design phases

• Technical deliverables should be validated in accordance with the groups security standards

• Security tests should be performed at the service level and not just the delivered integrated system level

• Security audits should be performed and reported on periodically


SOA Testing

December 13, 2011

SOA and NAS Experts Leverage• Subject Matter Experts / Domain Experts

– NAS expertise– SOA domain expertise– Software development expertise

• Certified Resources– Certified individuals– Certified tools

• Software Support– Commercial Off-The-Shelf (COTS) support contracts– Open Source community and committers

• Knowledge Sharing– Content repositories (SWIM Wiki, Knowledge Services Network

(KSN))– Working groups


SOA Testing

December 13, 2011

Test Tool Strategy Implementation

• SOA Test Tools

– iTKO LISA, soapUI

• XML Tools

– Altova XMLSpy, XML Copy Editor

• Integrated Development Environments (IDEs)

– FUSE IDE for Camel, Eclipse IDE, NetBeans IDE, IntelliJ IDEA

• Web Consoles and Browsers

– Web consoles for ActiveMQ, Camel, Oracle AS, Glassfish, etc.

– Fuse HQ, Lightweight Directory Access Protocol (LDAP) browsers, management and monitoring tools


SOA Testing

December 13, 2011

Return On Investment


SOA Testing

December 13, 2011

Business and Financial Metrics

• Efficiencies associated with service reuse

• Integration time savings

• Related opportunity costs

• Cost savings/avoidance

• Reduction in project and maintenance cost

SOA ROI ($) = Cost Savings/Efficiencies Achieved - All SOA-Related Investments

SOA ROI (%) = SOA ROI ($) All SOA-Related Investments

Source: Leo Shuster, who is responsible for National City Bank's SOA


SOA Testing

December 13, 2011

SOA Performance Metrics

• Overall ROI can be projected by per service ROI’s

• Overall ROI can be projected by per service revenue

• Monitor service growth rate/reuse to ensure reuse maximization

• Design to deploy business agility

• Mean time to service change agility measurement

• Reliability mean time to failure

• Service vitality and revenue tracked for 12 months


SOA Testing

December 13, 2011

Anticipated Future Challenges• Services Metadata

– Employing Governance in regards to managing and providing information on how services interact

• Level of Security– Protection against theft of sensitive data and application infrastructure

vulnerabilities • Test Coverage

– Continuous services testing by producers and consumers to ensure expected functionality

• Scalability– Specialized support for persistence, failover, and load balancing

• XML Message Size– Additional bandwidth and resources for parsing/processing

• Increase in Connections – Connection management for server burden relief– Shared services hosting to avoid latency from additional hops


SOA Testing

December 13, 2011

Conclusion

• Traditional Testing has its place

• Migrate to SOA where appropriate

• SOA brings benefits such as loose coupling, code/services reuse, and remote users

• SOA test products are numerous, mature and vary for many different use cases

• SOA challenges have been identified with many modeled solutions available to pursue


SOA Testing

December 13, 2011

Questions and Comments


SOA Testing

December 13, 2011

Back up


SOA Testing

December 13, 2011

Keys to a Successful SOA Solution

• Definition of SOA

• Find Business Partners

• Aim for Critical Mass

• Franchise

Presented to: SOA Brown Bag #12 By: Paul Caron, SWIM T&E Lead Date: December 13, 2011 Federal...

Documents

Transcript of Presented to: SOA Brown Bag #12 By: Paul Caron, SWIM T&E Lead Date: December 13, 2011 Federal...