Performance Engineering 1 Prof. Jerry Breecher Software Performance Engineering.
Software Performance Engineering Services
-
Upload
collaborative-consulting -
Category
Technology
-
view
110 -
download
1
description
Transcript of Software Performance Engineering Services
1
Collaborative Consulting: Software Performance Engineering Services
2
Contents and Overview
Value Proposition About Collaborative Services & approach
Performance Engineering’s high impact to Client brand and IT Risk
Mitigation
Collaborative has a rich history of success in
delivering client outcomes
Current and emerging client services
3
Contents and Overview
Value Proposition About Collaborative Services & approach
Performance Engineering’s high impact to Client brand and IT Risk
Mitigation
Collaborative has a rich history of success in
delivering client outcomes
Current and emerging client services
4
Why Collaborative? Our People, Value and Delivery
Our people
Our people are our strongest asset; we not only have the most senior individuals with deep domain expertise, but our IT technical support team is second to none in the industry.
Our strongest asset
At the heart of the organization is a team of committed individuals who are passionate about solving complex issues, and dedicated to providing the highest quality of service possible.
Heart of the organization
Our culture and work process is focused on pragmatism, problem-solving and results-oriented action.
Our culture and work process
We understand the “big picture”, as well as the macro business trends, but we really thrive on the specific mechanics and process of the business.
Our understanding
We bring a comprehensive methodology and approach that allow us to be confident in our work products, be rigorous in our approach, and speed “time-to-value” in all of our projects.
Our comprehensive methodology
We leverage our proven project accelerator tools and methodologies that reduce effort, improve quality and shorten timelines.
Our proven project accelerator tools
Our value Our delivery
5
Our solutions approach
Our flexible delivery model allows us to work with clients in the way that makes the most sense for the situation: as a trusted advisor, in a blended team scenario, or as a dedicated project team.
Flexibility
Enterprise Architecture
Business Technology
Technology strategy
Solution delivery
Business process optimization
Solution delivery
A keen understanding of enterprise
business issues and operations,
with practitioners
across multiple industries.
A discipline aimed at
building the right solution for
each client, based on their specific needs and business
objectives.
Aligning business and technology initiatives enables our clients to achieve results that deliver business value.
6
Services & Delivery Model
Life Sciences • Pharmaceutical • Medical Device
Understand and envision
Architect and design Develop and deploy Support and
maintain
Consulting Services Domestic Solution Center Services
Financial Services • Capital Markets • Insurance
Information Management
• Master Data Management
• Data Quality • Data Integration • Business Intelligence • Data Governance
Application Architecture and
Development • Architecture
Strategy • Application
Development • Systems Integration • Mobile Enablement • Application Support
Software Performance and Quality
• PE Advisory Services • Application Rescue • Performance Testing
& Tuning • Quality Assurance
Strategy • Software Quality
Testing
7
Contents and Overview
Value Proposition About Collaborative Services & approach
Performance Engineering’s high impact to Client brand and IT Risk
Mitigation
Collaborative has a rich history of success in
delivering client outcomes
Current and emerging client services
8
Performance Engineering Value
PE is the set of roles, skills, activities, practices, tools, and deliverables to be applied at each phase of the SDLC to ensure that a solution will be designed, implemented, and operationally supported
to meet the non-functional requirements defined for the solution.
What is PE?
• User experience: PE task and activities in the SDLC promote business and IT alignment through reliable, scalable, and performing systems. Performance is designed in early.
• Increased revenue: Ensure an organizations’ systems and applications are able to manage complexity, identify risk, and enable seamless and rapid business growth for increased revenue.
• Effective use and return on system resources. Capacity planning trending and forecasting can help manage IT system investments, eliminate budget surprises.
• Manage risk of technology adoption for business value. Performance evaluation and Proof-of-concept testing can help identify business risk.
Value delivered to the business
9
Benefits and Case for Action
We deliver high value, impactful services, resulting in business stability and client and customer satisfaction.
• Manage risk and complexity.
• Ensure service level agreement criteria is met and exceeded.
• Ensure application scales with business growth.
• Identify the system’s performance limits.
• Ensure the physical infrastructure
supports the business plan.
• There is a need for application service assurance and reliability.
• High availability and performance from enterprise applications are needed to achieve return on investment.
• There are multiple configuration options for distributed applications.
• Multiple integration points in a typical system installation can lead to performance and scalability issues.
Key Benefits Case for Action
10
The SPE Body of Knowledge – Key differentiator Kn
owle
dge
area
s
Enterprise Software Performance Engineering
Planning, coordination, information sharing, and control
SDLC & architecture
(SA)
Application performance management
(APM)
Problem detection
and resolution
(PDR)
Underlying competencies and roles for delivering
Techniques supporting the knowledge areas
Performance validation and testing
(PVT)
Capacity planning (CP)
Collaborative is the Performance Engineering Expert
11
PE: Methodology
PE encompasses the set of roles, skills, activities, practices, tools, and deliverables applied at every phase of the SDLC which ensures that a solution
will be designed, implemented, and operationally supported to meet the non-functional requirements defined for the solution.
Step-by-step framework for better software engineering
Collaborative’s PE Methodology is a step-by-step framework designed to ensure an organizations’ systems and applications are able to enhance competitive advantage, manage complexity, identify risk, and enable seamless and rapid business growth.
Focused on business objectives In short, our methodology helps companies determine if their technology can accommodate its business objectives.
Based on unique methodology
Our PE Methodology is unique in that it is applicable throughout a system’s entire life cycle. It also: • Describes the detailed approach to designing, building, executing and
interpreting performance benchmarks • Shows how to analyze performance of a system in production that falls
short of service level objectives
12
Contents and Overview
Value Proposition About Collaborative Services & approach
Performance Engineering’s high impact to Client brand and IT Risk
Mitigation
Collaborative has a rich history of success in
delivering client outcomes
Current and emerging client services
13
Contents and Overview
Value Proposition About Collaborative Services & approach
Performance Engineering’s high impact to Client brand and IT Risk
Mitigation
Collaborative has a rich history of success in
delivering client outcomes
Current and emerging client services
14
Performance Engineering Services
Software Performance Engineering
Collaborative’s performance
engineering services drive improvements
in information technology and
ensure scalability and stability for
existing and planned systems, including improving overall quality, reliability, capacity, security, and performance.
Advisory Services Is my current team delivering the right services? Organizational and operational planning, strategy and design services for transforming internal capabilities. Establishing PE Center of Excellence (COE)
Application and readiness
assessment
Is my application ready for the new volumes? or is the application response time getting worse over time. The goal is to technically assess the application to understand the risks to performance, scalability and the transaction response times.
Production Performance Rescue
A proven approach to production performance and stability issues, reviewing the end to end transaction. Minimizing disruption and quickly restoring functionality and capability.
Application & Service Performance Management
Are my key business transactions being monitored, do I know of slowdowns before my customers? Optimizing the performance of specific applications and services within a system. Define and implement an APM strategy. Identify key business transactions for monitoring. Measurement, and trending.
Technology optimization and
tuning
Is my technical architecture delivering the best end user response time and transaction throughput? Review technical components, includes hands on testing and measurement.
Benchmarks and Performance testing
Application performance and scalability test design and execution. Performance test lab definition and testing tool selection. We use local and remote team for test execution.
15
1. Advisory Services
• Review the current state of the team; review services provided, interaction model with development and operations teams, and work products
• Review the services provided to the organization • Determine the value provided • Understand what the organization needs from the PE team • Review the Application Selection Criteria for Performance Engineering and
performance testing • Engagement Model with the development teams, the operations teams, the
PMO process and the governance process.
Team and Organizational Assessment
• Services roadmap • Value proposition for the team • Organization structure, roles and responsibilities
Outcome
16
1. Advisory Services: Organizational Alignment Framework for PE
Communication & Education Programs
Communication & Education Programs
Communication & Education Programs
Communication & Education Programs
Competencies Provide PE processes, skills, and tools,
to excel in the new PE environment.
Resources Align resource allocation
changes with updates to role definition and responsibilities.
Assess the alignment of ongoing initiatives with the
business performance vision.
Business Vision Manage underlying policies and
conditions that influence behavior.
Culture Management
Ensure that the plan is successful by collaborating with key sponsors and leaders to
champion the initiative.
Sponsorship and Change Leadership
Assess the change to existing operations environment and business areas with regard to
workflow interactions, tools, resource allocation, responsibilities, and competencies.
Change Impact Assessment
17
1. Advisory Services Scope
Engineering Group Quality Assurance/ Performance Engineering Group
Operations
Platform/ Environment
Validation
Performance Benchmarking
Performance Regression
Performance Integration
Production Performance Monitoring
Configuration/ Customization
Design Implementation Application Testing Production
Support
Performance Engineering Advisory Scope
Define service level
objectives
Development Process
18
2. Application Readiness Assessment
Phase 1 Inspection of existing
application and technical architecture.
Significant risk
Phase 2 Risk evaluation. Code
inspection and performance testing.
Assessment report, recommendations, observations, and next steps.
Call to action: • Current application is slowing down, user response time is degrading. • Anticipating significant change in business volume. Will the current system scale? • Moving to a new technology platform or acquiring a new application. Will it support my business
volumes?
Yes
No
19
2. Application Readiness Assessment
Methodology is iterative and tactical
Review infrastructure & architecture
Define business activity profiles & service levels
• Identify risk areas • Review configuration
settings, topology, & sizing • Define points of
measurement
• Types & numbers of users • Business activities &
frequencies
Iterate testing & tuning
• Test data generation
• Create test scripts • User & transaction
profiles • Infrastructure
configuration
Design & build tests
Developing a clear picture of system capabilities helps companies form a strategy to enable seamless growth.
• Verify accurate inventories of applications and systems, and capabilities and limitations. • Determine whether certain applications need to be replaced, retired or upgraded. • Assess and improve performance engineering process and execution
20
3. Production Performance Rescue
Minimize disruption and quickly restore functionality and capability.
• Accelerated analysis and test design, including system architecture review
• Accelerated iteration of test execution and application and configuration changes
• Accelerated implementation of performance improvements.
• Mitigation/remediation of issue(s)
• Recommendation and planning of tactical short-term improvements and strategic longer-term enhancements
21
4. Application & Service Performance Management Align business and IT through proactive management of service levels
across IT and multiple lines of business.
Performance management data
repository
Performance repository administration:
• Create forecasts for new applications
• Modify existing forecasts
Performance data sources by: Tier, Application, Server
Platform Application Tier
Performance dashboard views: Canned Reports: J F M A M J J F M A M J J F M A M J
Legend: Performance Details Captured Performance Management Data Published
Combine monitoring and management tools with proven Performance Engineering experience and methodologies to link people, information and decision-making
processes throughout the organization.
22
4. APM: Watch Every Transaction for Every User
WEB
MQ/ESB
DCOM
CORBA
SERVER
APP SVR
PROXY
LDAP
Web Services
Datacenter
23
5. Technology Optimization and Tuning
• Identify business transactions and supporting system components
• Evaluate each tier in the technical architecture
• Leverage end to end transaction monitoring
• Use deep dive tool for code analysis and database analysis
Building a better user experience; and/or reduce resource consumption
• Improved and consistent user experience
• Improved scalability (process more transactions with the same system)
• Improved stability
• Transfer for knowledge to clients’ technical team
Outcome
24
6. Benchmarks and Performance Testing
What is changing? Where does it need to be validated? What (all) is being measured?
The project: Why are we testing and what are the success criteria?
Prerequisites:
Business goals and non-functional requirements are defined
Workload and scenarios defined
Environment defined (where are we testing)
Application in a stable state
Configuration management and release management ready
Core performance team identified
Supporting team committed
25
6. Benchmarks and Performance Testing
• Define a custom benchmark for clients, for a specific workload and technical architecture
• Application Baseline
• Stress and breakpoint
• Duration
• Peak load
• Component(s)
• Failover and Failback (baseline or with load)
• Antagonistic (degraded service times)
• Performance testing and code changes (iteration)
Performance Testing Categories
26
6. Performance test delivery – remote model
Distributed test execution team
Test execution options
• Remote team uses VPM to access client facility and execute tests
• Remote team use Cloud services to delivery load to client application
VPN
Cloud
Client facility Collaborative’s remote testing facility
Our performance engineer is onsite
Test execution team
27
• CASE • STUDY Healthcare Prescription Benefits Manager (PBM)
Approach
Objective •Major business impact due to production disruptions within customer’s PBM System. This resulted in major downtime of the system.
•No Performance Testing (PT) environment being used. Limited PT execution. Lack of disciple around PT and lack of coverage. No Performance Engineering occurring!
•Client looking to reduce risk and ensure that there will be no disruption in the future, and that all components of PBM System will scale and perform for all future releases.
Results
•In an effort to reduce risk and ensure a higher level of performance testing process, we were initially engaged to assess the PT and PE maturity across process, organization, roles and tools to identify and recommend areas where tactical changes can be applied to implement PE across the platform. •Following the assessment phase, presented roadmap including staffing and cost models to executive leadership. •Planned for partial FTEs and On Demand resources with a model to move towards more FTEs over time. •Focus has been on the entire Performance Engineering process across the SDLC, from design review, test execution, results review and analysis, and troubleshooting.
•Client has improved the ability to execute disciplined performance tests and to increase the number of meaningful tests. Client has been able to execute more tests for each release. •By improving the test preparation process and test review process, Client is able to reduce the number of retests due to improper configurations and settings. •Within first 6 months, Client has identified multiple significant issues, prior to Production deployment, resulting in immediate ROI, covering the cost of the group for the first two years. •Visibility into the performance of the application and infrastructure has been vastly increased by improving the monitoring process and increasing the number and type of metrics being collected. •Client is better positioned to manage risk and complexity across their PBM System.
28
• CASE • STUDY Financial Services Pension Provider
Approach
Objective
Client’s Online Group sought to proactively improve the performance of its systems and demonstrate previously achieved improvements. Client wants guarantees that the applications will perform without disruption, and given the frequency of enhancements and modifications being implemented for the platform, the organization wants to become more proactive and better understand the effect each enhancement has on the stability of the system before being deployed into production.
Results
•Phase 1: Assessment effort only. Began the strategic process of developing the PE process definition and PE roadmap, and PE process implementation. The assessment phase is a review of the key applications to assess the technical challenges of performance engineering each system. Within Phase 1 this included:
•Client secure account web application •Client Public site.
•Phase 2 began the tactical implementation of the roadmap, based on business priority.
We delivered an assessment including findings and recommendations, and developed a Roadmap including several key initiatives. Notably this included the following work streams:
•Baseline Client secure account web application and the Client Public site •PE test environment analysis •Client secure account web application and Public Site Production Performance Tuning •Organization Alignment •Non-production environment performance monitoring •Performance test data management •Application assessment – Administrative web application
29
• CASE • STUDY Retail
Approach
Objective • “Next generation” pharmacy system to replace legacy application is a highly complex system consisting of multiple
technologies and infrastructure components (2.6M lines of code; 25,000 function points). • Two PT environments leveraged to accelerate performance testing. Additional complexities and risks are also being identified
(e.g., system stability, level of code completeness) while executing performance testing. Concerns raised over the viability of aggressive PT timeframe.
• The company looking to reduce risk and ensure that the new solution will scale and perform for pilot and planned rollout to chain.
Results
•In an effort to reduce risk and ensure a higher level of performance testing process, Collaborative was initially engaged to assess the PT plan, process, organization, roles and tools to identify and recommend areas where tactical changes can be applied to improve the overall PT process.
•Following the assessment phase, Collaborative was engaged to manage and execute the large-scale performance testing. Current team includes a blend of resources from Collaborative, this company’s off-shore development partner based in India, and various on-shore support staff. Also working with key tech vendors (Oracle, IBM, EMC, etc).
•With aggressive program timelines, the team is running multiple shifts in a 5x24 mode.
•Focus has been on the entire testing process, from design review, test execution, results review and analysis, and troubleshooting.
•Collaborative technical architects are also working with the infrastructure team and the off-shore application development team to provide recommendations and resolve performance problems with the application and in the environment.
•Collaborative has improved the ability to execute performance tests and to increase the number of meaningful tests. By streamlining the testing process, Collaborative has been able to execute more tests for each shift during a 24 hour period.
•By improving the test preparation process and test review process, we were able to reduce the number of retests due to improper configurations and settings.
•Feedback provided to the test design process has enabled the test execution team to combine test cases into one test case. This also allowed the team to be more efficient with test execution.
•Visibility into the performance of the application and infrastructure has been vastly increased by improving the monitoring process and increasing the number and type of metrics being collected across all the tiers and components.
•The client is better positioned to manage their risk and complexity
30
Thank You