Rhonda Software Quality Assurance Services

Rhonda Software Quality Assurance (QA) Services

Configurable software quality assurance process: From requirements acquisition through testing and configuration management

to release and maintenance activities

Quality Assurance – to Serve Your Needs

“Quality is never an accident; it is always the result of intelligent effort.”

John Ruskin

“Software never was perfect and won’t get perfect. But is that a license

to create garbage? The missing ingredient is our reluctance to quantify

quality.” Boris Beizer

“Testing is an infinite process of comparing the invisible to the

ambiguous in order to avoid the unthinkable happening to the

anonymous.” James Bach

2© 2013 Rhonda Software: www.rhondasoftware.com

http://www.rhondasoftware.com/


TECHN

OLOGYflexible & powerful

tools

PROCESSlightweight & mature practices

PEOPLE

talented & skilled

professionals


QUALITYBUILDING BLOCKS



There is no secret that it is impossible to develop software product

without any defects in it. And the more complex product you create –

the more errors it will demand.

What?Quality Assurance (QA), as we see, is the process that should be part of each

and every stage of SW product development . It is a well chosen number of best

practices taken from CMM/CMMI and our 18-year experience to ensure high

quality of developed SW product. In the middle of QA is a set of testing activities

integrated into entire production cycle.

Why?Because escaped defects (errors left in released product) may cost too much. Too much money to fix

and too much reputational damage to live with.

When?The sooner the better. Everybody knows that the later a defect is found, the higher its impact and

price will be. However, it is never too late to think about quality – our QA process can be easily

adopted and integrated into any customer process on any project stage.


Better Safe Than Sorry

QUALITY

CONTROL

EXECUTE

PLAN



Every year, software errors cause massive amounts of problems all over the world. A 2002 study

commissioned by the National Institute of Standards and Technology found that software bugs cost

the US economy $59.5 billion every year (imagine the global costs). The study estimated that more

than a third of that amount, $22.2 billion, could be eliminated by improved testing.

Here is a couple of relatively fresh examples:

March, 2011. Japanese bank Mizuho Financial Group's clients

experienced a software glitch that collapsed its ATM network and

internet banking systems. The result was $1.5 billion in salary

payment delays and $9 billion in unprocessed payments.

August, 2012. In less than an hour, Knight Capital's computers

executed a series of automatic orders that were supposed to be

spread out over a period of days.

Millions of shares changed hands. The resulting loss, which was

nearly four times the company's 2011 profit, crippled the firm and

brought it to the edge of bankruptcy. Knight only survived after a

group of investors swooped in with a $440 million lifeline.


The Cost of Error

http://online.wsj.com/article/SB10001424052748704261504576205411460057114.html

http://money.cnn.com/2012/08/09/technology/knight-expensive-computer-bug/


Integrated QA Process


REQUIREMENTSMANAGEMENT

Identified requirements

Specifications

CODEDEVELOPMENT

Unit testing

Code best practices

Reviews

QUALITYASSURANCEPROJECT

MANAGEMENTProduct/Feature management

Task tracking

Resources

TESTPREPARATION

Requirements traceability

Design

Automation

TESTMANAGEMENT

Scope/Planning

Monitoring

Reporting

TESTEXECUTION

Test passesEscaped defects analysis

Defect tracking

CONFIGURATIONMANAGEMENT

Version control

Continuous integration

Release management


As any other project activity, QA should be carefully

considered in planning stage to define quality goals &

criteria, identify applicable QA/testing procedures and

pre-assign necessary resources (including personnel,

hardware and infrastructure environment).

In the large scale projects, there is an additional practice that

have major impact on resulting product quality.

Product/Feature Management• When there is an entire line of products built on a common code base• When there is incremental product evolution where new functionality is released in

new product versions, especially only in select product tiers• When there are a lot of features: to add something new, to change existing behavior, to

incorporate functionality from other product in the line, to exclude obsolete function,

and so on...

Products and features need to be managed very neatly. Especially since scope and

essence of on-project QA activity is directly connected with selected product feature set.

PROJECTMANAGEMENT

Product/Feature management

Task tracking

Resources

QA Core Components


QA Dependencies - Project Management


REQUIREMENTSMANAGEMENT

Identified requirements

Specifications

QA Core Components

Requirements, as customer / user expectations regarding

developed SW product features and capabilities always exist

on a project in some form… even just as a set of ideas in

a head of a project manager.

For the quality sake, it is better to keep product requirements

in a well organized way.

Requirements Specifications• To identify all important aspects of functionality and receive customer’s confirmation• To define restrictions and exceptions avoiding a case when assumed, but

undocumented behavior was never implemented• To establish common understanding across the entire project team• To provide basis for project estimates, code development and validation processes• To be a starting point for effective communication and change management during

product evolution process

Thoroughly identified software requirements, organized as a document may become one

of the major instruments during testing process since Requirements Coverage validation

is the powerful tool to verify that a developed product fulfills all given expectations.


QA Dependencies - Requirements Management


Same as Requirements, Configuration Management (CM)

is always present on any software development project,

though sometimes only as a single code version without

any history.

Needless to say that a well-organized tool-based CM is vital

for any project that is more complex then “Hello world!” app.

Version Control & Continuous Integration Environment• If you have a distributed team simultaneously working on the same code base• If you need to keep your code baseline free from breakages introduced by bug fixing• If you want to know code version where new error appeared for the first time, and to

identify what change had introduced it• If you value the possibility to test isolated code deltas knowing exact scope of changes• If you realize all advantages of automated testing on each and every code version

published

One cannot overestimate the impact of mature CM on resulting product quality, especially

when appropriate tools are selected and efficient CM practices are used by the entire

team to store on-project deliverables and track features, tasks and defects found.

CONFIGURATIONMANAGEMENT

Version control

Continuous integration

Release management

QA Core Components


QA Dependencies - Configuration Management


There are a lot of aspects in a mature Code Development

process that will influence product quality either directly

or indirectly.

Besides personal skills of talented developers, it is always

helpful to apply unified code production process to support

overall code quality during implementation phase.

Unit Testing, Coding Best Practices & Code Reviews• For unified coding style suitable for further product maintenance and updates without

single team member lock-in• For earliest breakage detection and the most cost effective bug fix• For test-oriented code design, especially helpful in embedded and remote testing• For precise error localization and minimal impact on dependent product components• For knowledge sharing among the development team members and personal skill

improvements

Well-established and regularly applied best code development practices allow to introduce

more effective product testing, since the test team can concentrate on behavioral defects

while architectural weaknesses and logical bugs will be filtered out on early stages.

CODEDEVELOPMENT

Unit testing

Code best practices

Reviews

QA Core Components


QA Dependencies – Code Development


Testing is the core process among all

other QA activities.

As a product matures, it proceeds

through test subprocesses and

activities until it is presented to

customers for their evaluation. Each test

activity adds a unique value to find defects as early

as possible in the product development life cycle.

Testing is implemented in separate and/or

overlapped test phases/types:

• Testing on different product levels:

unit, feature, features interaction, system;• Testing during product maintenance:

smoke, sanity, regression;• Testing of special aspects:

bring-up, stress, stability, performance, interoperability, compliance;

TESTPREPARATION


Design

Automation

TESTMANAGEMENT

Scope/Planning

Monitoring

Reporting

TESTEXECUTION


Defect tracking

QA Core Components


Testing Process Components


Testing is a complex set of many different test activities closely

related to other project processes. All of them require to be

planned / monitored (and re-planned during the project life, if

it is required) thoroughly and competently.

The Test Management subprocess is performed from the

very beginning through the end of a project to achieve goals of improved

product quality.

Major Objects of Test Management’s Attention

• Testing Scope / Strategy• Approaches and Constraints• Features / Components / Functional Areas to be or not to be Tested• Test Item Pass / Fail and Test Cycle Suspension / Resumption Criteria• Test Tasks and Deliverables, Environmental Needs• Responsibilities, Staffing and Training Needs, Collaboration with other project groups• Schedule, Risk and Contingencies• Reporting and Quality metrics

QA Core Components


Test Management

TESTMANAGEMENT

Scope/Planning

Monitoring

Reporting


Test Preparation subprocess defines tests and

test environment required for each test phase/type in order

to fully verify and validate a product according to test plans.

Basic Test Preparation Activities• Analysis of functional requirements, specifications and

functional areas' principles of work• Development of Requirements Traceability Matrix (RTM)• Design and development of the test cases for all planned phases/types. The following

approaches and techniques are used to ensure effective and optimal test suites:• Various test approaches (Black-box, White-box)• Effective test techniques (Functionality, Equivalence classes, Boundary, Negative/Positive,

Combinatorial, Model-based, Etc.)• Development of additional tools/utilities and Test automation• Development and setup of specific test environment (quite often it requires a very

complex hardware and software solution which in turn requires very high engineering skills from test engineers)

Obviously, test products themselves must have good quality and thus all project test

deliverables undergo a careful review and validation process.

QA Core Components


Test Preparation

TESTPREPARATION


Design

Automation


The goal of the Test Execution subprocess is straightforward –

apply test definitions elaborated during the Test Preparation

phase to the software products.

Test Execution activities are either scheduled according to

defined test plans or initiated on demand on an event-driven

basis.

Usual Test Execution Activities

• Preparation of developed test environment• Execution of test cases• Deep analysis of execution results• Collecting/Recording of test execution results and failures found • Support development team efforts to reproduce and to analyze failures, as well as

validation of fixes• Analysis of escaped defects• Update test cases/test environment according to results of test execution• Update test cases/test environment according to results of Escaped Defects Analysis

QA Core Components


Test Execution

TESTEXECUTION


Defect tracking


QA Core Components

Test automation, especially within Continuous

Integration (CI) development cycle, has proven to be

the most cost-effective approach while increasing

the testing speed and quality, eliminating human

errors and reducing testing expenses.

It is a useful instrument in a long-lead product

evolution and maintenance. In some cases

test automation is the only method when test

engineers can keep pace with software

developers and verify each software version with

the same reliability and quality level as previous ones.

However, it is necessary to understand that automated testing

is powerful aid to improving the return on the testing investment when used wisely. Only

competent project management, test preparation, including requirements analysis,

automated test tool selection/development, and test scenarios development can really

increase testing efficiency.


QA & AutomationAUTO-TEST MODE

L

AUTO-RUN

TEST RESUL

TS LOG

ANALYSIS

UPDATES & BUG

FIXES

CONTINUOUSINTEGRATION


QA Core Components

As per our experience, Testing is performed in a more

effective manner when suitable tools are used on the

project. They allow to keep all QA-specific information

in a definite common accessible places, perform

planning / monitoring / reporting more effectively and

collect valuable QA metrics.

Different types of QA management assets and tools used are:

• Project / Test Plans, Activities and Tasks• Environment and Hardware Resources• Requirements• Requirements Traceability (test cases vs. requirements)• Test Cases• Product Builds & Code Versions• Test Results• Failures Found (Defect tracking)• Project / Test processes progress metrics


QA Assets & Tools


Customizable QA

Here at Rhonda Software we know that each company

has its own SW development process with unique

characteristics and dedicated practices / activities

adapted for specific project needs.

There is no point to change things that are already good

enough and work as a well-oiled machine. Instead let’s

incorporate beneficial QA practices and improvements inline with yours development

framework where they give most noticeable effect.

Our goal is the lightweight and effective QA process and seamless integration of improvements into customers’ existing SW development process

Our customizable QA services are:• Core testing activities – test preparation, execution, management• Test automation• Requirements elicitation and management• CM and Continuous Integration• Introducing effective practices for using Task/defect tracking tools


Seamless Integration into Customer Process


Rhonda Software QA Services

Good technical skills helped our SW test

engineers to prepare a multi-network environment

and successfully provide testing services to one

of our customers to test dual modes mobile

products which support cdma2000, 1xEV-DO

and Wi-Fi networks. This very complicated and

labor-consuming task was done in a short period

of time without any external trainings or help

available to our team.

CDMA/Wi-Fi lab was successfully installed by

our engineers based on Agilent 8960 Series 10 Wireless Communications Test Sets and

Linksys Wi-Fi routers.

This test lab has since been used to verify mobile applications that require CDMA and

VoIP voice calls (including inter-networks calls), MO and MT SMS over CDMA and Wi-Fi

networks, 1xEVDO data services and data over Wi-Fi (browser, audio/video streaming,

Java applications), CDMA and Wi-Fi handoffs, etc.


Case Study – Multi-network Test Environment



These are some of CV systems developed by Rhonda:• myAudience demographics solutions• Traffic Lights monitoring system• Face Detector IP core for FPGA

In order to assure the quality of system we used different levels of testing:• Local modules unit-tests which can also be run on CI

server (Jenkins)• Functional testing in user scenarios• Web-portal UI acceptance auto testing (using Codeception framework)• End-to-end product auto-testing (scenarios start from device with camera and run till calculated

data is returned via API from web-portal)

In addition to usual testing practices, development of CV products required additional specific testing tasks.• Evaluate system performance metrics: recognition rate, classification rate, detection rate, counting

precision• Support optimal parameters calibration:

• CV algorithms often depend on multiple parameters that can not be easily found• One way to find good parameters is to run the system on different combinations, measure quality and

select the best ones


Case Study – Computer Vision (CV) Testing

http://www.myaudience.com/


One of current Rhonda Software clients provides

all-in-one cloud-based solution for disaster recovery,

backup and data storage. And it is a quite challenging

task to sustain best quality, performance and reliability

during continuous evolution and development cycle.

Specific QA activities are applied to ensure continuous defects free production

• Iteration-based Scrum agile process• Test-driven development approach with JUnit, RUnit unit tests• Continuous Integration environment with automated testing – functional, integration

and performance for each component• Additional UI auto-tests for Web-console user interaction component• Two-step deployment process after successful integration / auto-test:

• Mandatory staging proof cycle that includes complementary feature / bug fix & regression

testing in production-like test environment• Post deployment regression testing on production servers for escaped defects control

• Continuous system health remote monitoring



Case Study – QA in Cloud-based SaaS



There is no doubt that all aforementioned QA practices and

processes are very helpful and wholesome. However, there is

one more very (maybe the most) important QA process

component – our QA Team.

Practical skills and technical background allowed our QA

engineers to conduct testing on most modern and complex

equipment (especially in the telecom domain) on the customers’

side and to install and setup own test laboratories.

• Strong communication skill set proven by interaction experience with many of our partners around the world

• All engineers have Master’s degrees in Computer Science• Successful on-site testing to support testing process

whenever and wherever needed. Many of our QA engineers visited customers’ facilities in the USA, Great Britain, Japan, Italy, Germany, France, Israel, South Korea and Taiwan

• Deep knowledge of different platforms, programming languages and technologies• Experience in custom test tools and frameworks development, allowing to complete

sophisticated testing tasks


QA Team – Highlights and Areas of Expertise EMBEDDED

NETWORKINGMULTIMEDIACOMPUTER VISION

DRIVERS

PROTOCOLSUSER-LEVEL APPS



Our QA engineers have participated in numerous projects for different customers, for

example:

Motorola• Android Kernel/BSP• Multimedia & DRM• Email client• i-mode• CDMA/WiFi DMH• MDS Test process improvement

CSR (former Zoran)• Embedded solutions for digital camera processors• Image processing (still capture, video recording)• Networking

Exalt• Microwave Digital Backhaul radios

Mentor Graphics (Embedded Group)• Testing of Linux BSP ports on new silicon


QA Team – Experience & Customers



Rhonda Software as QA service provider deals

with various tools and instruments working on

customer’s projects.

Tools for QA planning and test management• MS Project – Commonly used tool for project and test

activities / tasks planning• JIRA – Tracker tool used to capture and organize issues, assign work, and follow team activity• Rational DOORS – Requirements management solution• Assembla – Set of cloud-based task and issue management tools for software developers• TestRail – Test case management software for QA and development teams• Selenium – Suite of tools to automate web applications for testing purposes across many

platforms• TeamCity – All-in-one pre-integrated solution for Continuous Integration• RSpec – Behavior-driven development (BDD) framework for Ruby programming language• Cucumber – Tool for running automated acceptance tests written in a BDD style• JUnit – Unit testing framework for Java programming language• Mockito – Open source testing framework for Java• EasyMock – Library that provides an easy way to use Mock Objects for given interfaces or classes• TestNG – Testing framework inspired from JUnit and NUnit with new useful features


QA Team – Instruments & Tools Skill

Tasks Tools Job Done

http://office.microsoft.com/en-us/project/

https://www.atlassian.com/software/jira

http://www-03.ibm.com/software/products/en/ratidoor

https://www.assembla.com/

http://www.gurock.com/testrail/

http://www.seleniumhq.org/

http://www.jetbrains.com/teamcity/

http://rspec.info/

http://cukes.info/

http://junit.org/

https://code.google.com/p/mockito/

http://easymock.org/

http://testng.org/doc/


Andrey MischenkoCEO, Rhonda Software

Mobile: +7(423)257-1008

E-mail: [email protected]

Denis PomogaevVP of Technology Innovations

Mobile: +1(224)715-1154E-mail: [email protected]


Thank You!

mailto:[email protected]

mailto:[email protected]


Rhonda Software Quality Assurance Services

Technology

Transcript of Rhonda Software Quality Assurance Services