Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data...
Transcript of Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data...
![Page 1: Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data Integrity Validation Europe 30 March 2017 Chris Wubbolt, QACV Consulting, LLC. Objectives](https://reader031.fdocuments.us/reader031/viewer/2022022604/5b6032887f8b9a7f038c0773/html5/thumbnails/1.jpg)
Approaches for Auditing
Software Vendors
Data Integrity Validation Europe
30 March 2017
Chris Wubbolt, QACV Consulting, LLC
![Page 2: Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data Integrity Validation Europe 30 March 2017 Chris Wubbolt, QACV Consulting, LLC. Objectives](https://reader031.fdocuments.us/reader031/viewer/2022022604/5b6032887f8b9a7f038c0773/html5/thumbnails/2.jpg)
Objectives
www.QACVConsulting.com 2
• Understanding impact of vendor processes on validation
• Review of Agile SDLC processes
• New approaches to auditing software vendors
• Understanding how SDLC and test tools are used by vendors
• How SaaS vendors impact your company’s validation approaches and data integrity controls.
![Page 3: Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data Integrity Validation Europe 30 March 2017 Chris Wubbolt, QACV Consulting, LLC. Objectives](https://reader031.fdocuments.us/reader031/viewer/2022022604/5b6032887f8b9a7f038c0773/html5/thumbnails/3.jpg)
Impact of Vendor Practices on
Validation
www.QACVConsulting.com 3
Validation Plan
User Requirements
Functional Specifications
Configuration Specification
Installation Qualification
System Testing
User Acceptance Testing
Traceability Matrix
Validation Summary Report
Standard Operating Procedures
Internal Validation Vendor
SDLC Deliverables
Software
Internal Validation vs. SaaS-based
![Page 4: Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data Integrity Validation Europe 30 March 2017 Chris Wubbolt, QACV Consulting, LLC. Objectives](https://reader031.fdocuments.us/reader031/viewer/2022022604/5b6032887f8b9a7f038c0773/html5/thumbnails/4.jpg)
Saas-based vs. Internal Validation
Validation Plan
User Requirements
Functional Specifications
Configuration Specification
Installation Qualification
System Testing
User Acceptance Testing
Traceability Matrix
Validation Summary Report
Standard Operating Procedures
SaaS Validation Vendor
SDLC Deliverables
Software
Validation Plan
User Requirements
User Acceptance Testing
Traceability Matrix
Validation Summary Report
Standard Operating Procedures
Functional Specifications
Configuration Specification
Installation Qualification
System Testing
Traceability Matrix
SOPs
Release Management
Quality Agreement
www.QACVConsulting.com 4
![Page 5: Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data Integrity Validation Europe 30 March 2017 Chris Wubbolt, QACV Consulting, LLC. Objectives](https://reader031.fdocuments.us/reader031/viewer/2022022604/5b6032887f8b9a7f038c0773/html5/thumbnails/5.jpg)
Software Vendor Truisms
www.QACVConsulting.com 5
Software vendors develop and maintain
software.
All software vendors are software developers.
“Quality” software development is essential to
the validation of a system.
21 CFR Part 11.10 (a):
Validation of systems to ensure accuracy, reliability,
consistent intended performance, and the ability to
discern invalid or altered records.
![Page 6: Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data Integrity Validation Europe 30 March 2017 Chris Wubbolt, QACV Consulting, LLC. Objectives](https://reader031.fdocuments.us/reader031/viewer/2022022604/5b6032887f8b9a7f038c0773/html5/thumbnails/6.jpg)
Software Quality Truisms
www.QACVConsulting.com 6
Quality cannot be tested into a
system.
Quality must be designed into a
system.
![Page 7: Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data Integrity Validation Europe 30 March 2017 Chris Wubbolt, QACV Consulting, LLC. Objectives](https://reader031.fdocuments.us/reader031/viewer/2022022604/5b6032887f8b9a7f038c0773/html5/thumbnails/7.jpg)
Software Development
www.QACVConsulting.com 7
Software Development Life Cycle (SDLC)
• The set of activities that constitute the processes that are mandatory for the development and maintenance of software.
• The management and support processes that continue throughout the entire life cycle, as well as all aspects of the software life cycle from concept exploration through retirement, are covered.
• Utilization of the processes and their component activities maximizes the benefits to the user when the use of this standard is initiated early in the software life cycle.(1)
(1) IEEE Standard for Developing Software Life Cycle Processes, 1992
![Page 8: Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data Integrity Validation Europe 30 March 2017 Chris Wubbolt, QACV Consulting, LLC. Objectives](https://reader031.fdocuments.us/reader031/viewer/2022022604/5b6032887f8b9a7f038c0773/html5/thumbnails/8.jpg)
SDLC Methodologies
www.QACVConsulting.com 8
Code and Fix
Waterfall
Prototyping
Incremental Development
Spiral
Rapid Application Development
Agile
(Cowboy Coding)
![Page 9: Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data Integrity Validation Europe 30 March 2017 Chris Wubbolt, QACV Consulting, LLC. Objectives](https://reader031.fdocuments.us/reader031/viewer/2022022604/5b6032887f8b9a7f038c0773/html5/thumbnails/9.jpg)
www.QACVConsulting.com 9
Require-ments
Design
Testing (unit,
module, system,
etc.)
Bug Fixes
Config-uration
Manage-ment
SQA Testing
Release Manage-
ment
Mainte-nance(Cus-tomer
Support)
Elements of an SDLC
![Page 10: Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data Integrity Validation Europe 30 March 2017 Chris Wubbolt, QACV Consulting, LLC. Objectives](https://reader031.fdocuments.us/reader031/viewer/2022022604/5b6032887f8b9a7f038c0773/html5/thumbnails/10.jpg)
www.QACVConsulting.com 10
Quality Manual
Document Management
Training Program
Quality Assurance
Supplier Management
CAPAs / Investigations
SDLC Procedures
Customer Support
Vendor Quality System Elements
![Page 11: Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data Integrity Validation Europe 30 March 2017 Chris Wubbolt, QACV Consulting, LLC. Objectives](https://reader031.fdocuments.us/reader031/viewer/2022022604/5b6032887f8b9a7f038c0773/html5/thumbnails/11.jpg)
Requirements
Analysis
Design
Implementation
Verification /
Testing
Operation /
Maintenance
Requirements
Analysis
Design
Implementation
Verification /
Testing
Operation /
Maintenance
www.QACVConsulting.com 11
Waterfall Methodology
![Page 12: Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data Integrity Validation Europe 30 March 2017 Chris Wubbolt, QACV Consulting, LLC. Objectives](https://reader031.fdocuments.us/reader031/viewer/2022022604/5b6032887f8b9a7f038c0773/html5/thumbnails/12.jpg)
www.QACVConsulting.com 12
SDLC – Agile Methodology
![Page 13: Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data Integrity Validation Europe 30 March 2017 Chris Wubbolt, QACV Consulting, LLC. Objectives](https://reader031.fdocuments.us/reader031/viewer/2022022604/5b6032887f8b9a7f038c0773/html5/thumbnails/13.jpg)
SDLC – Agile Methodology
www.QACVConsulting.com 13
![Page 14: Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data Integrity Validation Europe 30 March 2017 Chris Wubbolt, QACV Consulting, LLC. Objectives](https://reader031.fdocuments.us/reader031/viewer/2022022604/5b6032887f8b9a7f038c0773/html5/thumbnails/14.jpg)
www.QACVConsulting.com 14
SDLC – Agile Methodology
• Focus on short iterations of development
• Delivery of minimum viable product within short periods of time (2-3 weeks)
• Collaboration between end user and development team
• Continuous end user involvement is critical
![Page 15: Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data Integrity Validation Europe 30 March 2017 Chris Wubbolt, QACV Consulting, LLC. Objectives](https://reader031.fdocuments.us/reader031/viewer/2022022604/5b6032887f8b9a7f038c0773/html5/thumbnails/15.jpg)
An iterative and incremental agile development framework.
A flexible, holistic strategy where a development team works as a unit to reach a common goal.
Enables teams to self-organize by encouraging physical co-location or close online collaboration and daily face-to-face communication among all team members and disciplines in the project.
www.QACVConsulting.com 15
Agile - Scrum
![Page 16: Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data Integrity Validation Europe 30 March 2017 Chris Wubbolt, QACV Consulting, LLC. Objectives](https://reader031.fdocuments.us/reader031/viewer/2022022604/5b6032887f8b9a7f038c0773/html5/thumbnails/16.jpg)
A key recognition is that during end users can change their minds about the system requirements.
Scrum adopts an approach to deliver quickly and respond to emerging requirements.
Agile - Scrum
www.QACVConsulting.com 16
![Page 17: Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data Integrity Validation Europe 30 March 2017 Chris Wubbolt, QACV Consulting, LLC. Objectives](https://reader031.fdocuments.us/reader031/viewer/2022022604/5b6032887f8b9a7f038c0773/html5/thumbnails/17.jpg)
www.QACVConsulting.com 17
Software Vendor Truisms
All software vendors are software developers.
The software development life cycle
methodology is arguably the most important
process for a software vendor.
Requirements
Backlog
User StoriesDesign/Development
Unit Testing
Code Reviews
Design Documents
SQA Testing Release
Management
![Page 18: Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data Integrity Validation Europe 30 March 2017 Chris Wubbolt, QACV Consulting, LLC. Objectives](https://reader031.fdocuments.us/reader031/viewer/2022022604/5b6032887f8b9a7f038c0773/html5/thumbnails/18.jpg)
www.QACVConsulting.com 18
Why is this important?
Requirements
Backlog
User StoriesDesign/Development
Unit Testing
Code Reviews
Design Documents
SQA Testing Release
Management
1. The vendors SDLC determines the quality of
the software.2. For SaaS vendors, the SDLC documentation
may also be used as validation deliverables.3. The SDLC documentation is likely to be
maintained within vendor SDLC tools.
![Page 19: Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data Integrity Validation Europe 30 March 2017 Chris Wubbolt, QACV Consulting, LLC. Objectives](https://reader031.fdocuments.us/reader031/viewer/2022022604/5b6032887f8b9a7f038c0773/html5/thumbnails/19.jpg)
www.QACVConsulting.com 19
Use of SDLC and Test Tools
Requirements
Backlog
User StoriesDesign/Development SQA Testing Release
Management
Creation and Management of
Requirements & User Stories
Documentation of Unit
Testing, Code Reviews
& Design Documentation
SQA Test
DocumentationOften used as
“validation” tests.
Configuration / Source Code Management
Management of Bugs and Customer Support Tickets
![Page 20: Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data Integrity Validation Europe 30 March 2017 Chris Wubbolt, QACV Consulting, LLC. Objectives](https://reader031.fdocuments.us/reader031/viewer/2022022604/5b6032887f8b9a7f038c0773/html5/thumbnails/20.jpg)
SDLC/Vendor Tools
www.QACVConsulting.com 20
Requirements Management
Source Code Management
Configuration Management
Code Review and Unit Testing
Testing – including automated testing
Issue Management
Customer Support
Document Management
![Page 21: Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data Integrity Validation Europe 30 March 2017 Chris Wubbolt, QACV Consulting, LLC. Objectives](https://reader031.fdocuments.us/reader031/viewer/2022022604/5b6032887f8b9a7f038c0773/html5/thumbnails/21.jpg)
SDLC/Vendor Tools - Examples
www.QACVConsulting.com 21
Test Stuff
Test Track
CoSign
SharePoint
Wiki Pages
Salesforce.com
Team Foundation
Server (TFS)
HP Quality Center
HP Load Runner
Altassian (Jira)
Subversion
![Page 22: Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data Integrity Validation Europe 30 March 2017 Chris Wubbolt, QACV Consulting, LLC. Objectives](https://reader031.fdocuments.us/reader031/viewer/2022022604/5b6032887f8b9a7f038c0773/html5/thumbnails/22.jpg)
SDLC Tools
www.QACVConsulting.com 22
Team Foundation Server (TFS)
• Requirements Management
• Use Cases
• User Stories
• Design
• Code Review
• Unit Testing
• Traceability
• Testing
• Approvals
• Release Management
![Page 23: Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data Integrity Validation Europe 30 March 2017 Chris Wubbolt, QACV Consulting, LLC. Objectives](https://reader031.fdocuments.us/reader031/viewer/2022022604/5b6032887f8b9a7f038c0773/html5/thumbnails/23.jpg)
SDLC Tools – Questions to ask
www.QACVConsulting.com 23
What do the tools do?
Do the tools impact software quality?
Do the vendor’s procedures reflect the use of these tools?
Are the tools controlled, qualified, or validated?
How are the records maintained by the tools managed and controlled?
How are records approved?
![Page 24: Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data Integrity Validation Europe 30 March 2017 Chris Wubbolt, QACV Consulting, LLC. Objectives](https://reader031.fdocuments.us/reader031/viewer/2022022604/5b6032887f8b9a7f038c0773/html5/thumbnails/24.jpg)
SDLC Tools – What can go wrong?
www.QACVConsulting.com 24
Issue Management
• Vendor used a cloud “hosted” version of Jira, which was used for issue management and change control.
• The license was not renewed and all records were lost.
Electronic Approval
• Vendor used a local implementation of CoSign for approval of records.
• When license expired the electronic signatures applied previously could not be validated.
www.QACVConsulting.com 24
![Page 25: Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data Integrity Validation Europe 30 March 2017 Chris Wubbolt, QACV Consulting, LLC. Objectives](https://reader031.fdocuments.us/reader031/viewer/2022022604/5b6032887f8b9a7f038c0773/html5/thumbnails/25.jpg)
SDLC Tools – What can go wrong?
www.QACVConsulting.com 25
Document Management
• Vendor used SharePoint workflow for approval of quality documents. The SharePoint configuration was setup to delete workflows after 90 days.
• All workflows (and subsequent document approvals) were deleted for all quality documents.
Testing
• Test Stuff testing records could not be located for SQA testing.
www.QACVConsulting.com 25
![Page 26: Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data Integrity Validation Europe 30 March 2017 Chris Wubbolt, QACV Consulting, LLC. Objectives](https://reader031.fdocuments.us/reader031/viewer/2022022604/5b6032887f8b9a7f038c0773/html5/thumbnails/26.jpg)
SDLC Tools – What can go wrong?
www.QACVConsulting.com 26
Automated Testing
• Automated test tools passed failing results.
• Test tools were not qualified.
Tool Upgrades / Replacements
• Inability to migrate records from legacy tools.
Records
• Unable to present records of SDLC activities, including test results.
www.QACVConsulting.com 26
![Page 27: Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data Integrity Validation Europe 30 March 2017 Chris Wubbolt, QACV Consulting, LLC. Objectives](https://reader031.fdocuments.us/reader031/viewer/2022022604/5b6032887f8b9a7f038c0773/html5/thumbnails/27.jpg)
• GxP Electronic Recordkeeping Program
• Standard Operating Procedures
• Trained Personnel (including IT)
• Qualified Infrastructure
• Validated Applications
Data Integrity
Data Availability
Data Retention
Computerized Systems
www.QACVConsulting.com 27www.QACVConsulting.com 27
![Page 28: Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data Integrity Validation Europe 30 March 2017 Chris Wubbolt, QACV Consulting, LLC. Objectives](https://reader031.fdocuments.us/reader031/viewer/2022022604/5b6032887f8b9a7f038c0773/html5/thumbnails/28.jpg)
www.QACVConsulting.com 28
Historical
Software Applications
QMS
LIMS
www.QACVConsulting.com 28
![Page 29: Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data Integrity Validation Europe 30 March 2017 Chris Wubbolt, QACV Consulting, LLC. Objectives](https://reader031.fdocuments.us/reader031/viewer/2022022604/5b6032887f8b9a7f038c0773/html5/thumbnails/29.jpg)
www.QACVConsulting.com 29
Historical
Software Applications
QMS
LIMS
www.QACVConsulting.com 29
![Page 30: Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data Integrity Validation Europe 30 March 2017 Chris Wubbolt, QACV Consulting, LLC. Objectives](https://reader031.fdocuments.us/reader031/viewer/2022022604/5b6032887f8b9a7f038c0773/html5/thumbnails/30.jpg)
Pharma A
GxPElectronic Recordkeeping
Controls
Qualified Infrastructure
Standard Operating Procedures
Trained Personnel (including IT)
Validated ApplicationsSTILL NEED
Data Center Inc
www.QACVConsulting.com 30
Historical
www.QACVConsulting.com 30
![Page 31: Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data Integrity Validation Europe 30 March 2017 Chris Wubbolt, QACV Consulting, LLC. Objectives](https://reader031.fdocuments.us/reader031/viewer/2022022604/5b6032887f8b9a7f038c0773/html5/thumbnails/31.jpg)
Software as a Service
www.QACVConsulting.com 31
Fail Over Site
Software Applications
QMS
LIMS
Saas Provider
Data Center
31
![Page 32: Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data Integrity Validation Europe 30 March 2017 Chris Wubbolt, QACV Consulting, LLC. Objectives](https://reader031.fdocuments.us/reader031/viewer/2022022604/5b6032887f8b9a7f038c0773/html5/thumbnails/32.jpg)
Software
Vendor
• Quality System
• SDLC Processes
• Customer Support
Typically not directly regulated or inspected by regulatory agencies.
Audited by clients for adherence to standards.
Quality of SLC Documentation, Testing, etc. varies considerably for each
vendor.
Sponsor responsible for installation, validation, and electronic
recordkeeping controls at sponsor location.
Software as a Service Provider• Quality System
• SDLC Processes
• Customer Support
• Validation
• Data Integrity Controls
Hosted Environment is used for a direct GxPfunction (record keeping)
and is more likely to be inspected by regulatory agencies.
Audited by clients for adherence to standards (GxP, Part 11).
Quality of SDLC Documentation, Testing, etc. varies considerably for
each vendor.
SaaSprovider responsible for some aspects of installation, validation,
and electronic recordkeeping controls.
www.QACVConsulting.com 32
Software Vendor
Hosted
Environment
32
![Page 33: Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data Integrity Validation Europe 30 March 2017 Chris Wubbolt, QACV Consulting, LLC. Objectives](https://reader031.fdocuments.us/reader031/viewer/2022022604/5b6032887f8b9a7f038c0773/html5/thumbnails/33.jpg)
SaaS Vendor Responsibilities
• Validation (with Pharma Company)
• Change Control
• Incident Management
• Maintenance
• Security (Physical and Logical)
• Electronic recordkeeping
• Backup and Restore
• Disaster Recovery
www.QACVConsulting.com 33www.QACVConsulting.com 33
![Page 34: Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data Integrity Validation Europe 30 March 2017 Chris Wubbolt, QACV Consulting, LLC. Objectives](https://reader031.fdocuments.us/reader031/viewer/2022022604/5b6032887f8b9a7f038c0773/html5/thumbnails/34.jpg)
Vendor Audit Observations -
Considerations• Specifications
– Not complete
– Not updated periodically after changes
• Test Records
– No pre-approved Test Plans
– Results not reviewed by second person
– Integrity of test results
– No approved summary reports
• Release Management
www.QACVConsulting.com 34www.QACVConsulting.com 34
![Page 35: Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data Integrity Validation Europe 30 March 2017 Chris Wubbolt, QACV Consulting, LLC. Objectives](https://reader031.fdocuments.us/reader031/viewer/2022022604/5b6032887f8b9a7f038c0773/html5/thumbnails/35.jpg)
Vendor Audit Observations –
Considerations
• Test Record Integrity
– Results and signatures/initials typed into Word document or Excel spreadsheet
– No failures documented
– Test dates and times do not correlate
www.QACVConsulting.com 35www.QACVConsulting.com 35
![Page 36: Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data Integrity Validation Europe 30 March 2017 Chris Wubbolt, QACV Consulting, LLC. Objectives](https://reader031.fdocuments.us/reader031/viewer/2022022604/5b6032887f8b9a7f038c0773/html5/thumbnails/36.jpg)
Vendor Audit Observations –
Considerations• Record Integrity
– Lack of records to demonstrate successful backup
– Failed backups
– Lack of documentation of disaster recovery testing
www.QACVConsulting.com 36www.QACVConsulting.com 36
![Page 37: Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data Integrity Validation Europe 30 March 2017 Chris Wubbolt, QACV Consulting, LLC. Objectives](https://reader031.fdocuments.us/reader031/viewer/2022022604/5b6032887f8b9a7f038c0773/html5/thumbnails/37.jpg)
Summary
www.QACVConsulting.com 37
• Reviewed impact of vendor processes on validation
• Review of Agile SDLC processes
• Discussed new approaches to auditing software vendors
• Reviewed how SDLC and test tools are used by vendors
• Discussed ow SaaS vendors impact your company’s validation approaches and data integrity controls.
![Page 38: Approaches for Auditing Software Vendors - … · Approaches for Auditing Software Vendors Data Integrity Validation Europe 30 March 2017 Chris Wubbolt, QACV Consulting, LLC. Objectives](https://reader031.fdocuments.us/reader031/viewer/2022022604/5b6032887f8b9a7f038c0773/html5/thumbnails/38.jpg)
Questions
www.QACVConsulting.com 38
Chris Wubbolt
QACV Consulting, LLC
Telephone: 610-442-2250
E-mail: [email protected]