Principles of Software Verification and Validation

35
Principles of Software Verification and Validation for Medical Imaging Twin Spin October 7, 2010 07-OCT-2010

Transcript of Principles of Software Verification and Validation

Page 1: Principles of Software Verification and Validation

Principles of Software Verification and Validation for Medical Imaging

Twin SpinOctober 7, 2010

07-OCT-2010

Page 2: Principles of Software Verification and Validation

Topics

2

Short history of medical imaging How SW “Changed the Picture” Challenges and Solutions to Software

Verification 4 Areas of image trustworthiness Importance of SW Engineering Principles

Product Software Validation Summary

07-OCT-2010

Page 3: Principles of Software Verification and Validation

History

07-OCT-2010 3

Röntgen discovers X-rays in 1885, receives Nobel Prize in 1901.

Rapid research and discovery leading to working prototype x-ray imaging system by Edison in 1901

Other modalities follow rapid development in 20th century.

Page 4: Principles of Software Verification and Validation

12/10/2008 4

Boom in x-ray-associated marketing.

Page 5: Principles of Software Verification and Validation

Medical Imaging Today

Multiple Modalities Electron microscopy Radiographic – standard x-rays, fluoroscopy MRI Nuclear medicine – PET, gamma Thermography Tomography – CT scans Ultrasound Photoacoustic imaging - lasers + ultrasound.

12/10/2008 5

Page 6: Principles of Software Verification and Validation

Medical Imaging Today

607-OCT-2010

Page 7: Principles of Software Verification and Validation

Demonstration

2D images on e-film portable reader, courtesy my CT scan.

3D, 4D Images from Osirix open source reader. Courtesy

12/10/2008 7

Page 8: Principles of Software Verification and Validation

For purposes of understanding these principles, this talk will focus on CT technology.

07-OCT-2010 8

Page 9: Principles of Software Verification and Validation

How Software “Changed the Picture”Yesterday:

07-OCT-2010 9

Hardware config and control

Hardcopy output (film)

Manual archiving

Limited reviewHardware positioning

Page 10: Principles of Software Verification and Validation

Today:

07-OCT-2010 10

Software config and control console

Software patient positioning control

Digital raw data

Dedicated digital signal processing

Transmission over enterprise network DICOM data output

Archival File Server (PACS) PACS reading station Advanced visualization workstation

Page 11: Principles of Software Verification and Validation

The essential question:

Can I trust these pictures?

07-OCT-2010 11

Page 12: Principles of Software Verification and Validation

To help answer the question, we ask two more questions:

Q: What is essential in the image?A: This is validation. “What is the right information?”

Q: How much error does the essential information contain?A: This is verification. “How do I insure the information is as error-free as possible?”

07-OCT-2010 12

Page 13: Principles of Software Verification and Validation

The essential question changes:From:

Q: Can I trust these pictures?

To:Q: How much can I trust these pictures?

07-OCT-2010 13

Page 14: Principles of Software Verification and Validation

Answering Question 1: An Art Lesson

Visual artists have known that images are an interpretation of reality and exploit that fact to convey essential messages.

Medical imaging is an interpretation of reality too, intentionally distorted through reconstruction algorithms to convey essential diagnostic information (i.e. right

information).

07-OCT-2010 14

Interpretation involves subjectivity.

Boy or girl?

Page 15: Principles of Software Verification and Validation

How to create trust in a subjective process?

A: Experience

Through certification, advanced education, on-the-job training, etc.

07-OCT-2010 15

Page 16: Principles of Software Verification and Validation

Product Validation

Validation involves the customer or it isn’t validation. Validation of images by medical imaging professionals is

absolutely vital: radiologists, CT/MR technologists, etc. Validation can be done at nearly all steps in the

development. Beta field phase absolutely essential. Some issues can only be validated

Human factors: presentation of on-screen information, segmentation preferences, image fidelity preferences.

Visualization of small vessels or structures whose phantoms are too costly.

12/10/2008 16

Page 17: Principles of Software Verification and Validation

Product Validation Approaches

Internal panel of expert employees – training team, field application teams.

Internal panel of expert consultants – medical advisory board, focus groups.

Luminary sites willing to partner in product development.

Industry-acknowledged body of knowledge. Walter Reed colon datasets, Stanford bake-off.

12/10/2008 17

Page 18: Principles of Software Verification and Validation

07-OCT-2010 18

User errorModality quantization,

noise errorReconstruction algorithmic error “false images”

Data transmission protocol error

Database error User error, visual errors 3D processing, user, & algorithmic errors

Answering Question 2: Dataflow error

Page 19: Principles of Software Verification and Validation

Error in the dataflow.Type of Error Frequency Root source of error

User error at console Potentially often Human factors design, functional flaw.

Quantization and noise

Every scan Law of nature.

Reconstruction error Every scan Law of nature, functional flaw

Data transmission error

Rare and getting rarer

TCP/IP stack, DICOM stack, functional flaw.

Database error Rare and getting rarer

Configuration error, functional flaw.

PACS user error Potentially often Human factors design, image processing, functional flaw.

Viz workstation error Potentially often Human factors design, image processing, functional flaw.

07-OCT-2010 19

Page 20: Principles of Software Verification and Validation

The technical effort simplifies to reducing the impact of undetected functional error on the image to be less than or equal to the impact of inherent error on the image SO THAT the images and data derived from the images are trustworthy.

Any “questionable” image abnormalities visual artifacts, or other deficiencies are due to laws of physics inherent in the modality.

07-OCT-2010 20

Page 21: Principles of Software Verification and Validation

The four areas for an image to be trustworthy from the user perspective:

Image orientation Image fidelity Measurements Data integrity

07-OCT-2010 21

Page 22: Principles of Software Verification and Validation

Workstation error in the end image.Area of concern Root source of error

Image orientation Functional error processing DICOM header and placing orientation markers on screen. Manifested by rotation of image by 90 degree increments on screen.

Image fidelity Functional error introduced by incorrect graphic engine processing and algorithm application. Manifested by obscured anatomy, rendering, incorrect segmentation.

Measurements Functional error processing scaling factor from DICOM header or error in measurement algorithms. Manifested by rulers wrong by large fixed multiples, measurement of same anatomical structures changing as code changes.

Data Integrity Failure in data transfer, processing of incoming DICOM data, cross-check of header information, management of database, volume. Manifested by missing or repeated areas in the 2D and 3D images.

07-OCT-2010 22

Page 23: Principles of Software Verification and Validation

How to verify Image Orientation

07-OCT-2010 23

Scan multiple times

Manufacture an object of knowndimension, HU values, orientation

Check the result

The reference library of orientation datasets is ideal for regression automation.

Page 24: Principles of Software Verification and Validation

Phantom manufacturers

http://www.universalmedicalinc.com/diagnostic-imaging/imaging-quality-control/phantoms/ct-phantom

http://www.phantomlab.com/rsvp_head.html http://www.cirsinc.com

12/10/2008 24

Page 25: Principles of Software Verification and Validation

How to verify Image Fidelity

07-OCT-2010 25

Use reference datasets• Phantoms• Synthetic data

Check the result against reference image using image pixel checker

Problem: subtle differences due to video driver revs cause false failures. **Error below threshold of human eye** Tools must not flag errors that are not noticeable (or relevant) to the user. The sample image must be transferred through a daisy chain of imaging workstations to simulate the enterprise environment. Reprocessing the image can, in rare cases, lead to image degradation. User acceptance panels are another tactic – often for selecting default color tables.

DICOM xfer

Page 26: Principles of Software Verification and Validation

How to verify Measurements

07-OCT-2010 26

Use reference datasets• Phantoms• Synthetic data

Check the result against reference measurement

DICOM xfer

In some cases it is necessary to manually complete a typical patient “workup” or workflow, transfer the patient record to a second workstation, and verify the measurements maintain consistency in data transfer & processing.

All manual measurements are, by their nature, subject to human error. Define +- bounds

Page 27: Principles of Software Verification and Validation

How to verify Data Integrity

07-OCT-2010 27

Use reference datasets• Phantoms• Synthetic data•Large, small, •Multiple modalities•Load testing •Negative testing

Check the results at the database, not the user interface. We assume the user interface & visualization do not corrupt the data (it is prudent to verify this assumption if using this strategy).

DICOM xfer

DICOM Database

Data transfer testing is executed in isolation and in concert with orientation, fidelity, and measurement verification.

Page 28: Principles of Software Verification and Validation

Other things that affect trustworthiness Reliability Stability Security Installation & Upgrades System integration Localization Licensing Performance Manufacturing & distribution Etc.

07-OCT-2010 28

Page 29: Principles of Software Verification and Validation

In other words….

Very few of the verification factors for trustworthiness are medical imaging-specific. Most are core software engineering principles and good quality engineering. Good requirements development and management Good code development and management Good test case development and management

07-OCT-2010 29

Following good (not even best) practices automatically generates artifacts that audit agencies look for as proof of regulatory compliance.

Page 30: Principles of Software Verification and Validation

Example of SW Engineering lapse.

GE Healthcare, August 2008.Optovue, 2010

12/10/2008 30

Page 31: Principles of Software Verification and Validation

Risk-based verification

Everything can’t be tested Organizations inevitably take a risk-based approach

whether they know it or not. “Good enough” by instinct to begin “Good enough” by systematic classification by end.

Cross departmental effort to classify risks (patient risks, business risks)

Living document of decision-making as-you-go. Complies with regulatory risk assessment deliverables. High business value.

07-OCT-2010 31

Page 32: Principles of Software Verification and Validation

Risk-based verification derived from Business Needs/Risks

All risks are secondary to patient risk. Recall vs. patch

07-OCT-2010 32

Business Need: (product will not

harm patient)

Project Risk Assessment(schedule,

market, patient hazards, etc)

Design Mitigation

Risk-based system testing

Unit testingBusiness Need:( )

Business Need( )

Page 33: Principles of Software Verification and Validation

Summary

The key to medical imaging is trustworthiness of the image – even with known error sources.

Verifying medical imaging software is an engineering task approaching image orientation, image fidelity, measurements, and data integrity

Validating imaging software is a clinical user task overlapping with human factors and artistic questions of interpretation and presentation.

All other challenges are familiar core software engineering problems.

07-OCT-2010 33

Page 34: Principles of Software Verification and Validation

Biography

Alex Dietz has been applying product quality verification principles for 20+ years in telecommunications, data transmission, and medical imaging. He currently manages the Software Verification and Validation team for the EnSite cardiac mapping system at St. Jude Medical. He has spoken locally and nationally, most recently at the Software Design for Medical Devices conference in San Diego.

[email protected]

07-OCT-2010 34

Page 35: Principles of Software Verification and Validation

References

History Naked To The Bone: Medical Imaging In The Twentieth Century

by Bettyanne Kevles http://en.wikipedia.org/wiki/X-ray#History

Phantom Manufacturers http://www.phantomlab.com/ http://www.universalmedicalinc.com http://www.cirsinc.com

07-OCT-2010 35