Slide 1

Navigation Using Macros and Spreadsheets in a Regulated Environment Ludwig Huber E-mail: ludwig_huber@agilent.com Rev 1.00 Slide 2 Navigation 2 Content FDA requirements and inspection observations Example using functions built in spreadsheet to increase reliability, integrity, security Validation of spreadsheet calculations and Macros Policies and procedures for development and use of spreadsheets and Macros Problems with spreadsheet programs and Macros Slide 3 Navigation 3 FDA Requirements End users should validate any program Macros and other customizations that they prepare http://www.fda.gov/OHRMS/DOCKETS/98fr/001538gd.pdf Draft as of July 2002 No specific guidelines for M&S Spreadsheet Calculations and Macro Programs used in GxP environments should be validated Testing should cover full range Spreadsheet Calculations and Macro Programs should comply with 21 CFR Part 11 (E-records/signatures) www.labcompliance.com/e-signatures www.labcompliance.com/e-signatures Slide 4 Navigation 4 FDA Part 11 Validation Requirements Extent of validation should be based on risk assessment. Setting requirement specifications Verification of proper software installation Functional testing should be quantitative Functional testing should include stress conditions, boundary testing and high load testing Updates should be controlled and documented Regression testing after software changes, e.g. updates Slide 5 Navigation 5 FDA Part 11 Testing Requirement Test conditions should include not only "normal" or "expected" values, but also stress conditions http://www.fda.gov/OHRMS/DOCKETS/98fr/001538gd.pdf Test conditions should extend to boundary values, unexpected data entries, error conditions, reasonableness challenges (e.g., empty fields, and date outliers), branches, data flow, and combinations of inputs. Slide 6 Navigation 6 Warning Letters There is no documentation covering Excel application software, or any procedures instituted covering the protection of electronic records or an established back-up system (069) Failure to validate computer software used as part of the quality system for its intended use according to an established protocol as required by 21 CFR 820.70(i). For example: Software such as Excel, Access, and Word used to create and maintain data bases (rejects, complaints, and concessions) and electronic documents, is not validated (W 073) FDAs current thinking? Reference: www.fdawarningletter.com Slide 7 Navigation 7 Warning Letter - Full Range Testing Failure to use fully validated computer spreadsheets to calculate analytical results for in- process and finished product testing [21 CFR 211.165(e)]. For example, the computer spreadsheets used to calculate analytical results for... have not been validated. Ref: www.fdawarningletter.com QA/QC Spreadsheet Validation, is deficient in that only a small range of values are being used to challenge computerized spreadsheet mathematical calculations. Slide 8 Navigation 8 Question/Answer: Paul Motise Question: Can we still use spreadsheet programs like MS Excel, even though we know that they lack some functionality that are required by part 11, e.g., audit trail? Ref.: www.labcompliance.com/conferences/august99.htm Answer If you are using Excel for something that is required by a predicate rule, then you generate an electronic record and part 11 applies. No official FDA policy but important to know Slide 9 Navigation 9 Business Requirements More importantly, a number of systematic field have reported errors in 24% of the spreadsheets audited (studies since 1997), and these rates may be low because of limits in inspection methods. Overall, the evidence that spreadsheet errors are fairly common is quite strong. Ref: http://panko.cba.hawaii.edu/ssr/devexpt.htm http://panko.cba.hawaii.edu/ssr/cases.htm http://www.kamalasen.com/rajalingham-99.pdf http://www.bized.ac.uk/virtual/vla/theories/spreadsheets_background.htm Slide 10 Navigation 10 What are Macros? Mini programs that performs a specific task - to repeat steps perfectly and with much greater speed. In Excel Macros are written in a programming language called Visual Basic for Applications (VBA). Add functionality to, e.g., Word, Excel In Excel Macros may be but don't have to be created, they are recorded as keystrokes and mouse actions Slide 11 Navigation 11 Problems with Spreadsheets and Macros Easy access to programs Many different environments (operating systems, PC hardware) Anybody (not trained on validation and computer science) can write programs No validation, no documentation Calculations easy to change Many versions in use (local PC, server, inbox, delete folders) QA is not aware of spreadsheet use Typically do not comply with regulations (e.g., part 11) Slide 12 Navigation 12 Alternative Solutions to the Problems Use other programs e.g., perform calculations in secure and validated Agilent ChemStation or Cerity environments with functions for part11 Evaluate add-on software with more security and compliance functionality Develop, implement and enforce procedures for development and use of Macros and Spreadsheets with security, validation and compliance mind. Develop gap analysis, risk assessment, corrective action plan Slide 13 Navigation 13 3rd Part Software Wimmer Systems - DACS www.wimmersystems.com www.wimmersystems.com The eInfotree Digital Compliance Platform from CIMCON Software, Inc. is the life science industry's first and only available solution that provides 21 CFR Part 11 compliance for Excel spreadsheets, Databases and Files/Documents in a single system Cimcon Software eInfotree www.part11solutions.com www.part11solutions.com Stelex - Compliance Builder www.stelex.comwww.stelex.com `Add On` Software for Part 11 e.g., for Excel Slide 14 Navigation 14 Steps Towards Compliance Protect Spreadsheets and Macros using built in standard software and IT infrastructure (e.g., client/server) Develop, communicate and enforce a company policy and master plan for spreadsheet calculations and Macros Prepare a list with all computers running Macros and Spreadsheets Standardize development and use as much as possible Validate spreadsheet calculations and Macros Document your plans and other activities Slide 15 Navigation 15 Develop, Communicate and Enforce a Company Policy Development and Use of Macros and Spreadsheet calculations used in regulated environment must follow documented procedures. Minimum requirements for such Macros and Spreadsheet calculations are: All Macros/spreadsheet calculations used in GxP environment should be registered and controlled Should be validate Should be protected against against accidental and incidental changes Standalone document or Add on to Quality or Validation Policy Slide 16 Navigation 16 Define Applications in a Regulated Environment Travel expenses Time and labor Commercial business reports Evaluation of analytical data Ad hoc evaluations SOP administration Reagents administration Equipment calibration records Training records Stability data Others? __________ GxP relevant? Slide 17 Navigation 17 Standardize Software, Hardware, Development and Documentation Use the same vendor for spreadsheet programs, the same product and version for the application software and for the operating system Limit PC hardware to certain vendors, model, hardware configuration Standardize IT infrastructure Standardize security procedures Standardize validation during development and use Documentation (type, structure, content) example: system inventory, validation report Naming of data files, documents and attachments Efficiency, reliability smooth operation Slide 18 Navigation 18 Standardize System Documentation Use data base with sort functions for application - PC - user - department - validation status Spreadsheet program, version: Operating system: Spreadsheet name: Spreadsheet version: Application: Release date: PC hardware: Description: Vendor, Model: Asset number: Location: Date of installation: Contact for development: People authorized to use: Excel 2000 Windows 2000 Retention time.xls 5.01 Precision of retention times April 5, 2000 Labtop HP Omnibook 4100 237537 Application lab, building 423 May 5, 1999 Mike Win, Telnet 6745234 Curt Myers Slide 19 Navigation 19 Standardize During Development Code conventions Name conventions, e.g. mean_value Code annotations Formulas for specific tasks Colors for cells, e.g., use the same colors for entry fields (yellow), results in spec (green), alerts (red) etc. Spreadsheet annotations (e.g., file name, version) Validation and testing Guidelines for the developers Slide 20 Navigation 20 Use IT Infrastructure and Software Functions to Increase Security & Reliability Protect computer hardware and access to software (user ID, PW access to computer operating system) Run Macro/Spreadsheets from secure server Protect cells, worksheets and VBA scripts Use versioning for Macros and Spreadsheets and include file and version number in spreadsheet print-out Include file name and location in spreadsheet print-out Activate and protect tracking function Slide 21 Navigation 21 Validate Macros&Spreadsheets Validation policy Ordering info: www.labcompliance.com/books Includes Chapter on Macros& Spreadsheet Validation Validation master plan Validation project plan Life cycle approach Requirement specifications Design specifications/development Testing by developer and user Installation and testing On-going use and change control Validation report Slide 22 Navigation 22 Develop&Communicate a Validation Policy All Macros should be validated according to national and international regulations and according global company policy and validation master plan Only validated Macros should be installed and used Use life cycle principle for validation M&S should be under change control procedure Slide 23 Navigation 23 Add Macros&Spreadsheets to the Validation Master Plan Scope (add Macros and spreadsheet examples) Template: www.labcompliance.com/books/masterplan.htm With chapter on Macros/spreadsheets Responsibilities Inventory documentation, example Template for risk assessment Templates for project validation plan Change control Archiving of documentation and data Slide 24 Navigation 24 Life Cycle Model for Macros Change control User requirement/ Functional Specific. Design Specifications Code development Code review Test in development environment Installation and test in user environ. Functional testing can be combined for single users Modified V-model Slide 25 Navigation 25 Risk Assessment List possible risk scenarios Anticipated risk situation Probability Preventive action Severity Test equipment fails lowhigh Purchase redundant system Requirement specifications inadequate high Rigorous verification with many users Estimate probability and severity Suggest preventive actions Slide 26 Navigation 26 Requirement Specifications Describe what the problem is, how it is solved today and how the Macro can help to improve FDA Part 11 Validation Guidance (draft): Without first establishing end user needs and intended uses, we believe it is virtually impossible to confirm that the system can consistently meet them Describe workflow Collect inputs from other users Most important step of validation Frequent question: Why should I write requirement specifications, everybody knows what the spreadsheet does? Slide 27 Navigation 27 Attributes for User Requirements Clear Unambiguous Uniquely numbered Understandable by the programmer Testable and Linked to test cases Classify importance (optional) -high, medium, low- Slide 28 Navigation 28 Requirement Specifications Table (selected examples) Can use word processors, spreadsheets, data base, or specific application software NumberRequirement 3.1 Software must validate input range (e.g., 0.5 to 30 min) and alert the user in case input is out of range 3.2 Calculate and report relative standard deviation Test case T14, T15 T09 Slide 29 Navigation 29 Requirement Specifications - Good and Bad Examples - 1.Entry fields should be colored easy to recognize 2.Operating hints should be part of user interface 3.Must be self explanatory 4.Should not need extensive operating manuals 5.Must be easy to use Testable and quantifiable Slide 30 Navigation 30 Write&Review Design Specifications Define workbook structure e.g., single or multiple worksheets Review design specification with peers (and with users) Describe formulas and the rational behind them When using Excel define to use Excel functions or VBA script Define cell contents for spreadsheets Describe structure of code for Macros Describe how to implement security requirements Define how to handle error messages Defines how functional, security and regulatory requirements are best implemented Slide 31 Navigation 31 Design Macro/Spreadsheet for Error Detection Spreadsheet should detect errors during data entry - wrong type, e.g., string characters instead of numbers - wrong format (e.g., date) - wrong sequence of entries - wrong data range Spreadsheet should prompt the user in case of wrong entries, and not crash Can reduce error rates Slide 32 Navigation 32 Develop, Review & Annotate Code Write code using design specification as basis Document code using standards, e.g., for header - Anybody with similar knowledge should understand it and be able to modify- Review code with Peers. -ensure objectivity! Paul Motise, April 2002, IVT Conference Airlington The author can attend the formal review meeting but should not bias the objectivity Slide 33 Navigation 33 Testing - General Procedure White box/structural testing by programmer FDA Part 11 Validation Guidance (draft): Computer system validation should be performed by persons other than those responsible building the system. Black box/functional testing by user Develop test cases for the functions define acceptance criteria, test, evaluate, document Test plans written by developer or QA Slide 34 Navigation 34 Testing - General Recommendations Test cases should be designed for reuse FDA Part 11 Validation Guidance (draft): 1) End users should conduct functional testing of software that covers all functions of the program that the end user will use. 2) Test conditions should extend to boundary values, unexpected data entries, error conditions, reasonableness challenges (e.g., empty fields, and date outliers), branches, data flow, and combinations of inputs Test under stress and boundary conditions Tests should be linked to requirement specifications Tests should be quantified not just pass/fail Test plans should be authorized and signed before testing starts Apply the concept of risk assessment for the extent of testing Slide 35 Navigation 35 Risk Assessment The risk that the system poses to product safety, efficacy, and quality; note that product means the FDA regulated article (food, human or veterinary drug, biological product, medical device, or radiological product); When you determine the appropriate extent of system validation, the factors you should consider include (but are not limited to) the following Ref: FDA Part 11 Validation Guide (draft) The risk that the system poses to data integrity, authenticity, and confidentiality; and, The system's complexity; a more complex system might warrant a more comprehensive validation effort. Slide 36 Navigation 36 Boundary, Error Condition Testing FDA Part 11 Validation Guidance (draft): Test conditions: test conditions should include not only "normal" or "expected" values, but also stress conditions (such as a high number of users accessing a network at the same time). Test conditions should extend to boundary values, unexpected data entries, error conditions, reasonableness challenges (e.g., empty fields, and data outliers), branches, data flow, and combinations of inputs Define test cases at boundaries Define boundaries in requirement specifications Define test cases outside boundaries Slide 37 Navigation 37 Change Control Follow written procedures Part 11 Guidance on Validation and Glossary (draft) We consider regression testing analysis to be an extremely important tool that should be used to assess portions of a system that were themselves unchanged Regression testing: Rerunning test cases which a program has previously executed correctly in order to detect errors. Evaluate impact on validation Initiated by user, authorized by supervisor and QA Regression testing for all key functions Frequent changes: bug fixing, new functions Statement on testing Slide 38 Navigation 38 Example: Change Request and Control Form Form ID Date Location System ID Description of change Include reason for change Impact of change Evaluate impact on validation Authorization to change Name Signature Date Approved by Name Signature Date Completed by Name Signature Date Implemented Date of implementation Comments E.g., document observations, testing Reference: L.Huber, Validation of Computerized Analytical and Networked Systems, 2002 Interpharm Press Slide 39 Navigation 39 Existing Macros and Spreadsheets Describe the current use Extent of validation depends on risk assessment Describe the functions the M&S has Describe any experiences from past use Document the source code. Document formulas and other key design elements Perform functional testing. Develop test cases for on-going use Make a statement that the M&S is formally validated. Frequent question: Why should I validate a spreadsheet application that I have used successfully for many years? Minimum requirements Impact on product quality number of users frequency of use, complexity, experience Slide 40 Navigation 40 Write the Validation Report Validation approach A good validation master plan combined with a good validation report ensures high probability to survive inspections Summary of validation steps Summary of test results Formal validation statement To be signed by management representative Slide 41 Navigation 41 Problems During Routine Use - Reason for Problems - New users are working with the Spreadsheets without training Non authorized people are using it Users change the Worksheet and save the file under the same name without documentation Users make errors during data entry Most problems come from wrong operation Slide 42 Navigation 42 Routine Use - Recommendations - Train new users Restrict use to authorized persons Protect worksheets and cells Print directory/path/file name with results Limit type and range for data entry Avoid problems Read only access from servers Use original spreadsheets Slide 43 Navigation 43 Example Excel Spreadsheet from a laboratory, can be applied to other applications Calculates mean and absolute and relative standard deviations of retention times in chromatography Validates defined input values Report graphical and numerical results Protected against accidental and incidental changes to the extent possible Access the worksheet from the Server Slide 44 Navigation 44 Example - Tasks Calculate and report mean value Calculate and report absolute standard deviation Calculate and report relative standard deviation Check for and report highest data entry Check for and report lowest data entry Display retention times as graphic (value vs. number of entry) and as table Three categories of specifications tasks usability security/integrity Slide 45 Navigation 45 Example - Improve Usability Highlight entry fields with light yellow for easy recognition Highlight results that are not flagged in blue color Highlight results that are flagged in red color Create the output such that all information fits on a single page Slide 46 Navigation 46 Example - Avoid Errors During Data entry Check and alert the user if entry is below lower or upper limit. With VBA (not included in the example) Check and alert the user to verify if result is above limit Check and alert the user if a single data entry is 50% above average. Check and alert the user if a single data entry is 50% below average. Slide 47 Navigation 47 Spreadsheet Protection&Integrity Protect all cells not used for data entry With VBA (not included in the example) Protect use of Excel sheet by passwords Display directory, subdirectories, file name and sheet name at the bottom of the Spreadsheet Display&print operator name Disable menu and tool bars to limit function to the intended use Disable save and save as Disable cut/copy and paste control keys Directory, path, file name Display date and time of last data entry Slide 48 Navigation 48 Operator Name Data entry Yellow cells Results Green cells Operator Hints Flags Red cells Graphical Results Date &Time Directory Path&file Print on Single page Slide 49 Navigation 49 Additional Resource Macro & Spreadsheet Quality Package Interactive Video Presentation for training Ordering info: www.labcompliance.com/books/macros Primer as reference; checklists Spreadsheet with validation documents (e.g., 21 requirement specs and associated test protocols) SOPs FDA Warning letters Slide 50 Navigation 50 Thank You Ludwig Huber