VARTAN – Validation Reporting Templates Jürgen Teutsch, NLR CAATS Workshop, 16-Feb-2006,...

VARTAN – Validation Reporting Templates

Jürgen Teutsch, NLRCAATS Workshop, 16-Feb-2006, Lanzarote

VARTAN – Project Objectives

VARTAN is concerned with...

Definition of Validation Reporting Templates in accordance with European Operational Concept Validation Methodology (E-OCVM)

Analysis of E-OCVM

Refinement of Life Cycle Model and Case-based View

Context of templates in Life Cycle Model

Study for automated transfer of template contents into the Validation Data Repository (VDR)

VARTAN – Analysis Issues

Performance expressed in structured way?– Key Performance Areas (PRC/ICAO)– Key Performance Indicators

Evidence with right level of quality?– Validity and accuracy of results (and objectives)

Right levels of detail at different decision-making levels?– Strategy, programme, project, and experiment level

Information needs of all decision makers addressed?– Airlines, service providers etc.

E-OCVM comprehensible for less proficient user?– Simplify methodology for different E-OCVM views

VARTAN – Results Overview

Suggested (or similar) refinements are envisaged for implementation in the updated methodology:

Improve comprehensibility of Life Cycle Phase Model Refine case-building process description Describe and give examples of possible stakeholder needs and

associated stakeholder analysis issues Consider confidence in results

Activities for preparation of document standards with automation:

Build templates at different planning levels and determine triggers for reporting

Provide automation example for experiment-level validation reporting template

Describe process for working with result templates

VARTAN – Suggested Life Cycle Refinements

Plan

Do

Check

Act

Define Goals and Cases

Decide onWay to Proceed

Carry OutExperiments

Study Results of Experiment

Deming Cycle for ATM Concept Validation

I mplementedProcedure or ProcessI nitial I dea

Scope Operability Performance I ndustrialisation I mplementation

V1 V2 V3 V4 V5

Concept Principles

Techniques•High-level Modelling•Fast-time Simulations•Expert J udgement

KPA/ Cases•Acceptability•Access and Equity•Capacity and Delay•Cost Efficiency•Efficiency•Environment•Flexibility•Interoperability•Safety•Security

Expected Results•Stakeholder I ssues•Time Planning•Scenario Definitions•Benefit Mechanisms•Potential Benefits•Showstoppers•Risks

Feasibility

Techniques•Requirement Capturing•Rapid Prototyping•Operational Simulations•Procedure Analysis•Scenario Evaluation

KPA/ Cases•Acceptability•Efficiency•Human Factors•Interoperability•Predictability•Safety

Expected Results•System Operability•User Operability•Procedure Description•Non-nominal Cases•HMI Prototype•Workload Estimates•Concept Baseline•Risks and Mitigation

Concept I ntegration

Techniques•Real-time Simulations•Live Trials•Shadow-mode Trials•Cost-Benefit Analyses•Safety Analysis

KPA/ Cases•Acceptability•Capacity and Delay•Cost Efficiency•Efficiency•Environment•Safety

Expected Results•Performance Data•Stakeholder Approval•Acquisition Process•Detailed Procedures•Detailed Concept

I ndustrialisation

•Procedure Approval•Acquisition•Factory Test

I mplementation

•In-Service Test•In-Service

VARTAN – Suggested Case-building Refinements

From KPA and KPI to case definition: which cases are necessary for decision making?

Interrelationships between defined areas (cases)

Differences in focus from V1 to V3; from identification of showstoppers (V1) to implementation decision (V3)

Discern Verification and Validation (especially in V2)

Different levels of detail in cases: what level of detail is necessary for decision making per lifecycle phase?

VARTAN – Proposals for Stakeholder Needs

Conflicting stakeholder objectives/goals or interests– Goal balancing– Harmonisation of concepts and ideas

Different stakeholder information needs– Different levels of reporting

Different levels of quality (validity/accuracy) of results – Use of standard methods to increase (or find a

common level of) validity

VARTAN – General Template Considerations

• Pre-defined introductory part (background, context, scope)

• Summary of planning document

• Differences between planning and actual achievements

• Report confidence in results

• Objective should be related to a certain KPA

• Correlation between results / conclusions / recommendations and objectives (from high-level to low-level to hypotheses)

VARTAN – Planning and Reporting LevelsPlanning Levels

Life Cycle Phase

Strategic Level (Decision Makers)Set overall and intermediate goalsDetermine timeframeDetermine Key Performance Areas (KPA)Solve stakeholder issuesPrioritise ATM needs

Programme Level (Programme Managers)Address particular intermediate goal(s)Focus on particular KPAIdentify interrelations between KPAsDetermine KPIs inherent to KPAsDefine cases necessary to support goals

Project Level (Project Managers)Define validation strategyImplement case(s)Identify hypotheses based on goals/objectivesProduce scenarios and experiment planProduce analysis plan

Experiment Level (Experiment Leader)Prepare platform or facilityPerform pre-exercise testing and trainingConduct validation experiment(s)Perform analysisStart reporting cycle on lowest level

Experiment Level Results Report

Summary of planningExperiment executionExperiment resultsExperiment analysisConclusionsRecommendations

Programme Level Results Report

Goals and ObjectivesKPA, KPI and casesProgramme resultsProgramme analysisConclusionsRecommendations

Project Level Results Report

Project ObjectivesValidation strategyImplemented case(s)Project resultsProject analysisConclusionsRecommendations

Continue ?

Continue ?

Continue ?

YES

YES

NO

NO

NO YES

Strategic Level Planning Report

KPAATM ProblemsATM NeedsPrioritisationOp. ImprovementsExpected BenefitsProgramme Outline

Programme Level Planning Report

ATM NeedsOperational ConceptOI ClustersProgr. ObjectivesProgr. StakeholdersProgr. DependenciesFactors of InfluenceTools and TechniquesProgramme OverviewResourcesDeliverablesRisks

Project Level Planning Report

Project ContextProj. ObjectivesProj. StakeholdersProj. DependenciesIdentified CasesFactors of InfluenceTools and PlatformsProject Overview ResourcesDeliverablesRisks

Experiment Level Planning Report

N/A

Considered to betoo prescriptive

Experiment Level

Strategy Level

Programme Level

Project Level

VARTAN – Planning and Reporting Levels

Planning Levels

Life Cycle Phase

Strategic Level (Decision Makers)Set overall and intermediate goalsDetermine timeframeDetermine Key Performance Areas (KPA)Solve stakeholder issuesPrioritise ATM needs

Programme Level (Programme Managers)Address particular intermediate goal(s)Focus on particular KPAIdentify interrelations between KPAsDetermine KPIs inherent to KPAsDefine cases necessary to support goals

Programme Level Results Report

Goals and ObjectivesKPA, KPI and casesProgramme resultsProgramme analysisConclusionsRecommendations

Continue ?

Continue ?

YES

NO

NO YES

Strategic Level Planning Report

KPAATM ProblemsATM NeedsPrioritisationOp. ImprovementsExpected BenefitsProgramme Outline

Programme Level Planning Report

ATM NeedsOperational ConceptOI ClustersProgr. ObjectivesProgr. StakeholdersProgr. DependenciesFactors of InfluenceTools and TechniquesProgramme OverviewResourcesDeliverablesRisks

VARTAN – Planning and Reporting Levels

Project Level (Project Managers)Define validation strategyImplement case(s)Identify hypotheses based on goals/objectivesProduce scenarios and experiment planProduce analysis plan

Experiment Level (Experiment Leader)Prepare platform or facilityPerform pre-exercise testing and trainingConduct validation experiment(s)Perform analysisStart reporting cycle on lowest level

Experiment Level Results Report

Summary of planningExperiment executionExperiment resultsExperiment analysisConclusionsRecommendations

Project Level Results Report

Project ObjectivesValidation strategyImplemented case(s)Project resultsProject analysisConclusionsRecommendations

Continue ?

YES

NO

Project Level Planning Report

Project ContextProj. ObjectivesProj. StakeholdersProj. DependenciesIdentified CasesFactors of InfluenceTools and PlatformsProject Overview ResourcesDeliverablesRisks

Experiment Level Planning Report

N/A

Considered to betoo prescriptive

relevantVDR information

VARTAN – Unexpected Results

Validation Reporting Triggers (Exceptional triggers)

Exceptional situations with unexpected results or events might lead to a re-evaluation of the current activity

Need to report to higher levels immediately and therefore trigger a special report

Important: there must be a way to stop the validation cycle and reconsider the complete approach

Generally, such reporting is part of a good management approach (not part of E-OCVM guideline structure)

VARTAN – Validation Reporting Process (1/3)

Computer with

MS-Word Client

I nternet

Validation ReportingTemplates

VDR Website

VARTAN – Template Life Cycle

Empty Template

Accepted Template

Prepared Template

Empty Template

Accepted Template

Prepared Template

Filled-in Template

Received Template

Approved Template

Filled-in Template

Received Template

Approved Template

Author accepts data

VDR Data Collectoradds data

Author does notaccept data

VDR Administratorapproves template

VDR Administratorrejects template Author

transmits template

Authorfills in template

Authorupdates template


VDR(Oracle Database)

Computer with

MS-Word Client

I nternet

Filled-in Templates


Empty Template

Accepted Template

Prepared Template

Empty Template

Accepted Template

Prepared Template

Filled-in Template

Received Template

Approved Template

Filled-in Template

Received Template

Approved Template

Author accepts data





transmits template




Validation Expert

VDR Administrator

TCP/IP

HTTP

Web Services – SOAP (XML)

Apache Axis

Java Servlet

VDR (Oracle)

VDR Interface

SQLStatements


Empty Template

Accepted Template

Prepared Template

Empty Template

Accepted Template

Prepared Template

Filled-in Template

Received Template

Approved Template

Filled-in Template

Received Template

Approved Template

Author accepts data





transmits template



VARTAN – Template Contents:Help Text

VARTAN – Template Contents:Introduction

1. Introduction

1.1 Project/Experiment Background and Context

1.2 Concept Overview

1.3 Purpose of Document

1.4 Scope and Structure of Document

1.5 References

1.6 Acronyms

VARTAN – Template Contents:Project Level Strategy and Planning

2. Summary of Project Strategy and Planning

2.1Stakeholder Requirements

2.2Supporting Cases

2.3Expected Project Outcomes and Objectives

2.3.1 Description of Expected Project Outcomes2.3.2 Summary of Expected Project Outcomes2.3.3 Description of Project Objectives2.3.4 Summary of Project Objectives

2.4Decision Processes in Experiment Definition

2.5Experiment Context and Planning

2.6Analysis Strategy

VARTAN – Template Contents:Project Level Conduct and Results

3. Conduct of Validation Project3.1Executed Experiments3.2Deviations from the Planning

4. Results of Experiments4.1Results of Different Experiments per Key Performance Area4.2Confidence in Results of Different Experiments

4.2.1 Quality of Results of Different Experiments

4.2.2 Significance of Results of Different Experiments

4.3Unexpected Project Results

5. Consolidation of Project Results5.1Consequences of Confidence in Experiment Results5.2Consequences of Experiment Results for Project Objectives

5.2.1 Interpretation of Results with respect to High-level Objectives

5.2.2 Interpretation of Results with respect to Key Performance Areas

6. Conclusions and Recommendations for Programme Level6.1Conclusions6.2 Recommendations

VARTAN – Template Contents:Experiment Level Strategy and Planning

2. Summary of Experiment and Strategy Planning

2.1 Expected Experiment Outcomes, Objectives and Hypotheses

2.1.1 Description of Expected Experiment Outcomes

2.1.2 Summary of Expected Experiment Outcomes

2.1.3 Description of Experiment Objectives

2.1.4 Summary of Experiment Objectives

2.1.5 Description of Experiment Hypotheses

2.1.6 Summary of Experiment Hypotheses

2.2 Choice of Metrics and Measurements

2.3 Choice of Methods and Techniques

2.4 Scenario Specifications

2.5 Experimental Variables and Design

VARTAN – Template Contents:Experiment Level Conduct and Results

3. Conduct of Validation Exercise Runs3.1Experiment Preparation3.2Executed Experiment Schedule3.3Deviations from the Planning

4. Experiment Results4.1Measured Experiment Results4.2Confidence in Experiment Results

4.2.1 Quality of Results of Experiment

4.2.2 Significance of Results of Experiment

4.3Unexpected Experiment Results

5. Analysis of Experiment Outcomes5.1Analysis of Outcomes on the Basis of Determined Hypotheses5.2Analysis of Consequences of Outcomes for Experiment Objectives

6. Conclusions and Recommendations for Project Level6.1Conclusions6.2 Recommendations

VARTAN - Template Automation: Basic Information

VARTAN - Template Automation: Standard Structure

VARTAN - Template Automation: Entry Fields

VARTAN – Outlook into the Future

Data definition and requirements for (static) document data and pre-formatted data

Data structures for VDR entry including mutual relations

Encryption standards for usernames and passwords

Data protection modes and protected sections

List of all property fields, bookmarks and macros in use by the automated solution to avoid conflicts

Define rollback mechanisms for all automated changes

Templates available at VALUG website: http://www.eurocontrol.int/valug/

Website offers contact information and is available for feedback on methodology and standard methods

Questions

VARTAN – Validation Reporting Templates Jürgen Teutsch, NLR CAATS Workshop, 16-Feb-2006,...

Documents

Transcript of VARTAN – Validation Reporting Templates Jürgen Teutsch, NLR CAATS Workshop, 16-Feb-2006,...