E-Government Interoperability Government Interoperability ...
Lessons Learned from Army Interoperability Certification ... · Lessons Learned from Army...
Transcript of Lessons Learned from Army Interoperability Certification ... · Lessons Learned from Army...
Lessons Learned from Army Interoperability Certification Testing
Robert BoerjanCTSF Capability Set Coordinator, CTSF(254) 554-1888(254) 532-8321 x2759RDCS Technology, [email protected]
Analysis of Software Block 2 (SWB 2) Test Incident Reports (TIRs)
Background: • The CTSF is the Army Interoperability Certification (AIC) agent for
LandWarNet/Battle Command systems• The Army incorporates a blocking process of multiple systems to
introduce new capability set into the Army• Software Block 2 had 50+ systems • SWB 2 was initially scheduled for 9 months of Test-Fix-Tes, AIC and
Backward Compatibility (BWC) testing – however it took over 2 years before the CIO G6 certified the block and fielding began.
Oct 06 Jan07 Apr07 Jul07 Oct 07 Jan08 Apr08 Jul08 Oct 08 Jan09 Apr09
Original Plan
TFT AIC 1 TIR Val AIC 2 RegressionBWC
First Unit Equipped
Methodology
Results
BCID PCR Master Tracking
Sheet
Configuration Management
Database
TIR Database
Focused on Level 1-3 TIRs Date range is 10/06 thru 12/15/08 Did not evaluate BWC issues Segregated the TIRs into 5 Test Windows: Test-Fix-Test; AIC 1; TIR Evaluation; AIC 2; Regression 1 & 2
Evaluated the following type of data: Number of TIRs over time for entire 2-year period Number of TIRs over time for each Test Window Individual system TIRs and sub-category by severity level TIRs by Issue Category System of System (SoS) vs. System TIRs Graphic TIRs CTSF Configuration Management database of software deliveries: date and purpose of deliveries Battle Command Integration Directorate (BCID) SWB 2 Product Change Request (PCR) Master Log:
Requirements Changed
Suspense date for TCM & PM certification of threads was 18 Sept 06.
Start of TFT to Dec 08, 216 approved PCR changes (count after removing PCR duplicates, archived Threads, and denied PCRs)
38% of PCRs occurred during AIC 1
Maneuver and Aviation PM/TCMs requested the majority of PCRs during AIC 1
Majority of PCRs during Validation and AIC 2 were in response to Thread and SW TIRs
High number of INTEL PCR changes during Regression due to DCGS-A replacing ASAS
6
2512
5 439 4
70
143 1
36
5315
10
1020304050607080
TFT AIC 1 TIR Val AIC 2 REG
Approved Thread PCR Requests by BOS
SANMMVRINTELFSCSSAVNADA
0%20%40%60%80%
100%
2020 9 3 5
1081 41 10 74
PCRs vs. TIRs for SWB 2
PCRs
Admin Thread TIRs + Thread TIRs
Changing Requirements Consume Resources
216 PCRs
71
10
17
10
51 1 2
67
15 14
5 52
4
97
1 2 15
1
20
2 1 13
1
26
51
31
0
10
20
30
40
50
60
70
80
SW Error Thread TIR Class Error Admin Thread TIR
Requirements Standards Architecture Hardware
TFT AIC 1 Validation AIC 2 Regression
SWB 2 TIRS by CategoriesSW Error: Software failure that was fixed with a SW patchThread: System’s requirement not accurately depicted by the threadTIR Class Error: A TIR that either should not have made it out of the DAG; post- score discovery issue was operator induced; or post-score discovery of an inaccurate assessment. Admin Thread TIR: TIRs levied against the thread proponent to ensure thread issue is corrected. Requirements: Change in requirements as directed from TCMs; for example System A received TIR for inability to display graphic from System B, and TCM determined System A did not have a requirement to display graphic from System B.Standards: Conflict between two standards – for example USMTF & VMFArchitecture: Data product issues or missing systems from a particular echelonHardware: Hardware failures
193
39 34
18 11 11 10 3
319 Total TIRs
Total Level 1-3 TIRs for SWB 2
14
31
2420 18
10
39
56
17
10 12
4 610
7 5
13
2 2
116
20
10
20
30
40
50
60
10/1/
2006
11/1/
2006
12/1/
2006
1/1/20
072/1
/2007
3/1/20
074/1
/2007
5/1/20
076/1
/2007
7/1/20
078/1
/2007
9/1/20
0710
/1/20
0711
/1/20
0712
/1/20
071/1
/2008
2/1/20
083/1
/2008
4/1/20
085/1
/2008
6/1/20
087/1
/2008
8/1/20
089/1
/2008
10/1/
2008
11/1/
2008
SWB 2 TIRs by Test WindowsTFT
AIC 1TIR Validation
AIC 2Regression 1 …
Test Windows
Total of 319 LvL 1-3 TIRs 6.9% of TIRs were LvL 1 81.5% of TIRs were LvL 2 11.6% of TIRs were LvL 3 5 systems out 31 Systems with TIRs accounted for 52% of TIRs
SWB 2 Remove
d off Test Floor
14
31
24
2018
10
39
56
17
1012
46
107
5
13
2 2
11
6
2
0
10
20
30
40
50
60
Unexpected Results of AIC & Regression Testing
Expected Outcome
Actual Outcome
There was a only a 1.6% decrease in LvL 1-3 TIRs going from the TFT (117 TIRs) to the AIC 1 (112 TIRs)
The TIR Validation Event was to validate PM SW patches corrected AIC 1 TIRs. In addition to closing TIRs, there was an additional 26 TIRs scored
There was a .6% increase in TIRs from TIR Validation to AIC 2 (28 TIRs)
There was a 2.5% increase in TIRs from AIC2 through Regression Windows (36 TIRs)
Last 3 Windows averaged 30 TIRs per Test Window
SWB 2 Remove
d off Test Floor
TFTAIC 1
TIR ValidationAIC 2
Regression 1 …
Test Windows
SWB 2 TIRs by Severity Level
10 72 3 0
05
1015
LvL 1 TIRs
LvL 1 TIRs
99 101
20 17 23
050
100
TFT AIC 1 TIR Validation
AIC 2 Regression
LvL 2 TIRs
LvL 2 TIRs
84 4
813
05
1015
TFT AIC 1 TIR Validation
AIC 2 Regression
LvL 3 TIRs
LvL 3 TIRs
10 LvL 1 TIRs during TFT were within expectations because it was first time systems evaluated within a robust integrated architecture
7 LvL 1 TIRs for AIC 1 not expected and resulted in inability to fully test systems’ capability
AIC 1 accounted for 39% of all LvL 2 TIRs
The TIR Validation window had 20 new LvL 2 TIRs
The 17 LvL 2 TIRs from AIC 2 were corrected during Regression, but an additional 16 TIRs were scored ( NOTE: 4 OOC systems first test against SWB 2 accounted for an additional 7 LvL 2 TIRs)
LvL 3 TIRs actually increased during AIC 2 and Regression (NOTE: A LvL 3 TIR is a LvL 1 or 2 TIR that has been reduced to a LvL 3 with a valid Technical Bulletin.)
Graphics
Requirements9%
Standards1%
SW Error74%
TIR Class Error16%
SWB 2 Graphics TIRs
1
18
118 8
3
1922
14
1 1 1 105
10152025
SWB 2 Graphic TIRs over Time SWB2 had 108 Graphic related TIRs out of 319 TIRs
Graphic Software Error TIRs accounted for 41% (80 out of 193) of all Software Error TIRs for SWB 2
74% of Graphic TIRs were software errors (80 out of 108) – Systems delivered software that fixed issue
TIR Class Errors result of operators building incorrectly and systems configured incorrectly
Graphic TIRs became a non-factor after AIC 1 window
Not a new issue – SWB 1 had two Graphic Summits to work graphics
TFTAIC 1
TIR ValidationAIC 2
Regression 1 & 2
Test Windows108
Software Deliveries 43 Systems turned in software or software
patches after the TFT Window through the first week of AIC 1 Window.
16 of those systems contributed 88 LvL 1-3 TIRs scored during AIC 1 Window
Fire Support systems and Aviation systems did not participate in the TFT event. 10 LvL 1-2 TIRs are attributed to Aviation assets and issues with FBCB2 Operation Center (OPS CNTR) and Fire support.
45 software deliveries to CTSF CM for inclusion in TIR Validation from Nov 07 thru 01 Feb 08. 12 of the SW deliveries were multiple drops from 6 systems. 7 of the systems accounted for 23 out of the 26 TIRs
16 systems delivered software during AIC 2. 6 of the systems accounted for 17 LvL 1-3 TIRs
85
11
4 3 47
10
5
10
15
4-Mar-07 14-Mar-07 24-Mar-07 3-Apr-07 13-Apr-07 23-Apr-07
PM SW Deliveries pre-AIC 1
# SW Deliveries per Date
10
39
3 26 5 7
0
20
5-Oct-07 25-Oct-07 14-Nov-07 4-Dec-07 24-Dec-07 13-Jan-08 2-Feb-08 22-Feb-08
# SW Deliveries During TIR Validation
# SW Deliveries per Date
1 14 6
3 4
05
10
2-Feb-08 17-Feb-08 3-Mar-08 18-Mar-08 2-Apr-08 17-Apr-08 2-May-08 17-May-08
# of SW Deliveries During AIC 2
# of SW Deliveries per Date
SoS Maturation is required after systems deliver software
SoS vs. System TIRs 74% of LvL 1-3 Software TIRs were SoS TIRs
37% of LvL 1-3 Software TIRs found in AIC 1 were System TIRs.
9 of the systems, with AIC 1 TIRs delivered SW for the TIR Validation Test and received additional TIRs. 5 out of the 9 systems had TIRs that were System TIRs
0 10 20 30 40 50 60 70
Regression
AIC 2
TIR Validation
AIC 1
TFT
8
6
5
25
10
28
22
4
42
61
SoS vs. System TIRs
SoSSystem
1
2
1
2
SoS Events find both SoS and system errors
Summary
SoS Interoperability development is a process that is dependent upon:– Stable Requirements– Software Maturity– SoS Integration Capability
Goal is a disciplined and repeatable process