SDS Testing / Verification @ the CERES Delta Design Review

SDS Testing / Verification @ the CERES Delta Design Review

Victor Buczkowski

Agenda

• Progress & Schedule• Testing Process• Progress Accounting & Reporting• Requirements Verification Progress• Score Card

Test Activities Completed, In-progress & Plannedpart 1 of 2

• Test proxy and synthetic data Ongoing– Currently have one Orbit of test data– Awaiting 6 orbits of data– Need 72 hours + of continuous data or dupe the 6 orbits

• System Test Plan Ongoing– Test Plan completed– Refining the number of requirements that each element has (in final stages)– Test Procedures provided by the element for each test

• SDS Functional Test #1 – 3/2008 thru 12/2008– Test Readiness Reviews 3 days prior to the test Ongoing– Test Execution (usually on Thursdays)– Test Results Report (a week after the test)

• SDS NCT3 Readiness Test – 3/2009 - All Elements– Test Readiness Reviews ? days prior to the test In Planning– Test Execution (3 days ?)– Test Results Report (a week after the test)

Test Activities Completed, In-progress & Planned part 2 of 2

•SDS Functional Test #2 – 3/2008 thru 10/2009 In Planning

–Test Readiness Reviews 3 days prior to the test (usually on Monday)–Test Execution (usually on Thursdays)–Test Results Report (a week after the test)

•SDS Capacity/Performance Test – 10/2009 - All Elements In Planning

–Test Readiness Reviews 3 days prior to the test (usually on Monday) –Test Execution (usually on Thursdays)–Test Report (a week after test)

•SDS 3 Day in the Life Test – 10/2009 - All Elements In Planning

–Test Readiness Reviews 3 days prior to the test (usually on Monday) –Test Execution (usually on Thursdays)–Test Results Report (a week after the test)

Testing Process1 of 2

• Identify when an element is ready to test (single point coordination = Vic)– Need Build schedule (1, 2 or more deliveries?)– Identify Testable Requirements being delivered (what's

testable based on the elements capabilities and their interfaces capabilities)

– Schedule a test date– Convene a test discussion meeting– Determine what test data is available (real or fabricated)

• SD3E can help identify what’s available as well as what other elements have available.

– Develop the test procedure (the steps to be used to verify a particular requirement. This becomes your SOP)

Testing Process2 of 2

– Conduct the Test Readiness Review (TRR usually three days prior to the test to insure everything is ready)

– Execute the test– Provide the test results/report

TRR Date Test Date StatusElement Internal

SD3E to Ozone 3/24/2008 3/27/2008 CompleteSD3E to Ocean 5/5/2008 5/15/2008 CompleteSD3E to LAND 5/19/2008 5/22/2008 CompleteSD3E to NICSE 6/2/2008 6/5/2008 CompleteSD3E to Atmosphere ? ? ?SD3E to Sounder 7/14/2008 7/17/2008 Complete

Ocean to PSOE ? ? ?Ocean to I&TSE ? ? ?

Land to CERES 12/1/2008 12/1/2008 ?Land to NICSE 5/19/2008 5/22/2008 CompleteLand to PSOE 7/14/2008 7/18/2008 CompleteLand to I&TSE 7/14/2008 7/18/2008 Complete

Ozone to PSOE 3/24/2008 3/27/2008 CompleteOzone to I&TSE 3/24/2008 3/27/2008 Complete

NICSE to Land PEATE 5/19/2008 5/22/2008 CompleteNICSE to PSOE 5/28/2008 5/28/2008 CompleteNICSE to I&TSE 6/2/2008 6/5/2008 Complete

Atmosphere to PSOE ? ? ?Atmosphere to I&TSE ? ? ?

Sounder to PSOE 7/14/2008 7/17/2008 CompleteSounder to I&TSE 7/14/2008 7/17/2008 Complete

PEATE Externals

Ozone to CLASS/ADS 3/24/2008 3/27/2008 CompleteOzone to CasaNOSA 3/24/2008 3/27/2008 CompleteOzone to Science Team 3/24/2008 3/27/2008 CompleteOzone to C3S (WRS) 12/1/2008 12/1/2008 ?

SDS Functional Test # 1 Schedule & Progress1 of 2

Ocean to CLASS/ADS ? ? ?Ocean to CasaNOSA ? ? ?Ocean to Science Team ? ? ?Ocean to C3S ? ? ?

Land to CLASS/ADS 5/19/2008 5/22/2008 CompleteLand to CasaNOSA 7/14/2008 7/18/2008 CompleteLand to Science Team 7/14/2008 7/18/2008 CompleteLand to C3S 7/14/2008 7/18/2008 Complete

NICSE to CLASS/ADS 6/2/2008 6/5/2008 CompleteNICSE to CasaNOSA 1/15/2008 1/15/2008 CompletePSOE to C3S ? ? ?

Atmosphere to CLASS/ADS ? ? ?Atmosphere to CasaNOSA ? ? ?Atmosphere to Science Team ? ? ?Atmosphere to C3S ? ? ?

Sounder to CLASS/ADS 7/14/2008 7/17/2008 CompleteSounder to CasaNOSA 7/14/2008 7/17/2008 CompleteSounder to Science Team 7/14/2008 7/17/2008 CompleteSounder to C3S 7/14/2008 7/17/2008 Complete

CERES to CLASS/ADS TBD TBDCERES to CasaNOSA TBD TBDCERES to Science Team TBD TBDCERES to C3S (WRS) TBD TBD

SD3E External

SD3E to IDPS 9/8/2008 9/8/2008 ConfirmedSD3E to CLASS/ADS 9/9/2008 9/9/2008 ConfirmedSD3E to NSIPS 9/30/2009 9/30/2009 ?SD3E to C3S (WRS) 8/4/2008 8/04/008 Confirmed

SDS Functional Test # 1 Schedule & Progress2 of 2

NPP/SDS Schedule

Accounting Process1 of 2

Test Accounting• Conduct Test Readiness Reviews (TRR)

– Identify what is being tested• Schedule, Coordinate and Execute the tests• Collect, Review & Maintain test report(s)

– What requirements were verified and how– Pass, Fail, Partial, Status assessment– Issues and discrepancies must be identified, tracked

and, retested as necessary– These materials are maintained by the SDS Test

Coordinator• The test verification matrix and a score card will

be maintained regularly

• Test Results:• As testing proceeds, results will be

identified as:– Verified (worked as expected and planned)

– Failed (discrepancies will be identified by generating a Test Discrepancy Report (TDR) and tracked via Bugzilla)

– Workaround (verified, but needed help – all functionality not available)

– Deferred (some may not be verified until after launch)

– Waived (no-longer needed to support the mission at this time)

Accounting Process2 of 2

Reporting ( page 1 of 2)

Sample Procedure

2. Land PEATE - SD3E Data Ingest Tests

Description of requirements and their status related to this test:

Reqmt Short text Long Text Status 3.4.1.1 PEATEs xDR Ingest from the SD3E The PEATEs shall have the capability of ingesting xDRs from

the SD3E. Passed

3.4.1.2 PEATEs IP Ingest from the SD3E The PEATEs shall have the capability of ingesting IPs from the SD3E

Passed

3.4.1.3 PEATEs Official Ancillary Data Ingest from the SD3E

The PEATEs shall have the capability of ingesting official ancillary data from the SD3E.

Passed

3.4.1.4 PEATEs Calibration Product Ingest from the SD3E

The PEATEs shall have the capability of ingesting and storing calibration products from the SD3E.

Deferred

3.4.1.5 Request listing The PEATEs shall have the capability of requesting: a listing of current products in SD3E, a listing of products for SD3E to re-acquire, and a listing of products for SD3E to retransmit

Deferred

Reporting ( page 1 of 2)

1) Manually Ingest Detailed Steps

Step Description/Action Result/Notes Pass/Fail

/

1 Open a terminal session on the NPPDAPS system with v2lp1 account, change the directory to ~v2lp1/tmp

ssh v2lp1@lpeate1 cd ~v2lp1/tmp

v2lp1 on NPPDAPS account is required for this test

2 Login to SD3E ftp server using ncftp with anonymous account

ncftp nppsds1.nascom.nasa.gov

Login successfully

3 Verify the existence of test files and download the test files from different target directories

for GEO, EDR test files cd /pub/DailyIngest/20080331/Land ls mget GMTCO_npp_d20030125_t084705_e084830_b00015_c20080131015913_gsfc_SCI.h5* mget VI1BO_npp_d20030125_t084705_e084830_b00015_c20080131015912_gsfc_SCI.h5* for Ancillary file cd /pub/DailyIngest/20080401/Land ls mget off_NCEP-GFS-06HR-ANC_GFS_NCEP_006f_20020906_200209060600Z_20020906060000Z_ee20020906085959Z_np.grib2* for RDR, SDR, IP cd /pub/NPP_Products/VIIRS/20030125/IP/IICMO ls mget IICMO_npp_d20030125_t104352_e104517_b00016_c20080417111100_den_SWC.h5* cd /pub/NPP_Products/VIIRS/20030125/RDR/RVIRS ls mget RVIRS_npp_d20030125_t065434_e065559_b00014_c20080110165312_gsfc_SCI.h5* cd /pub/NPP_Products/VIIRS/20030125/SDR/SVI01/ mget SVI01_npp_d20030125_t084705_e084830_b00015_c20080131014006_gsfc_SCI.h5*

Sample of ncftp ls and mget outputs: ncftp ...lyIngest/20080331/Land > ls GMTCO_npp_d20030125_t084705_e084830_b00015_c20080131015913_gsfc_SCI.h5*

GMTCO_npp_d20030125_t084705_e084830_b00015_c20080131015913_gsfc_SCI.h5

GMTCO_npp_d20030125_t084705_e084830_b00015_c20080131015913_gsfc_SCI.h5.sig

ncftp ...lyIngest/20080331/Land > mget GMTCO_npp_d20030125_t084705_e084830_b00015_c20080131015913_gsfc_SCI.h5*

...084705_e084830_b00015_c20080131015913_gsfc_SCI.h5: 77.41 MB 86.62 MB/s

...05_e084830_b00015_c20080131015913_gsfc_SCI.h5.sig: 65.00 B 1.62 kB/s

Sample Procedure

Reporting Flow

Test Execution

Test Report & Verification Results

Test ManagerUpdates Test Matrix

Score Card Updated

Score Card Results Report Delivered to Management & Review Boards

Verification Progress as of 8/12/20081 of 3 Green = Passed

# of Req’s SD3E Atmosphere Land Ocean Ozone Sounder I&TSE NICSE CERES PSOE External 1. 3.1.1.1 3.4.1.1 3.4.1.1 3.4.1.1 3.4.1.1 3.4.1.1 3.5.1.1 3.3.1.1 3.2.1 3.6.1

2. 3.1.1.2 3.4.1.2 3.4.1.2 3.4.1.2 3.4.1.2 3.4.1.2 3.5.1.2 3.3.1.5 3.2.1.1 3.6.2

3. 3.1.1.3 3.4.1.3 3.4.1.3 3.4.1.3 3.4.1.3 3.4.1.3 3.5.2.1 3.3.1.10 3.2.1.2 3.6.3

4. 3.1.1.4 3.4.1.4 3.4.1.4 3.4.1.4 3.4.1.4 3.4.1.4 3.5.3.1 3.3.1.11 3.2.1.3 3.6.4

5. 3.1.1.5 3.4.1.5 3.4.1.5 3.4.1.5 3.4.1.5 3.4.1.5 3.5.4.1 3.3.1.15 3.2.1.4 3.6.5

6. 3.1.1.6 3.4.1.6 3.4.1.6 3.4.1.6 3.4.1.6 3.4.1.6 3.5.4.2 3.3.1.17 3.2.1.5 3.6.6

7. 3.1.1.7 3.4.1.7 3.4.1.7 3.4.1.7 3.4.1.7 3.4.1.7 3.5.5.1 3.3.2.1 3.2.2 3.6.7

8. 3.1.1.8 3.4.1.8 3.4.1.8 3.4.1.8 3.4.1.8 3.4.1.8 3.5.5.2 3.3.2.3 3.2.2.1 3.6.8

9. 3.1.1.9 3.4.1.9 3.4.1.9 3.4.1.9 3.4.1.9 3.4.1.9 3.5.5.3 3.3.2.5 3.2.2.2 3.6.9

10. 3.1.1.10 3.4.1.10 3.4.1.10 3.4.1.10 3.4.1.10 3.4.1.10 3.5.6.1 3.3.2.6 3.2.2.3 3.6.10

11. 3.1.1.11 3.4.1.11 3.4.1.11 3.4.1.11 3.4.1.11 3.4.1.11 3.5.6.2 3.3.2.7 3.2.3.1

12. 3.1.1.12 3.4.1.12 3.4.1.12 3.4.1.12 3.4.1.12 3.4.1.12 3.5.6.3 3.3.2.8 3.2.3.2

13. 3.1.1.13 3.4.1.13 3.4.1.13 3.4.1.13 3.4.1.13 3.4.1.13 3.3.2.10 3.2.3.3

14. 3.1.1.14 3.4.1.14 3.4.1.14 3.4.1.14 3.4.1.14 3.4.1.14 3.3.2.11 3.2.4.1

15. 3.1.1.15 3.4.1.15 3.4.1.15 3.4.1.15 3.4.1.15 3.4.1.15 3.3.3.1 3.2.4.2

16. 3.1.1.16 3.4.2.1 3.4.2.1 3.4.2.1 3.4.2.1 3.4.2.1 3.3.3.2

17. 3.1.1.17 3.4.3.1 3.4.3.1 3.4.3.1 3.4.3.1 3.4.3.1 3.3.3.4

18. 3.1.1.18 3.4.4.1 3.4.4.1 3.4.4.1 3.4.4.1 3.4.4.1 3.3.3.5

19. 3.1.2.1 3.4.4.2 3.4.4.2 3.4.4.2 3.4.4.2 3.4.4.2 3.3.3.6

20. 3.1.3.1 3.4.4.3 3.4.4.3 3.4.4.3 3.4.4.3 3.4.4.3 3.3.3.7

21. 3.1.3.2 3.4.4.4 3.4.4.4 3.4.4.4 3.4.4.5 3.4.4.5 3.3.3.8

22. 3.1.3.3 3.4.4.5 3.4.4.5 3.4.4.5 3.4.4.6 3.4.4.6 3.3.3.9

23. 3.1.3.4 3.4.4.6 3.4.4.6 3.4.4.6 3.4.4.7 3.4.4.7 3.3.4.1

24. 3.1.3.5 3.4.4.7 3.4.4.7 3.4.4.7 3.4.4.8 3.4.4.8 3.3.4.3

25. 3.1.3.6 3.4.4.8 3.4.4.8 3.4.4.8 3.4.4.9 3.4.4.9 3.3.4.5

26. 3.1.4.1 3.4.4.9 3.4.4.9 3.4.4.9 3.4.4.10 3.4.4.10 3.3.4.6

27. 3.1.4.2 3.4.4.10 3.4.4.10 3.4.4.10 3.4.4.11 3.4.4.11 3.3.4.7

28. 3.1.4.3 3.4.4.11 3.4.4.11 3.4.4.11 3.4.4.12 3.4.4.12 3.3.4.8

29. 3.1.4.4 3.4.4.12 3.4.4.12 3.4.4.12 3.4.4.13 3.4.4.13 3.7.1 30. 3.1.4.5 3.4.4.13 3.4.4.13 3.4.4.13 3.4.4.14 3.4.4.14 3.7.2 31. 3.1.4.6 3.4.4.14 3.4.4.14 3.4.4.14 3.4.4.15 3.4.4.15 3.7.3 32. 3.6.1 3.4.4.15 3.4.4.15 3.4.4.15 3.4.4.16 3.4.4.16 3.7.4 33. 3.6.2 3.4.4.16 3.4.4.16 3.4.4.16 3.4.4.17 3.4.4.17 3.7.5 34. 3.6.7 3.4.4.17 3.4.4.17 3.4.4.17 3.4.4.18 3.4.4.18 3.7.6 35. 3.8.1.1 3.4.4.18 3.4.4.18 3.4.4.18 3.4.4.19 3.4.4.19 3.7.8.1

36 3.8.1.2 3.4.4.19 3.4.4.19 3.4.4.19 3.4.4.20 3.4.4.20 3.7.8.2 37 3.8.1.3 3.4.4.20 3.4.4.20 3.4.4.20 3.4.4.21 3.4.4.21 3.7.8.3 38 3.8.1.4 3.4.4.21 3.4.4.21 3.4.4.21 3.4.4.22 3.4.4.22 3.7.8.4

39 3.8.2.1 3.4.4.22 3.4.4.22 3.4.4.22 3.4.5.1 3.4.5.1 3.7.9 40 3.8.3.1 3.4.4.23 3.4.4.23 3.4.4.23 3.4.5.2 3.4.5.2 41 3.7.1 3.4.4.24 3.4.4.24 3.4.4.24 3.4.9.1 3.4.10.1

42 3.7.2 3.4.5.1 3.4.5.1 3.4.5.1 3.4.9.2 3.4.10.2 43 3.7.3 3.4.5.2 3.4.5.2 3.4.5.2 3.4.9.4 3.4.10.3 44 3.7.4 3.4.6.1 3.4.7.1 3.4.8.1 3.4.9.5 3.7.1 45 3.7.5 3.4.6.2 3.4.7.2 3.4.8.2 3.4.9.4.1 3.7.2 46 3.7.6 3.7.1 3.7.1 3.7.1 3.4.9.4.2 3.7.3 47 3.7.7 3.7.2 3.7.2 3.7.3 3.4.9.4.3 3.7.4 48 3.7.8.1 3.7.3 3.7.3 3.7.4 3.4.9.4.4 3.7.5 49 3.7.8.2 3.7.4 3.7.4 3.7.5 3.4.9.4.5 3.7.6 50 3.7.8.3 3.7.5 3.7.5 3.7.6 3.4.9.4.6 3.7.7 51 3.7.8.4 3.7.6 3.7.6 3.7.7 3.4.9.4.7 3.7.8 52 3.7.9 3.7.7 3.7.7 3.7.8.1 3.7.1 3.7.8.1 53 3.7.1 3.7.8 3.7.8 3.7.8.2 3.7.2 3.7.8.2 54 3.7.8.1 3.7.8.1 3.7.8.3 3.7.3 3.7.8.3 55 3.7.8.2 3.7.8.2 3.7.8.4 3.7.4 3.7.8.4 56 3.7.8.3 3.7.8.3 3.7.9 3.7.5 3.7.9 57 3.7.8.4 3.7.8.4 3.7.6 58 3.7.9 3.7.9 3.7.7 59 3.7.8 60 3.7.8.1 61 3.7.8.2 62 3.7.8.3 63 3.7.8.4 64 3.7.9

# Verified 23 of 53 0 of 58 23 of 58 16 of 56 33 of 64 10 of 56 7 of 12 12 of 39 0 of? 0 of 15 0 of 10 SD3E Atmosphere Land Ocean Ozone Sounder I&TSE NICSE CERES PSOE External

Verification Progress as of 8/12/20082 of 3 Green = Passed

Verification Progress as of 8/12/20083 of 3

• The previous two slides show the following– The total number of requirements for each element– The Level 3 requirement ID number– The number of requirements verified to date highlighted in green– Other colors not yet used – Red = Failed, Blue = Deferred, Waived = Peach

Ocean PEATE Level 3’s

SD3E

NICSE

Ozone PEATE Sounder PEATE (JPL) I&TSE Level 3 Requirements

PSOE Level 3 Requirements

Algorithm EnhancementsIP Gen. Request

RDR, SDREDR, IP

RDR, SDREDR, IP

TDR, RDR, SDR,EDR, IP

RDR, SDR,EDR, IP

IDPS S/Wupdates

QualityControl,

Cal. LUTand S/Wupdates

EDRAssessmentAlgorithmEnhancements

Statusreports,Managementdirection

Status reports, Management direction

Status reports,Managementdirection

NPPScienceTeam

Not all data flows shown here.

SDS Score Card as of 08/12/2008

Algorithm EnhancementsIP Gen. Request

Level 3 Requirements

(T) - Total Requirements = 53(F) - Failed = 0(V) - Verified = 23(D) - Deferred or = 0 Workaround(W) - Waived = 0(%) - Percent Verified = 43.0%

(T) - Total Requirements = 64(F) - Failed = 0(V) - Verified = 33(D) - Deferred or = 0 Workaround(W) - Waived = 0(%) - Percent Verified = 52%

Level 3 Requirements

Level 3 RequirementsLevel 3 Requirements

Total SDS Level 3’s

(T) - Total Requirements = 411(F) - Failed = 0(V) - Verified = 124(D) - Deferred or = 0 Workaround(W) - Waived = 0

(%) - Percent Verified = 30.2%

Atmosphere PEATE (U of Wisconsin) Level 3’s(T) - Total Requirements = 58(F) - Failed = 0(V) - Verified = 0(D) - Deferred or = 0 Workaround(W) - Waived = 0(%) - Percent Verified = 0%

Land PEATE Level 3’s(T) - Total Requirements = 58(F) - Failed = 0 (V) - Verified = 23(D) - Deferred or = 0 Workaround(W) - Waived = 0(%) - Percent Verified = 40%


Score as Of08/12/2008





CERESLevel 3 Requirements

(T) - Total Req’s = TBD(F) - Failed = 0(V) – Verified = 0(D) - Deferred or = 0 Workaround(W) – Waived = 0(%) - Verified = 0

• The previous slide reports the level 3 progress by element and total SDS progress.

• Requirements status are scored:– Verified (worked as expected and planned)– Failed (discrepancies will identified and tracked using

Bugzilla)– Workaround (verified, but needed help – all

functionality not available)– Deferred (some may not be verified until after launch)– Waived (no-longer needed to support the mission at

this time)

SDS Score Card (page 2 of 2)

SDS Testing / Verification @ the CERES Delta Design Review

Documents

Transcript of SDS Testing / Verification @ the CERES Delta Design Review