NPP Product Validation for the Ozone Mapping and Profiler Suite (OMPS)
TEMPLATE DESIGN © 2008 Facilitation of OMPS Dark Table Production Transition to GRAVITE by STAR...
1
TEMPLATE DESIGN © 2008 www.PosterPresentations.com Facilitation of OMPS Dark Table Production Transition to GRAVITE by STAR Algorithm Integration Team (AIT) Bigyani Das 1 , Weizhong Chen 1 , Kristina Sprietzer 1 , Walter Wolf 2 1 IMSG, Rockville, MD 20852, USA 2 NOAA/NESDIS/STAR, College Park, MD 20740, USA Abstract The Center for Satellite Applications and Research (STAR) Algorithm Integration Team (AIT) was tasked to facilitate the Ozone Mapping and Profiler Suite (OMPS) Dark Table production transition process from the OMPS Product Evaluation and Analysis Tool Element (PEATE), at NASA Goddard Space Flight Center, to NOAA’s Government Resources for Algorithm Verification, Independent Test and Evaluation (GRAVITE) facility. In this work we present the process we have used to create OMPS Dark tables for OMPS Nadir Profiler (NP) and OMPS Nadir Mapper (NM) within STAR. OMPS Dark Tables Currently the Dark tables are produced at OMPS PEATE through a series of algorithms that are termed as Algorithm Plugin Package (APP). The Dark files in HDF format are processed at STAR to create binary Ground Tables that are delivered to IDPS. These Dark tables are weekly updated in the operation through a process that involves creating a CCR and going through AERB review. Update of weekly Dark tables is important for correct radiance values and accuracy of other downstream Ozone products. STAR AIT Steps Explained: Example Nadir Profiler Case STAR AIT provides expertise on integration of JPSS algorithms into operational systems and performs the following tasks: Code Testing and Integration in Algorithm Development Library (ADL), Communication with Science Teams and DPE, Troubleshooting, Change Request Submission, Consultancy to Science Teams and DPES Facilitation of Lifecycle Reviews Dark Table Production Steps The Dark production has 4 main processes: 1) Creating L1A files from Raw Data Record (RDR) files 2) Using those L1A files to produce Sensor Data Record (SDR) files 3) Using SDR files to create Dark files in HDF5 format 4) Using the Dark HDF5 files to produce the final binary files. The binary files are directly used operationally within NOAA's Interface Data Processing Segment (IDPS) facility. These Dark files are integrated into IDPS every week. OMPS Sunday orbits are used to produce the Dark files. Before starting the 4 main processes, installation of various libraries are needed. The process also needs the ORBDEF file to get information about orbit start and end times. Science Package/Library Version HDF5 1.6.7/1.8.3 SDP Toolkit 5.2.16 zlib 1.2.3 jpeg 6 szip 2.1 HDF4 4.2r4 HDF-EOS5 1.12 HDF-EOS2 16v1 flex 2.5.37 YAML 0.1.4 bison 2.5 XML2 2.7.3 XSLT 1.1.28 Python Module Version Docx 0.2.0 H5py 2.0.1 libxml2_python 2.6.9 lxml 3.1.1 Numpy 1.6.2 PyYAML 3.10 Perl Module Version YAML 0.84 YAML::Shell 0.6 Date::Format 1.2 Log::Log4perl 1.37 Hash::Merge 0.12 Utlity/Script Name AppTug askUser.sh AppConfigurationGlobal.txt wrapperMake SuperApp.mk Shared_Top.mk Shared_End.mk Makefile.config.$compiler Help.mk Compare.mk Makefile (master) config2IDL.pl h5svnExportAndRevKeywordSub.py Libraries for the Dark Production APPs: NP_L1A_CAL_NASA-0.9.70 NP_L1A_CAL_DIAG-0.9.70 OMPS_L1A_NASA-0.9.70 CONFIG_READER-0.9.19 MESSAGES-0.9.2 O3_PEATE_COMMON-0.9.14 YAML_For_APPs-0.9.5 PRODUCT_VERSIONING-1.0.0 Step 1: RDR to L1A – 8 APPs Other Tables: Eud/ naif0009.tls TNM.yml Step 2: L1A to SDR-CAL -15 APPs Main APPs: NP_SDR_CAL_DIAG_CHAIN – 0.9.82 NP_SDR_CAL_CHAIN – 0.9.82 CHAIN_Shr-0.9.10 COMMON_GEO-b.9.34 CONFIG_READER-0.9.15 MESSAGES-0.9.2 Nadir_BINNER-0.9.48 NP_GEO_NASA-0.9.30 NP_SDR_CAL_NASA-0.9.34 O3_PEATE_COMMON-0.9.11 O3_PEATE_SHR-0.9.10 PRODUCT_VERSIONING-0.9.12 SDR_CAL_NASA_Shr-0.9.53 SDR_NASA_Shr-0.9.20 TC_GEO_NASA-c.9.42 Inputs Outputs OMPS-NPP-NP_L1A_CAL_DIAG OMPS-NPP-NP_SDR_CAL_NASA- p061 & OMPS-NPP-NP_SDR_CAL_NASA- pxx OMPS-NPP-NP_L1A_CAL OMPS-NPP-NP_SDR_CAL_NASA- p058 Other Tables: HDF5 Files: OMPS-NPP-LCC OMPS-NPP-NP_DRK-p058 (old date) OMPS-NPP-NPP-NP_GON OMPS-NPP-NPP-NP_IRD OMPS-NPP-NPP-NPP-OPF OMPS-NPP-NPP-NP_STB OMPS-NPP-NP_SRG OMPS-NPP-TP YML Files: OMPS-NPP-TPV_ALT OMPS-NPP-TPV_NP_SDR_CAL_CHAIN (old) OMPS-NPP- TPV_NP_SDR_CAL_DIAG_CHAIN (old) Other Files: utcpole.dat leapsec.dat Step 3: SDR to Dark Table HDF5 File – 6 APPs Main App: NP_DARKCAL-0.9.24 DARKCAL-0.9.28 CONFIG_READER– 0.9.15 MESSAGES – 0.9.2 O3_PEATE_COMMON – 0.9.11 PRODUCT_VERSIONING-0.9.12 NP_Darks_LUT_For_IDPS (1.0.4) Independent APP Python Scripts Step 4: Dark Table HDF5 File to Binary File – 1 APP Inputs Outputs OMPS-NPP-NP_SDR_CAL_NASA- p061 OMPS-NPP-NP_SDR_CAL_NASA- p058 OMPS-NPP-NP_DRK-p058 Inputs Outputs OMPS-NPP-NP_DRK-p058 OMPS-NPP-NP_L1A_CAL OMPS-NPP-NPP-OMPS-NP- DARKS-LUT-RUNCONTROLTABLE OMPS-NP-DARKS-GND-PI_npp (.bin) OMPS-NPP-NPP-OMPS-NP- DARKS-GND-PI- DELIVERYLETTER (.docx) OMPS-NPP-NPP-OMPS-NP- DARKS-GND-PI-TARFILE (.tar) Wrapper Script: gravite_transition_dark_OMPS_processing.sh Master Script: dark_OMPS_setup_and_execute_scripts.pl User Input File: user_file.list Date Specific Configuration File: user_config_131006.cfg File for Runtime Parameters: runtime_parameters_for_control_file.txt Command to Execute the Script: ./gravite_transition_dark_OMPS_processing.sh user_file.list Script Functionality: 1 ) Reads in variables from Description.txt from the APP dir. 2 ) Reads in variables from user configuration file. 3 ) Copies files based on selected date, from archive_path directory, to the working directory. 4 ) Creates a control template file with variables necessary to run the OMPS program 5 ) Creates a run_tmp directory specific to an orbit number, in the location of the application directory 6 ) Creates a control file specific to the orbit number and places it in the run_tmp direcotry 7 ) Copies binaries, tables, LUTs and input files into the run_tmp directory 8 ) Executes application/PGE/product commands 9 ) Copies output data back to the archive directory. Subroutines: 33 read_in_Description_file get_input_files sub read_user_file{ sub print_out_control_file sub split_and_remove_space{ sub split_and_remove_space_2var{ sub mkdir_run_tmp{ sub copy_bin_tbl_luts_input_files_to_run_tmp sub check_return_code sub get_all_orbit_numbers sub get_file_names_for_profile_id_58_and_61 sub add_filename_to_profile_list sub copy_outputs_to_work_dir sub reset_hashes sub read_in_template_control_file sub read_in_runtime_parameters sub get_list_of_RONDC_files sub get_list_of_RONPC_files sub store_print_esdt_hash sub add_ProductionTime_to_orbit_control_file sub run_executable sub exit_and_print_date_format_message sub check_directory_variable_for_exist_and_blanks sub list_of_input_files_check_return_code_add_to_array sub create_symbolic_link sub fix_SAABox_line sub obtain_orbit_def_info sub check_user_file_input The Script 2015 NOAA Satellite Conference, April 27-May 1, Greenbelt Marriott Hotel in Greenbelt, Maryland 20740 Corresponding Author: [email protected] Poster # 2-11 STAR Design Process Get the APPs from OMPS PEATE Get the ORBDEF File Collect the needed Tables and YML files Install the needed libraries Test each step individually with test data from NASA PEATE Test each step individually using production from the upstream step in downstream step Write a master script to be able to string together the 4 steps, start with the RDR files and create the Ground PI file and delivery letter. Write a wrapper script to take care of everything, sets up the environment, and given a date and time, selects the orbits, the files and produces the Ground Pis. Inputs Outputs RONDC Files: OMPS Nadir Profile Diagnostic Calibration OMPS-NPP-NP_L1A_CAL_DIAG RONPC Files: OMPS Nadir Profile Calibration RDR OMPS-NPP-NP_L1A_CAL Abbreviations CCR: Configuration Change Request AERB: Algorithm Engineering Review Board
-
Upload
elfreda-fowler -
Category
Documents
-
view
214 -
download
1
Transcript of TEMPLATE DESIGN © 2008 Facilitation of OMPS Dark Table Production Transition to GRAVITE by STAR...
TEMPLATE DESIGN © 2008
www.PosterPresentations.com
Facilitation of OMPS Dark Table Production Transition to GRAVITE by STAR Algorithm Integration Team (AIT)
Bigyani Das1, Weizhong Chen1, Kristina Sprietzer1, Walter Wolf2
1IMSG, Rockville, MD 20852, USA2NOAA/NESDIS/STAR, College Park, MD 20740, USA
Abstract The Center for Satellite Applications and Research (STAR) Algorithm Integration Team (AIT) was tasked to facilitate the Ozone Mapping and Profiler Suite (OMPS) Dark Table production transition process from the OMPS Product Evaluation and Analysis Tool Element (PEATE), at NASA Goddard Space Flight Center, to NOAA’s Government Resources for Algorithm Verification, Independent Test and Evaluation (GRAVITE) facility. In this work we present the process we have used to create OMPS Dark tables for OMPS Nadir Profiler (NP) and OMPS Nadir Mapper (NM) within STAR.
OMPS Dark Tables
Currently the Dark tables are produced at OMPS PEATE through a series of algorithms that are termed as Algorithm Plugin Package (APP). The Dark files in HDF format are processed at STAR to create binary Ground Tables that are delivered to IDPS. These Dark tables are weekly updated in the operation through a process that involves creating a CCR and going through AERB review. Update of weekly Dark tables is important for correct radiance values and accuracy of other downstream Ozone products.
STAR AIT
Steps Explained: Example Nadir Profiler Case
STAR AIT provides expertise on integration of JPSS algorithms into operational systems and performs the following tasks:
Code Testing and Integration in Algorithm Development Library (ADL), Communication with Science Teams and DPE, Troubleshooting, Change Request Submission, Consultancy to Science Teams and DPES
Facilitation of Lifecycle Reviews
Dark Table Production StepsThe Dark production has 4 main processes:1) Creating L1A files from Raw Data Record (RDR) files 2) Using those L1A files to produce Sensor Data Record (SDR) files3) Using SDR files to create Dark files in HDF5 format 4) Using the Dark HDF5 files to produce the final binary files. The binary
files are directly used operationally within NOAA's Interface Data Processing Segment (IDPS) facility.
These Dark files are integrated into IDPS every week. OMPS Sunday orbits are used to produce the Dark files.
Before starting the 4 main processes, installation of various libraries are needed. The process also needs the ORBDEF file to get information about orbit start and end times.
Science Package/Library Version HDF5 1.6.7/1.8.3 SDP Toolkit 5.2.16 zlib 1.2.3 jpeg 6 szip 2.1 HDF4 4.2r4 HDF-EOS5 1.12 HDF-EOS2 16v1 flex 2.5.37 YAML 0.1.4 bison 2.5 XML2 2.7.3XSLT 1.1.28
Python Module Version Docx 0.2.0H5py 2.0.1libxml2_python 2.6.9lxml 3.1.1Numpy 1.6.2PyYAML 3.10
Perl Module Version YAML 0.84 YAML::Shell 0.6 Date::Format 1.2 Log::Log4perl 1.37 Hash::Merge 0.12
Utlity/Script Name AppTug askUser.sh AppConfigurationGlobal.txt wrapperMake SuperApp.mk Shared_Top.mk Shared_End.mk Makefile.config.$compiler Help.mk Compare.mk Makefile (master) config2IDL.pl h5svnExportAndRevKeywordSub.py
Libraries for the Dark Production
APPs:NP_L1A_CAL_NASA-0.9.70NP_L1A_CAL_DIAG-0.9.70 OMPS_L1A_NASA-0.9.70CONFIG_READER-0.9.19MESSAGES-0.9.2O3_PEATE_COMMON-0.9.14YAML_For_APPs-0.9.5PRODUCT_VERSIONING-1.0.0
Step 1: RDR to L1A – 8 APPs
Other Tables:Eud/naif0009.tlsTNM.yml
Step 2: L1A to SDR-CAL -15 APPsMain APPs:NP_SDR_CAL_DIAG_CHAIN – 0.9.82NP_SDR_CAL_CHAIN – 0.9.82
CHAIN_Shr-0.9.10COMMON_GEO-b.9.34CONFIG_READER-0.9.15MESSAGES-0.9.2Nadir_BINNER-0.9.48NP_GEO_NASA-0.9.30NP_SDR_CAL_NASA-0.9.34O3_PEATE_COMMON-0.9.11O3_PEATE_SHR-0.9.10PRODUCT_VERSIONING-0.9.12SDR_CAL_NASA_Shr-0.9.53SDR_NASA_Shr-0.9.20TC_GEO_NASA-c.9.42
Inputs OutputsOMPS-NPP-NP_L1A_CAL_DIAG OMPS-NPP-NP_SDR_CAL_NASA-p061
&OMPS-NPP-NP_SDR_CAL_NASA-pxx
OMPS-NPP-NP_L1A_CAL OMPS-NPP-NP_SDR_CAL_NASA-p058
Other Tables:HDF5 Files:OMPS-NPP-LCCOMPS-NPP-NP_DRK-p058 (old date)OMPS-NPP-NPP-NP_GONOMPS-NPP-NPP-NP_IRDOMPS-NPP-NPP-NPP-OPFOMPS-NPP-NPP-NP_STBOMPS-NPP-NP_SRGOMPS-NPP-TP
YML Files:OMPS-NPP-TPV_ALTOMPS-NPP-TPV_NP_SDR_CAL_CHAIN (old)OMPS-NPP-TPV_NP_SDR_CAL_DIAG_CHAIN (old)
Other Files:utcpole.datleapsec.dat
Step 3: SDR to Dark Table HDF5 File – 6 APPsMain App: NP_DARKCAL-0.9.24 DARKCAL-0.9.28 CONFIG_READER– 0.9.15 MESSAGES – 0.9.2 O3_PEATE_COMMON – 0.9.11PRODUCT_VERSIONING-0.9.12
NP_Darks_LUT_For_IDPS (1.0.4)Independent APPPython Scripts
Step 4: Dark Table HDF5 File to Binary File – 1 APP
Inputs Outputs
OMPS-NPP-NP_SDR_CAL_NASA-p061OMPS-NPP-NP_SDR_CAL_NASA-p058
OMPS-NPP-NP_DRK-p058
Inputs Outputs
OMPS-NPP-NP_DRK-p058OMPS-NPP-NP_L1A_CAL
OMPS-NPP-NPP-OMPS-NP-DARKS-LUT-RUNCONTROLTABLE
OMPS-NP-DARKS-GND-PI_npp (.bin)
OMPS-NPP-NPP-OMPS-NP-DARKS-GND-PI-DELIVERYLETTER (.docx)
OMPS-NPP-NPP-OMPS-NP-DARKS-GND-PI-TARFILE (.tar)
Wrapper Script: gravite_transition_dark_OMPS_processing.shMaster Script: dark_OMPS_setup_and_execute_scripts.plUser Input File: user_file.listDate Specific Configuration File: user_config_131006.cfgFile for Runtime Parameters: runtime_parameters_for_control_file.txt
Command to Execute the Script: ./gravite_transition_dark_OMPS_processing.sh user_file.list
Script Functionality:1 ) Reads in variables from Description.txt from the APP dir.2 ) Reads in variables from user configuration file.3 ) Copies files based on selected date, from archive_path directory, to the working directory.4 ) Creates a control template file with variables necessary to run the OMPS program5 ) Creates a run_tmp directory specific to an orbit number, in the location of the application directory6 ) Creates a control file specific to the orbit number and places it in the run_tmp direcotry 7 ) Copies binaries, tables, LUTs and input files into the run_tmp directory8 ) Executes application/PGE/product commands9 ) Copies output data back to the archive directory.
Subroutines: 33read_in_Description_fileget_input_filessub read_user_file{sub print_out_control_filesub split_and_remove_space{sub split_and_remove_space_2var{sub mkdir_run_tmp{sub copy_bin_tbl_luts_input_files_to_run_tmp sub check_return_codesub get_all_orbit_numberssub get_file_names_for_profile_id_58_and_61sub add_filename_to_profile_listsub copy_outputs_to_work_dirsub reset_hashessub read_in_template_control_filesub read_in_runtime_parameterssub get_list_of_RONDC_filessub get_list_of_RONPC_filessub store_print_esdt_hashsub add_ProductionTime_to_orbit_control_filesub run_executablesub exit_and_print_date_format_messagesub check_directory_variable_for_exist_and_blankssub list_of_input_files_check_return_code_add_to_arraysub create_symbolic_linksub fix_SAABox_linesub obtain_orbit_def_infosub check_user_file_inputsub get_previous_chain_run_datasub specific_input_for_product_namesub output_input_names_modify_variables_insidesub handle_TPV_product_name_filessub StartTime_for_file_name
The Script
2015 NOAA Satellite Conference, April 27-May 1, Greenbelt Marriott Hotel in Greenbelt, Maryland 20740 Corresponding Author: [email protected]
Poster # 2-11
STAR Design ProcessGet the APPs from OMPS PEATEGet the ORBDEF FileCollect the needed Tables and YML filesInstall the needed librariesTest each step individually with test data from NASA PEATETest each step individually using production from the upstream step in downstream stepWrite a master script to be able to string together the 4 steps, start with the RDR files and
create the Ground PI file and delivery letter.Write a wrapper script to take care of everything, sets up the environment, and given a date
and time, selects the orbits, the files and produces the Ground Pis.Inputs OutputsRONDC Files: OMPS Nadir Profile Diagnostic Calibration
OMPS-NPP-NP_L1A_CAL_DIAG
RONPC Files: OMPS Nadir Profile Calibration RDR
OMPS-NPP-NP_L1A_CAL
AbbreviationsCCR: Configuration Change RequestAERB: Algorithm Engineering Review Board
