Pipeline: A data integration workload unit in Azure Data ...
Data Pipeline
description
Transcript of Data Pipeline
Data Pipeline
Wei Zheng
Johns Hopkins University
Granada
Requirements
• Science images for ACS/WFC, WFC3/UVIS and WFC3/IR
• Detection image over all bands• Filter Images with a common WCS grid• Source catalogs• Photometric redshift catalogs
September 20, 2010
Granada
APSIS
• Image registration • Sky subtraction• Distortion correction• Cosmic ray rejection• Edge mask• Astrometric correction• Detection image
September 20, 2010
• Extinction correction• Extractor source
catalogs• Multicolor catalog• BPZ catalog
Granada
Modification
• ACEX: ACS image drizzling• WFEX: WFC3 image (UVIS and IR) drizzling• DREX: Register images of different passbands
September 20, 2010
Check data on CLASH central storage via fast-track ftp No
All data present?
Flowchart
Yes WFC3 or ACS? Run ACEX
Run WFEX
ACS
WFC3
Run ACEX
Run WFEX
Image Align
Image Combine
.felt files
Image Align
Image Combine
Run DREX
(0.065”)
Master Detection
Image
Co-added images in
native resolution
Extractor
Source Catalogs in
each passband
Multi-color catalog
PHOTO-Z Code(s)
Co-added images on
common pixel grid
PHOTO-Z Catalog
CLASH Science Team Data Archive
Granada
Astrometric Accuracy
September 20, 2010
WFEX WFC3RED
Granada
Pipeline Output: Images
• Detection Image (0.065” scale)• Combined images in each band (0.065” scale)• Combined UVIS and WFC images in original
scale
September 20, 2010
Granada
Pipeline Output: Catalogs
• Running Sextractor• Source catalogs in each band• Multicolor catalog• Photometric redshifts • Mag_iso, mag_auto, and mag_aper at various
sizes.
September 20, 2010
Granada
Processing Plan
• One run for every visit• Accumulated data for each version• Typically eight versions of processed data per
cluster• The final version has the best quality
September 20, 2010
Granada
Further improvements
• Slight alignment on drizzled images• Flag and mask consolidation• CTE correction for ACS• Data server
September 20, 2010
Granada
Summary
• A working pipeline • Need more in-depth comparisons and quality
checks• Need to accommodate new science
requirements
September 20, 2010
Granada
Many Thanks!
• Amit Saraff• John Blakeslee, Larry Bradley• Warren Hack, Andy Fruchter
September 20, 2010