Radar Data Quality Control
description
Transcript of Radar Data Quality Control
![Page 1: Radar Data Quality Control](https://reader033.fdocuments.us/reader033/viewer/2022061610/56816781550346895ddc8c81/html5/thumbnails/1.jpg)
Radar Data Quality Control
Warn-on-Forecast & High Impact Weather Workshop8 February 2012
Kevin L. ManrossOU/CIMMS/NOAA/NSSL
![Page 2: Radar Data Quality Control](https://reader033.fdocuments.us/reader033/viewer/2022061610/56816781550346895ddc8c81/html5/thumbnails/2.jpg)
Importance of Radar QC
• Radar data assimilation: “Garbage in – garbage out… on steroids”– “Any source of data bias will cause bias in the
resulting analysis, even if it is localized in space. For example, poorly removed clutter causes problems when assimilating radial velocity, as these cause errors in the obtained winds that persist in time and may ultimately falsely trigger instabilities…” Fabry (2011 Radar Conference)
![Page 3: Radar Data Quality Control](https://reader033.fdocuments.us/reader033/viewer/2022061610/56816781550346895ddc8c81/html5/thumbnails/3.jpg)
2½ Radar QC Techniques
• Multiple ways to skin a cat– There are a number of ways to QC Radar data
• Manually• Automated
• RDA vs RPG– Signal Processing– After products Generated
• Control– End user– Data Collector (radar operator)
• Combination
![Page 4: Radar Data Quality Control](https://reader033.fdocuments.us/reader033/viewer/2022061610/56816781550346895ddc8c81/html5/thumbnails/4.jpg)
1) RDA QC
Operate on spectral data at the Radar Data Acquisition (RDA) step
(timeseries” or “Level I”)
Examples• Notch Filter• Clutter Mitigation Decision (CMD)
Algorithm• Gaussian Model Adaptive Processing
(GMAP)• Staggered Pulse Repetition Time
(SPRT)• Phase coding• Many others• Controlled by radar operator*
*If implemented, If operator considers need, If operator trained to do so, If…
![Page 5: Radar Data Quality Control](https://reader033.fdocuments.us/reader033/viewer/2022061610/56816781550346895ddc8c81/html5/thumbnails/5.jpg)
2) Product Algorithms (After the RPG)
• Operate on “products” (Reflectivity, Velocity, Correlation Coefficient, etc.) after the Radar Product Generation (RPG) step
• End user enhancements (Control)
• Operates on gridded data
• ExamplesDealiasing (Legacy, Other)Clutter removal (AP-Remove, QCNN, CREM)
• This is where our focus will be
![Page 6: Radar Data Quality Control](https://reader033.fdocuments.us/reader033/viewer/2022061610/56816781550346895ddc8c81/html5/thumbnails/6.jpg)
2.5) Dual-Pol
• New RDA• New Products• Great at discriminating
non-hydrometeors / hydrometeors– IDs ground clutter and
biologicals very well• Game Changer (moving
forward)
KMHX21:50Z, 10/19/2011
![Page 7: Radar Data Quality Control](https://reader033.fdocuments.us/reader033/viewer/2022061610/56816781550346895ddc8c81/html5/thumbnails/7.jpg)
Current Radar QC on Products
• Efforts at CAPS• Efforts in DART• NSSL MRMS
– Comparison
![Page 8: Radar Data Quality Control](https://reader033.fdocuments.us/reader033/viewer/2022061610/56816781550346895ddc8c81/html5/thumbnails/8.jpg)
RADAR QC AT CAPS
![Page 9: Radar Data Quality Control](https://reader033.fdocuments.us/reader033/viewer/2022061610/56816781550346895ddc8c81/html5/thumbnails/9.jpg)
Reflectivity Quality Control Flowchart
Read Tilt
Anomalous Radial Removal
Despeckle &Median Filter (opt)
Assemble Volume
Ground Clutter Removal
Despeckle
Continue to Remapping
For All Elev < 1.0
All Tilts
![Page 10: Radar Data Quality Control](https://reader033.fdocuments.us/reader033/viewer/2022061610/56816781550346895ddc8c81/html5/thumbnails/10.jpg)
KDDC Sunset & Clutter
RawObs
AnomalousRadialRemoved
Clutter Removal
![Page 11: Radar Data Quality Control](https://reader033.fdocuments.us/reader033/viewer/2022061610/56816781550346895ddc8c81/html5/thumbnails/11.jpg)
Radial Velocity Quality Control Flowchart
Read Tilt
Spectrum Width Filter
Assemble Volume
Ground Clutter Removal
Despeckle
Continue to Unfolding
For All Elev < 1.0
Despeckle &Median Filter (opt)
All Tilts
![Page 12: Radar Data Quality Control](https://reader033.fdocuments.us/reader033/viewer/2022061610/56816781550346895ddc8c81/html5/thumbnails/12.jpg)
Radial Velocity Quality Control Flowchart
Compare to mean wind
Gate-to-Gate Shear Check
Quadratic Check atGates Marked
Uncertain
Calculate MeanWind Profile
Model Data or
Sounding …Continued
Create perturbation Vr Field
Mean Wind Profile
Continue to Remapping
![Page 13: Radar Data Quality Control](https://reader033.fdocuments.us/reader033/viewer/2022061610/56816781550346895ddc8c81/html5/thumbnails/13.jpg)
KTLX 10 May 2010 6.5° Scan
Raw Obs
Mean Wind
ShearCheck
Quad Fit
![Page 14: Radar Data Quality Control](https://reader033.fdocuments.us/reader033/viewer/2022061610/56816781550346895ddc8c81/html5/thumbnails/14.jpg)
DART RADAR QC
![Page 15: Radar Data Quality Control](https://reader033.fdocuments.us/reader033/viewer/2022061610/56816781550346895ddc8c81/html5/thumbnails/15.jpg)
Radar-Data Quality Control inData Assimilation Research Testbed (DART) System
• Observation rejection– Observation likelihoods more than a specified number of ensemble standard
deviations away from the prior ensemble mean are not assimilated.• Factors: observation, observation error, ensemble mean, ensemble standard deviation
• Doppler-velocity dealiasing (Miller et al. 1986; Dowell et al. 2010)– Velocities are locally dealiased during preprocessing (e.g., objective analysis).– Final dealiasing occurs within DART immediately before the observation is
assimilated (i.e., the observation is unfolded into the Nyquist-velocity bin closest to the prior ensemble mean).
locally-unfolded,objectively-analyzed
Doppler velocity beforefinal DART dealiasing
![Page 16: Radar Data Quality Control](https://reader033.fdocuments.us/reader033/viewer/2022061610/56816781550346895ddc8c81/html5/thumbnails/16.jpg)
NSSL’S MR/MS
![Page 17: Radar Data Quality Control](https://reader033.fdocuments.us/reader033/viewer/2022061610/56816781550346895ddc8c81/html5/thumbnails/17.jpg)
For Kevin Manross 22
NMQ “Bloom/AP Removal” Flowchart
Reference: Tang et al. 2011 http://ams.confex.com/ams/35Radar/webprogram/Paper191296.html 2/2/12
![Page 18: Radar Data Quality Control](https://reader033.fdocuments.us/reader033/viewer/2022061610/56816781550346895ddc8c81/html5/thumbnails/18.jpg)
For Kevin Manross 23
NMQ Bloom/AP QC example: KCRP & KBRO 06:50Z, 10/13/2011
QCNNRAW BloomAP_QC
RAW BloomAP_QC
2/2/12
![Page 19: Radar Data Quality Control](https://reader033.fdocuments.us/reader033/viewer/2022061610/56816781550346895ddc8c81/html5/thumbnails/19.jpg)
For Kevin Manross 24
NMQ Remaining challenges: bloom/AP mixed with rain
2/2/12
![Page 20: Radar Data Quality Control](https://reader033.fdocuments.us/reader033/viewer/2022061610/56816781550346895ddc8c81/html5/thumbnails/20.jpg)
Implementation and Comparison of Techniques
• Several techniques identified and implemented to be run in realtime
• Manually cleaned cases• Comparison method:
– Compare • algorithm to raw (unedited)• algorithm to truth (manually edited)• Do gate-by-gate for every elevation scan available• Track gates removed/added/changed
![Page 21: Radar Data Quality Control](https://reader033.fdocuments.us/reader033/viewer/2022061610/56816781550346895ddc8c81/html5/thumbnails/21.jpg)
Techniques ImplementedLabel Concept Z/V Strengths Weaknesses
AP-Remove Use 3D structure to determine precipitating echoes from AP Z Works well in precip, and sea
clutterStruggles with widespread/strong clear air echo; removes fine features such as gust fronts
QCNN Neural-net to distinguish between good and bad echoes Z Robust and , and can
incorporate neighboring radars; holistic approach for spatial contiguity
Takes a while to train neural net; multi-radar errors are additive; may need to be retrained for climatology
CREM Realtime clutter map collected during non-clear air sensing Z Adaptable; rerun periodically to
update clutter mapsClutter map needs to probably be run frequently in transition seasons
Legacy Dealias Check against neighboring (previous) radials V Simple and fast; can effectively
incorporate wind profile for improved accuracy
Failures compound; struggles in sparse data areas, particularly near echo tops
2D-Dealias 2D least mean squares run on entire elevation scan V Simple; removes “noisy” velocity
fields (seen in upper tilts); being implemented by NEXRAD
Assumption of smooth field; fails in strong shear (though upgraded improvement)
AR-VAD hi-res VAD, essentially performed at each gate V Good correction without false
dealiasingRejects data in sharp inversions; requires adequate data coverage for VAD (fails at long range isolated cells)
![Page 22: Radar Data Quality Control](https://reader033.fdocuments.us/reader033/viewer/2022061610/56816781550346895ddc8c81/html5/thumbnails/22.jpg)
Cases• Using SOLOii to manually edit• Students trained
• 20090605 (VORTEX2 – strongly tornadic)– KCYS (~21-00z; 40 vol. scans; 558 elev. scans)– KFTG (~21-00z; 36 vol. scans; 502 elev. scans)
• 20090611 (VORTEX2 – weakly tornadic)– KPUX (~22-01z; 39 vol. scans; 544 elev. scans)– KGLD (~23-01z; 26 vol. scans; 362 elev. scans)
• 20110524 (strongly tornadic)– KTLX✪ (~20-22z; X vol. scans; Y elev. scans)– KFDR✪(~20-22z; X vol. scans; Y elev. scans)– MPAR (~20-22z; 108 vol. scans*; 1512 elev. Scans)
✪ In progress* Up to 19.5 deg elevation
![Page 23: Radar Data Quality Control](https://reader033.fdocuments.us/reader033/viewer/2022061610/56816781550346895ddc8c81/html5/thumbnails/23.jpg)
Reflectivity QC
QCNN
CREM/QCNN
Algo-Raw Truth-Algo
![Page 24: Radar Data Quality Control](https://reader033.fdocuments.us/reader033/viewer/2022061610/56816781550346895ddc8c81/html5/thumbnails/24.jpg)
Reflectivity QC
DIFFERENCE N hit miss fa ch
Truth-Raw 102,881,778 41,719,382 706 61,161,676 14
Truth-QCNN 45,372,205 37,428,076 4,198,257 3,745,858 14
Truth-CREM 44,881,268 33,740,708 5,439,618 3,321,613 2,379,329
QCNN-Raw 102,615,371 41,173,948 0 61,441,423 0
CREM-Raw 102,434,663 36,697,101 0 62,993,013 2,744,549
![Page 25: Radar Data Quality Control](https://reader033.fdocuments.us/reader033/viewer/2022061610/56816781550346895ddc8c81/html5/thumbnails/25.jpg)
Velocity QC
2D
Legacy0.5 deg
4.0 deg
2D Dealiasing Legacy vs 2D
![Page 26: Radar Data Quality Control](https://reader033.fdocuments.us/reader033/viewer/2022061610/56816781550346895ddc8c81/html5/thumbnails/26.jpg)
Velocity QC
DIFFERENCE N hit miss fa
2D-Raw62,898,61
6 61,036,300 1,697,813 164,503
Legacy-Raw62,898,61
6 61,039,078 1,696,103 163,435
Truth-2D62,899,32
3 62,881,353 6,754 8,587
Truth-Legacy62,899,32
3 62,873,003 10,982 12,675
Truth-Raw62,899,32
3 61,033,726 1,698,514 167,063
![Page 27: Radar Data Quality Control](https://reader033.fdocuments.us/reader033/viewer/2022061610/56816781550346895ddc8c81/html5/thumbnails/27.jpg)
Future Work
• Xu’s AR-VAD method
![Page 28: Radar Data Quality Control](https://reader033.fdocuments.us/reader033/viewer/2022061610/56816781550346895ddc8c81/html5/thumbnails/28.jpg)
Variational Dealiasing Method
Alias operator: vro = Z[vrt + o, vN]
First guess b from combined AR-VAD analysis.
Analysis a minimizes J = (a-b)TB-1(a-b) + ∑i{Z[Hia - vroi, vN]}2/o2
with vroi = vro(fi) filtered by Z[Hib – vroi, vN] ≤ (1 - a)vN,
where a = ¾, ½, ¼ in iteration 1, 2, 3. (Xu et al. 2009a,b Tellus)
vro
vr+ovN
-vN
b
a
vro
-vN
vNIce storm case at 04:36UTC on 1/29/09vro at 1.5o from KTLX with vN = 11.5 m/s
raw obs dealiased
Xu et al. 2011, 2012 JTech (X11, X12 hereafter)
Illustrative example:
![Page 29: Radar Data Quality Control](https://reader033.fdocuments.us/reader033/viewer/2022061610/56816781550346895ddc8c81/html5/thumbnails/29.jpg)
Multi-Step Hybrid Dealiasing Method for fine-scale vortices
Basic idea
Use different techniques for different scales and structures as listed below:
1. Variational dealiasing of X11 for broad areas, but flag local misfit on each tilt;
2. Block-to-point continuity check of X12 for local misfit, but flag discontinuities;
3. Beam-to-beam discontinuity check for small areas with discontinuities.
Tornadic case at 22:41UTC on 5/24/2011 vro at 0.5o from KTLX with vN = 28 m/s
Dealiased in step 3Dealiased in step 1
Norman
raw obs
![Page 30: Radar Data Quality Control](https://reader033.fdocuments.us/reader033/viewer/2022061610/56816781550346895ddc8c81/html5/thumbnails/30.jpg)
Future Work
• Dual-Pol• SPRT• If/When implemented, future is bright!
![Page 31: Radar Data Quality Control](https://reader033.fdocuments.us/reader033/viewer/2022061610/56816781550346895ddc8c81/html5/thumbnails/31.jpg)
Questions