Flagging: When Good Data Go Bad Scott Schnee & Amy Kimball Nov 9, 2011 Interferometry Discussion...
-
Upload
cecil-campbell -
Category
Documents
-
view
216 -
download
0
description
Transcript of Flagging: When Good Data Go Bad Scott Schnee & Amy Kimball Nov 9, 2011 Interferometry Discussion...
![Page 1: Flagging: When Good Data Go Bad Scott Schnee & Amy Kimball Nov 9, 2011 Interferometry Discussion Group.](https://reader034.fdocuments.us/reader034/viewer/2022051710/5a4d1af37f8b9ab05997f6f8/html5/thumbnails/1.jpg)
Flagging: When Good Data Go Bad
Scott Schnee & Amy KimballNov 9, 2011
Interferometry Discussion Group
![Page 2: Flagging: When Good Data Go Bad Scott Schnee & Amy Kimball Nov 9, 2011 Interferometry Discussion Group.](https://reader034.fdocuments.us/reader034/viewer/2022051710/5a4d1af37f8b9ab05997f6f8/html5/thumbnails/2.jpg)
Initial Flagging
• Shadowing• Pointing Errors– Reported– Unreported
• Observing Log• Other obvious problems
![Page 3: Flagging: When Good Data Go Bad Scott Schnee & Amy Kimball Nov 9, 2011 Interferometry Discussion Group.](https://reader034.fdocuments.us/reader034/viewer/2022051710/5a4d1af37f8b9ab05997f6f8/html5/thumbnails/3.jpg)
Initial Flagging
• Shadowing– Issue at low elevations– Issue for compact arrays
In CASA for an ALMA data set:flagdata(vis=‘vis.ms’, mode=‘shadow’, diameter=12.0)
In MIRIAD for a CARMA data set:csflag(vis=‘vis.ms’, carma=true)
![Page 4: Flagging: When Good Data Go Bad Scott Schnee & Amy Kimball Nov 9, 2011 Interferometry Discussion Group.](https://reader034.fdocuments.us/reader034/viewer/2022051710/5a4d1af37f8b9ab05997f6f8/html5/thumbnails/4.jpg)
Initial Flagging
• Pointing Errors– Reported (e.g. wind)– Unreported (e.g. pointing model)
In MIRIAD for a CARMA data set:uvflag(vis=‘vis.ms’, select=“pointing(5,100000)”, flagval=flag)uvflag(vis=‘vis.ms’, select=“el(85,90)”, flagval=flag)
![Page 5: Flagging: When Good Data Go Bad Scott Schnee & Amy Kimball Nov 9, 2011 Interferometry Discussion Group.](https://reader034.fdocuments.us/reader034/viewer/2022051710/5a4d1af37f8b9ab05997f6f8/html5/thumbnails/5.jpg)
Initial Flagging
• Observing Log
Many observatories will note problems that affect the system for part or all of a track. This could be weather-related or hardware-related, and often requires that some data be flagged.
![Page 6: Flagging: When Good Data Go Bad Scott Schnee & Amy Kimball Nov 9, 2011 Interferometry Discussion Group.](https://reader034.fdocuments.us/reader034/viewer/2022051710/5a4d1af37f8b9ab05997f6f8/html5/thumbnails/6.jpg)
Initial Flagging
• Other obvious problemsSource Sys Temps (K) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15URANUS 235 216 1443 327 369 290 150 182 195 165 158 188 184 980 2043C84 183 163 1477 257 308 225 103 130 136 110 107 131 126 880 1203C111 187 167 1106 263 314 231 100 133 141 113 110 135 129 892 125L1451MM 183 165 1675 261 314 224 105 132 138 118 107 134 128 883 131L1451MM 184 164 1445 259 310 226 103 131 137 110 108 133 127 887 1280336+323 185 164 1327 258 311 226 107 132 139 112 9820 133 127 887 1223C111 185 165 1231 262 312 230 102 133 139 113 9449 134 129 884 1353C111 184 164 1320 258 311 225 103 131 136 114 9979 133 126 890 131L1451MM 181 162 1412 258 309 225 107 130 135 112 9928 131 126 876 121L1451MM 182 163 1318 257 308 226 106 130 136 110 105 131 126 892 1180336+323 182 163 1406 257 309 225 103 130 135 112 105 131 125 878 1213C111 183 163 1506 259 309 226 106 132 136 115 106 132 127 895 114
![Page 7: Flagging: When Good Data Go Bad Scott Schnee & Amy Kimball Nov 9, 2011 Interferometry Discussion Group.](https://reader034.fdocuments.us/reader034/viewer/2022051710/5a4d1af37f8b9ab05997f6f8/html5/thumbnails/7.jpg)
Initial Flagging• Other obvious problems
Tsys plots from NGC 3256 ALMA CASA Guide
![Page 8: Flagging: When Good Data Go Bad Scott Schnee & Amy Kimball Nov 9, 2011 Interferometry Discussion Group.](https://reader034.fdocuments.us/reader034/viewer/2022051710/5a4d1af37f8b9ab05997f6f8/html5/thumbnails/8.jpg)
What to look for?
• Plots of amplitude and phase vs time and channel• Iterate over– Antenna– Spectral window– Source
• Make plots of bandpass and gain calibrators first– Easy to find bad data of a bright point source– Hard to find bad data of a faint extended source
![Page 9: Flagging: When Good Data Go Bad Scott Schnee & Amy Kimball Nov 9, 2011 Interferometry Discussion Group.](https://reader034.fdocuments.us/reader034/viewer/2022051710/5a4d1af37f8b9ab05997f6f8/html5/thumbnails/9.jpg)
What to look for?
• Smoothly varying phases and amplitudes can be calibrated
• Discontinuities can not be calibrated• Features in the calibrators that may not be in the
target data can cause problems
![Page 10: Flagging: When Good Data Go Bad Scott Schnee & Amy Kimball Nov 9, 2011 Interferometry Discussion Group.](https://reader034.fdocuments.us/reader034/viewer/2022051710/5a4d1af37f8b9ab05997f6f8/html5/thumbnails/10.jpg)
Amplitude vs TimeFrom TW Hydra ALMA GuideColor: PolarizationOne spectral window (spw) plotted
![Page 11: Flagging: When Good Data Go Bad Scott Schnee & Amy Kimball Nov 9, 2011 Interferometry Discussion Group.](https://reader034.fdocuments.us/reader034/viewer/2022051710/5a4d1af37f8b9ab05997f6f8/html5/thumbnails/11.jpg)
Locating the Bad Data in plotms
Draw a box around the suspected bad data.
![Page 12: Flagging: When Good Data Go Bad Scott Schnee & Amy Kimball Nov 9, 2011 Interferometry Discussion Group.](https://reader034.fdocuments.us/reader034/viewer/2022051710/5a4d1af37f8b9ab05997f6f8/html5/thumbnails/12.jpg)
Locating the Bad Data in plotms
Click locate and CASA will send information about thedata to the logger.
![Page 13: Flagging: When Good Data Go Bad Scott Schnee & Amy Kimball Nov 9, 2011 Interferometry Discussion Group.](https://reader034.fdocuments.us/reader034/viewer/2022051710/5a4d1af37f8b9ab05997f6f8/html5/thumbnails/13.jpg)
Locating the Bad Data in plotms
Bad data can be flagged by pressing this button or using the flagdata tast at the CASA prompt.
![Page 14: Flagging: When Good Data Go Bad Scott Schnee & Amy Kimball Nov 9, 2011 Interferometry Discussion Group.](https://reader034.fdocuments.us/reader034/viewer/2022051710/5a4d1af37f8b9ab05997f6f8/html5/thumbnails/14.jpg)
Locating the Bad Data in plotms
Flagger’s remorse can be corrected by unflagging good data
![Page 15: Flagging: When Good Data Go Bad Scott Schnee & Amy Kimball Nov 9, 2011 Interferometry Discussion Group.](https://reader034.fdocuments.us/reader034/viewer/2022051710/5a4d1af37f8b9ab05997f6f8/html5/thumbnails/15.jpg)
From TW Hydra CASA GuideBrown and Green show phase calibratorsOrange shows TW Hya
Amplitude vs FrequencyBirdies
![Page 16: Flagging: When Good Data Go Bad Scott Schnee & Amy Kimball Nov 9, 2011 Interferometry Discussion Group.](https://reader034.fdocuments.us/reader034/viewer/2022051710/5a4d1af37f8b9ab05997f6f8/html5/thumbnails/16.jpg)
Amplitude vs FrequencyEdge Channels
![Page 17: Flagging: When Good Data Go Bad Scott Schnee & Amy Kimball Nov 9, 2011 Interferometry Discussion Group.](https://reader034.fdocuments.us/reader034/viewer/2022051710/5a4d1af37f8b9ab05997f6f8/html5/thumbnails/17.jpg)
Amplitude vs FrequencyEdge Channels
Data that should be flagged
![Page 18: Flagging: When Good Data Go Bad Scott Schnee & Amy Kimball Nov 9, 2011 Interferometry Discussion Group.](https://reader034.fdocuments.us/reader034/viewer/2022051710/5a4d1af37f8b9ab05997f6f8/html5/thumbnails/18.jpg)
Amplitude vs FrequencySpectral Lines in Bandpass Calibrator
From TW Hydra Band 7 GuideSpectral line in Titan
![Page 19: Flagging: When Good Data Go Bad Scott Schnee & Amy Kimball Nov 9, 2011 Interferometry Discussion Group.](https://reader034.fdocuments.us/reader034/viewer/2022051710/5a4d1af37f8b9ab05997f6f8/html5/thumbnails/19.jpg)
Phase vs TimePhase Jumps
First batch of data
Second batch of data
From Antennae ALMA CASA Guide
![Page 20: Flagging: When Good Data Go Bad Scott Schnee & Amy Kimball Nov 9, 2011 Interferometry Discussion Group.](https://reader034.fdocuments.us/reader034/viewer/2022051710/5a4d1af37f8b9ab05997f6f8/html5/thumbnails/20.jpg)
Possible Flagging Technique
1. Flag obviously bad data2. Calibrate the data3. Flag newly found bad data4. Re-calibrate5. Iterate (3, 4) or declare victory
![Page 21: Flagging: When Good Data Go Bad Scott Schnee & Amy Kimball Nov 9, 2011 Interferometry Discussion Group.](https://reader034.fdocuments.us/reader034/viewer/2022051710/5a4d1af37f8b9ab05997f6f8/html5/thumbnails/21.jpg)
After Calibration, Look Again
From NGC 3256 ALMA CASA guideAmplitude vs Time, after calibration
![Page 22: Flagging: When Good Data Go Bad Scott Schnee & Amy Kimball Nov 9, 2011 Interferometry Discussion Group.](https://reader034.fdocuments.us/reader034/viewer/2022051710/5a4d1af37f8b9ab05997f6f8/html5/thumbnails/22.jpg)
Sage Advice
From Rick Perley to a much younger Scott Schnee:“When in doubt, throw it out.”
![Page 23: Flagging: When Good Data Go Bad Scott Schnee & Amy Kimball Nov 9, 2011 Interferometry Discussion Group.](https://reader034.fdocuments.us/reader034/viewer/2022051710/5a4d1af37f8b9ab05997f6f8/html5/thumbnails/23.jpg)
Online flags• (from Steve Myers and Josh Marvil)
• Tbuff (before August 2011)
• Quacking sometimes necessary
![Page 24: Flagging: When Good Data Go Bad Scott Schnee & Amy Kimball Nov 9, 2011 Interferometry Discussion Group.](https://reader034.fdocuments.us/reader034/viewer/2022051710/5a4d1af37f8b9ab05997f6f8/html5/thumbnails/24.jpg)
Known RFI at science.nrao.edu• http://www.gb.nrao.edu/IPG/rfiarchivepage.html• https://science.nrao.edu/facilities/evla/observing/RFI/index
![Page 25: Flagging: When Good Data Go Bad Scott Schnee & Amy Kimball Nov 9, 2011 Interferometry Discussion Group.](https://reader034.fdocuments.us/reader034/viewer/2022051710/5a4d1af37f8b9ab05997f6f8/html5/thumbnails/25.jpg)
A flagging/calibration recipe
• EXAMINE bandpass/flux calibrator(s)• FLAG bandpass/flux calibrators• APPLY bandpass/flux calibration to itself
• APPLY bandpass/flux cal to phase cal sources
• EXAMINE phase cal sources• FLAG phase cal sources• APPLY phase calibration to itself
• APPLY bandpass/flux/phase cal to targets
• EXAMINE targets• FLAG targets
Iterate
Iterate
Repeat as necessary
![Page 26: Flagging: When Good Data Go Bad Scott Schnee & Amy Kimball Nov 9, 2011 Interferometry Discussion Group.](https://reader034.fdocuments.us/reader034/viewer/2022051710/5a4d1af37f8b9ab05997f6f8/html5/thumbnails/26.jpg)
Flagversion control in CASA
• Beware of applycal!
![Page 27: Flagging: When Good Data Go Bad Scott Schnee & Amy Kimball Nov 9, 2011 Interferometry Discussion Group.](https://reader034.fdocuments.us/reader034/viewer/2022051710/5a4d1af37f8b9ab05997f6f8/html5/thumbnails/27.jpg)
Recognizing low-frequency RFI
• Average all times: RFI visible on the shortest baselines
One EVLA user’s recommendation: flag these frequencies on all baselines at all times
![Page 28: Flagging: When Good Data Go Bad Scott Schnee & Amy Kimball Nov 9, 2011 Interferometry Discussion Group.](https://reader034.fdocuments.us/reader034/viewer/2022051710/5a4d1af37f8b9ab05997f6f8/html5/thumbnails/28.jpg)
Individual timeranges can be bad