MNRAS 000, 1–11 (2019) Preprint 16 January 2019 Compiled using MNRAS LATEX style file v3.0

The impact of interference excision on 21-cm Epoch ofReionization power spectrum analyses

A. R. Offringa,1,2? F. Mertens2 and L. V. E. Koopmans21Netherlands Institute for Radio Astronomy (ASTRON), 7991 PD Dwingeloo, The Netherlands2Kapteyn Astronomical Institute, University of Groningen, PO Box 800, NL-9700 AV Groningen, the Netherlands

Accepted 2019 January 11. Received 2018 November 9; in original form 2018 July 23

ABSTRACTWe investigate the implications of interference detection for experiments that are pur-suing a detection of the redshifted 21-cm signals from the Epoch of Reionization.Interference detection causes samples to be sporadically flagged and rejected. As anecessity to reduce the data volume, flagged samples are typically (implicitly) inter-polated during time or frequency averaging or uv-gridding. This so-far unexploredsystematic biases the 21-cm power spectrum, and it is important to understand thisbias for current 21-cm experiments as well as the upcoming SKA Epoch of Reionizationexperiment. We analyse simulated data using power spectrum analysis and Gaussianprocess regression. We find that the combination of flagging and averaging causes tinyspectral fluctuations, resulting in “flagging excess power”. This excess power does notsubstantially average down over time and, without extra mitigation techniques, canexceed the power of realistic models of the 21-cm reionization signals in LOFAR obser-vations. We mitigate the bias by i) implementing a novel way to average data using aGaussian-weighted interpolation scheme; ii) using unitary instead of inverse-varianceweighting of visibilities; and iii) using low-resolution forward modelling of the data.After these modifications, which have been integrated in the LOFAR EoR processingpipeline, the excess power reduces by approximately three orders of magnitude, andis no longer preventing a detection of the 21-cm signals.

Key words: dark ages, reionization, first stars – methods: observational – techniques:interferometric

1 INTRODUCTION

The Epoch of Reionization (EoR) is a phase in the evolu-tion of our Universe of which, at present, relatively littleis known. A promising way to study the EoR is by usinga low-frequency interferometric array to statistically detectthe redshifted 21-cm signals of neutral hydrogen from theEpoch of Reionization using 21-cm power spectrum analy-ses (Iliev et al. 2002; Morales 2005; Furlanetto et al. 2006;McQuinn et al. 2006). Several telescopes have been designedto study the 21-cm EoR power spectrum, such as the Low-Frequency Array (LOFAR; Van Haarlem et al. 2013); theDonald C. Backer Precision Array for Probing the Epoch ofReionization (PAPER; Parsons et al. 2012); and the Murchi-son Widefield Array (MWA; Beardsley et al. 2016), and itis one of the planned key science drivers of the Square Kilo-metre Array (SKA).

The faint signals from the EoR are hidden behind stronggalactic and extragalactic foregrounds, which are orders of

? E-mail: offringa@astron.nl

magnitude brighter (Jelic et al. 2008; Bernardi et al. 2010).There are several methods that are pursued to achieve adetection. First of all, a large part of the foregrounds canbe subtracted from the data by creating accurate sky mod-els (Yatawatta et al. 2013; Carroll et al. 2016; Procopioet al. 2017). Furthermore, the foregrounds and 21-cm signalsare expected to have different spectral behaviour, and aretherefore distinguishable in different parts of a cylindrically-averaged power spectrum (Liu et al. 2009; Datta et al.2010; Vedantham et al. 2012; Morales et al. 2012; Offringaet al. 2016). Finally, techniques have been designed thatcan statistically separate the (residual) spectrally-smoothforegrounds from the spectrally-fluctuating 21-cm signals(Harker et al. 2009; Chapman et al. 2013; Mertens et al.2018). To some level, all these techniques assume that theforegrounds are measured extremely accurately: if rapidly-fluctuating features are introduced by either the instru-ment or the processing that are not modelled, it may nolonger be possible to separate foregrounds from 21-cm sig-nals. This sets strong requirements on the accuracy to whichthe data are calibrated (Barry et al. 2016; Patil et al. 2017;

© 2019 The Authors

arX

iv:1

901.

0475

2v1

[as

tro-

ph.I

M]

15

Jan

2019

2 A. R. Offringa et al.

Ewall-Wice et al. 2017; Trott & Wayth 2017). In this paper,we analyse the contaminating effect that interference detec-tion can introduce, assert whether these effects are strongenough to cause problems for a 21-cm power spectrum de-tection and introduce techniques to mitigate the issue.

All major interferometric EoR experiments use someform of radio-frequency interference (RFI) rejection, for ex-ample by detecting outlier samples and “flagging” these asbeing contaminated (Winkel et al. 2006; Middelberg 2006;Prasad & Chengalur 2012; Peck & Fenech 2013; Offringaet al. 2015). Further processing (calibration, imaging, powerspectrum generation) will subsequently ignore those sam-ples. RFI detection is most effective at high resolution (Of-fringa et al. 2010), while many of the scientific goals, in-cluding EoR power spectrum studies, do not require high-resolution products. Therefore, the recorded data are typi-cally averaged in time and frequency to a resolution of sev-eral seconds and tens of kilohertz to reduce its volume afterflagging. Furthermore, pipelines that work from a griddeduv-plane or from images employ binning of visibilities basedon their uv-values. During the gridding process, visibilitiesare also averaged together. If the RFI-flagged samples areremoved during these data averaging steps, the irregular dis-tribution of missing samples will result in fluctuations in thevisibilities. In this paper, we will analyse the magnitude ofthese effects on the 21-cm power spectrum within the con-text of the LOFAR EoR experiment.

In Section 2 we introduce the standard LOFAR EoRprocessing methodology. Section 3 describes the methodsused in this work to analyse and mitigate the flagging excesspower. In Section 4 we show the impact of flagging on 21-cm power spectrum analyses, and present the results of themitigation methods. In Section 5, we draw our conclusions.

2 THE LOFAR EOR PROCESSING PIPELINE

This paper analyses results within the context of the LO-FAR EoR data processing methodology (Patil et al. 2016).Therefore, we briefly summarize the default LOFAR EoRprocessing steps, which are: i) RFI rejection using aoflag-ger (Offringa et al. 2012); ii) data averaging using dppp;iii) (optionally) data compression using dysco (Offringa2016); iv) direction-independent calibration using dppp orsagecal (Kazemi et al. 2011); v) direction-dependent com-pact source removal using sagecal-co (Yatawatta 2016);vi) imaging using wsclean (Offringa et al. 2014) or ex-con (Yatawatta 2014); vii) residual foreground removal us-ing GPR (Mertens et al. 2018); and viii) power spectrumcalculation using both a Python and a C++ power spec-trum pipeline (§3.1).

Typical LOFAR EoR observations are performed at atime and frequency resolution of 2 s and 3 kHz, respectively,which will be referred to as “high resolution data” in thepaper. Because RFI detection is most effective at high timeand frequency resolution (Offringa et al. 2010), it is the firstprocessing step that is performed after recording the data.Afterwards, the data is averaged down by about a factorof 100 to typical resolutions of 10 s and 40 to 60 kHz. Thisdata product will be referred to as the “low resolution data”.Averaging decreases the data volume considerably, while itis still of high enough resolution to not cause any significant

Table 1. Details of simulated dataset & observed RFI flags.

Observing start 4 Feb 2018, 16:49 (UTC)Observing end 5 Feb 2018, 6:43 (UTC)

LOFAR ID L628584

Duration 50001 s (∼ 14 h)

RA 0h 00

Dec 90 00

Band 113.8–128.1 MHz

21-cm redshift 11.5–10.1

RFI percentage 1.12 %

Polarization XX, XY, YX, YY

0

1

2

3

4

5

6

7

8

9

10

RFI

(%

)

114

116

118

120

122

124

126

128

Frequency

(M

Hz)

18:00 20:00 22:00 0:00 2:00 4:00 6:00Time (UTC, hh:mm:ss)

Figure 1. Detected RFI occupancy over time and frequency. TheRFI detection is performed at a resolution of 3 kHz / 2 s. The

occupancy shown here is averaged over all baselines in bins of 200kHz / 12 s.

time or frequency smearing within the 5 LOFAR primarybeam.

3 SIMULATED DATA & METHODS

To accurately simulate the effect of flagging, we use the RFIdetection flags from an actual LOFAR observation of theNorth Celestial Pole (NCP) — one of the target fields of theLOFAR EoR project. During typical processing, the flaggedhigh-resolution data with a resolution of 2 s and 3 kHz arenot written to disk, but are immediately averaged down,and the high-resolution input data are removed afterwardsto free up space for further observations. To gain access tothe high resolution flags for this study, the full raw data areintercepted before averaging down and are written to disk.Subsequently, aoflagger is run on the data without av-eraging afterwards. Thereafter, the observed visibilities arereplaced by simulated visibilities for a realistic model. Theresult is a simulated high-resolution visibility set with real-istic RFI detection flags.

Table 1 summarizes the observation that was used forRFI detection. Offringa et al. (2013) have previously studiedthe RFI environment of LOFAR using two 24 hour observa-tions targetting the NCP. They show that the 114-128 MHzfrequency range is one of the least RFI affected frequencyranges in the HBA, which is also reflected in the detectedRFI occupancies (1.1% found in this study vs. an average of3.2% in the HBA as found by Offringa et al. 2013). Fig. 1

MNRAS 000, 1–11 (2019)

RFI excision effects in 21-cm power spectra 3

shows an overview over time and frequency of the detectedRFI occupancies of the data used in this work.

Although an extensive model of the NCP field is avail-able, we chose to use a simulated point-source model basedon a population study, which prevents selection effects andartefacts in the model. We use a simple randomly generatedpopulation distribution that follows the empirically deter-mined distribution by Franzen et al. (2016):

dNdS= 6998 S−1.54Jy−1Sr−1. (1)

We analytically predict (using the direct Fourier transform)the contribution for sources with a flux density between10 mJy and 10 Jy, and assign a random spectral index (SI)to each source with an average of α = −0.8 and a standarddeviation of 0.2 (with S(ν) = S0(ν/ν0)α). This is a reasonabledistribution at the corresponding frequency (e.g., Hurley-Walker et al. 2017), albeit that we ignore flattening of fainter(starburst) galaxies and ignore special classes of sources suchas USS, CSS or GPS sources that can have steep or curvedspectra at the frequencies of interest (see Callingham et al.2017 for an overview). We also do not simulate any diffuseemission. Using the dppp software, the flux density contribu-tion of each source in our final source model is predicted atthe resolution of our data and multiplied by the correspond-ing gain of the LOFAR HBA beam model that combines thestation array factor and the tile beam. Because of the highresolution of our data, this step is the most expensive stepin the processing, and takes about a week of computing on16 high-performance nodes, each with 40 CPU-cores.

After having predicted the simulated foregrounds intothe observation, we create two averaged sets from these: onein which the flagged samples are excluded in the averaging,which is how RFI flagging would affect a regular observation,and one in which the flags are ignored and all data is used,which simulates an observation that is completely free ofRFI.

3.1 Image-based calculation of power spectra

The LOFAR EoR project makes use of two independentpower spectrum pipelines: a Python pipeline and a C++pipeline. The Python pipeline is written to integrate Gaus-sian process regression, while the C++ pipeline is used forquick analysis. Having multiple pipelines has been very use-ful for the verification of results, as also noted by other teams(Jacobs et al. 2016). When using the same settings and data,our two pipelines produce similar results.

For calculating the power spectra, we follow the defi-nitions from Parsons et al. (2012) and Trott et al. (2016),where the power spectrum is calculated as1

P(k) ≡ V |T(k)|2. (2)

Here, V is the comoving volume of the data cube, T is definedas the (normalized) discrete Fourier transform of T :

T(k) ≡ 1N

∑x

T(x)e−ik·x, (3)

1 Our definition of P differs from Eq. (2) in Parsons et al. (2012),

where a factor of V is mistakenly left out. From their definitions,it can be shown that their Eq. (3) is correct, and that equation

implies the factor of V in the definition of P.

x is a physical coordinate, k is inverse scale and N is thenumber of voxels in the data cube.

For this work, both pipelines use wsclean (Offringaet al. 2014) as a first step to grid the data and decreasethe data volume. Nevertheless, the Python pipeline can alsowork directly from ungridded visibilities (Ghosh et al. 2018).We perform the imaging with increased-accuracy settings forwsclean, which includes a larger gridding kernel, higheroversampling rate and increased number of w-layers, com-pared to default settings of wsclean. The output of theimager is a naturally-weighted primary-beam-corrected im-age with units of Jy/beam. Commonly, conversion to Kelvinis performed by fitting the synthesized beam to a Gaussian,followed by evaluation of

T(x) ≡ SJy/B(x)10−26c2

2kBν2Ωpsf, (4)

with SJy/B(x) the data cube in units of Jy/beam. However,we found that the synthesized beam of LOFAR deviatesfrom a Gaussian function, causing the power to be under-estimated by a factor of 1.5 when using this approach. Toovercome this, wsclean stores2 the factor which it has di-vided the data by, and from which the “per beam” term canbe calculated. This allows us to accurately convert the imageto units of Jy/pixel3. While naturally weighted images leadto the best power spectrum sensitivity (Morales & Matejek2009), it requires normalization of the uv-cells in order notto bias the power spectrum. We do this by dividing the uv-plane of the data by the uv-plane of the point-spread func-tion (PSF). In principal, this can lead to undefined valueswhen the uv-plane is not fully covered. The uv-plane cor-responding to the PSF can in this case contain very smallvalues due to rounding errors and small gridding kernel val-ues. Propagating the PSF uv values and using these as dataweights mostly solves this issue and improves the sensitivityof the final power spectrum. Both our pipelines do this. Inthe case of LOFAR, the perpendicular scales that we targetcorrespond with a baseline range of 50 − 250λ, and all uv-cells are sufficiently covered within this range by the LOFARarray configuration.

Combining the above, the Fourier transform T is calcu-lated as

T(k) = 10−26c2

2kBΩANν

∑ν

©­­«1ν2 e−iν

∑l

S(x)e−iu·l∑l

P(x)e−iu·lª®®¬ , (5)

with P the value of the PSF. Because of lost edge channelsdue to the poly-phase filter of LOFAR, the bandwidth is non-uniform. Moreover, as mentioned before, each data samplehas an associated weight with it, which is to be used in theline-of-sight Fourier transform. Both our pipelines solve thisby performing an inverse covariance weighted, least-squaresspectral analysis (LSSA) of the line-of-sight Fourier trans-form operation (Trott et al. 2016).

2 Stored as the WSCNORM keyword inside the FITS file.3 It is not strictly necessary to convert from Jy/beam to Jy/pixel,

because in Eq. 5 the scaling factor appears in both the numeratorand denominator of the rightmost division, and therefore cancels

out. However, the factor is required when propagating the errors.

MNRAS 000, 1–11 (2019)

4 A. R. Offringa et al.

In simulations where the foreground has not been sub-tracted from the data, we use a window function to preventleakage from the wedge to higher k ‖ values and adapt thebox volume V to accommodate for the decreased line-of-sightdimension, for example a Blackman-Nuttall window (Nuttall1981) as described by Vedantham et al. (2012). This step isnormally not taken when processing real data within theLOFAR EoR project: in that case, power spectra are madefrom direction-dependently calibrated data, and a windowfunction is not necessary because the foreground wedge isnot present.

3.2 Gaussian Process Regression

After calibration and direction-dependent compact sourceremoval of data in the LOFAR EoR project, the remainingforegrounds, composed of extragalactic emission below theconfusion noise level and diffuse galactic emission, are stillapproximately 3 to 4 orders of magnitude brighter than the21-cm signal. The LOFAR EoR project uses the technique ofGaussian Process Regression (GPR; Mertens et al. 2018) tomodel and remove these residual foregrounds. In this frame-work, the different components of the observations, includingthe astrophysical foregrounds, mode-mixing contaminants,and the 21-cm signal, are modelled as a Gaussian process(GP). A GP is the joint distribution of a collection of nor-mally distributed random variables (Rasmussen & Williams2005). The covariance matrix of this distribution is speci-fied by a covariance function, which defines the covariancebetween pairs of observations (e.g. at different frequencies).The covariance function determines the structure that theGP will be able to model, for example its smoothness. InGPR, we use GP as parametrized priors, and the Bayesianlikelihood of the model is estimated by conditioning thisprior to the observations. Standard optimization or MCMCmethods can be used to determine the parameters of thecovariance functions.

Formally, we model our data d observed at frequenciesν by a foreground, a 21-cm and a noise signal n (Mertenset al. 2018):

d = ffg(ν) + f21(ν) + n. (6)

The foreground signal can be separated from the 21-cm sig-nal by exploiting their different frequency behaviour: the 21-cm signal is expected to be largely uncorrelated on scales ofa few MHz, while the foregrounds are expected to be smoothon that scale. The covariance function of our GP model canthen be composed of a foreground covariance function Kfgand a 21-cm signal covariance function K21,

K = Kfg + K21. (7)

The foregrounds covariance kernel itself is decomposedin two, accounting for the large frequency coherence-scale of the intrinsic foreground and the smaller frequencycoherence-scale (about 1 to 5 MHz) of the mode-mixing com-ponent. We use an exponential covariance function for the21-cm signal, as we found that it was able to match well thefrequency covariance from simulated 21-cm signal (Mertenset al. 2018).

The joint probability density distribution of the obser-vations d and the function values ffg of the foreground model

ffg at the same frequencies ν is then given by,[dffg

]∼ N

( [00

],

[(Kfg + K21) + σ2

n I KfgKfg Kfg

] ). (8)

using the shorthand K ≡ K(ν, ν). After GPR, the foregroundspart of the model is retrieved:

E(ffg) = Kfg[K + σ2

n I]−1

d (9)

cov(ffg) = Kfg − Kfg[K + σ2

n I]−1

Kfg. (10)

and is subtracted from the original data.

3.3 Spectral fluctuations caused by data flagging

To introduce the symptoms of flagged data and demonstratewhy a study is necessary, we compare two simulated imagecubes: a flagged cube and a non-flagged“ground-truth”cube.For the flagged cube, the high-resolution RFI detection flagsfrom a real LOFAR observation are transferred before av-eraging the observation down. Fig. 2 shows slices throughthe spectral and spatial direction of several image cubes.The left plot is from data that includes flags, and showssmooth structure from sources and their sidelobes. After re-moving a 5th-order polynomial fit from each line-of-sight (inuv space), about an order of magnitude of flux from the datais removed, revealing residual higher order, smooth structurefrom the sources, but also rapid spectral fluctuations on theorder of tens of mJy (Fig. 2, centre). These fluctuations areonly present when the flags from RFI detection are added tothe high-resolution data, before averaging down. The rightplot of Fig. 2 shows the simulated data without flags, aftera polynomial fit.

The spectral fluctuations caused by data flagging causessome part of the power of the foregrounds to have spectralfluctuations that correspond with the redshifted signals fromthe Epoch of Reionization. For example, the cylindrically-averaged power spectra corresponding to the simulated cleanand flagged data in Fig. 3 show that flagging causes a sig-nificant increase of power at high k ‖ , above the foregroundwedge. In the next sections, we explain the source of thefluctuations and describe methods to mitigate them.

3.4 Temporal and spectral averaging

Data averaging is a common step in radio astronomy: it isimplemented in several pipelines, including the LOFAR EoRpipeline, and can be performed with several tools (e.g. casa,dppp and cotter). In this study we find that the standardmethod of averaging data increase the spectral fluctuationscaused by flagging.

The way averaging in time and frequency are imple-mented in the aforementioned tools is by binning samplesthat are close together in time and frequency. When sam-ples are flagged, they are excluded from the bin, causing thetime-frequency bin to not exactly be equal to the mean of thevisibility function of the sky. It is effectively equal to interpo-lating the flagged sample and replacing it by the average ofthe mean of the other samples in that bin. While the errormade by this method (the difference between the interpo-lated value and the true value of the interpolated visibility)is small and mostly negligible in regular radio-astronomical

MNRAS 000, 1–11 (2019)

RFI excision effects in 21-cm power spectra 5

-0.5

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5

Flux d

ensi

ty (

Jy/b

eam

)

119

120

121

122

123

124

125

126

127

128

Freq

uency

(M

hz)

-100 -50 0 50 100l (pixels)

Flagged data

-0.04

-0.03

-0.02

-0.01

0

0.01

0.02

0.03

0.04

Flux d

ensi

ty (

Jy/b

eam

)

119

120

121

122

123

124

125

126

127

128

Freq

uency

(M

hz)

-100 -50 0 50 100l (pixels)

Flagged, after polynomial fit

-0.04

-0.03

-0.02

-0.01

0

0.01

0.02

0.03

0.04

Flux d

ensi

ty (

Jy/b

eam

)

119

120

121

122

123

124

125

126

127

128

Freq

uency

(M

hz)

-100 -50 0 50 100l (pixels)

No flags, after polynomial fit

Figure 2. Slices through a simulated image cube, showing the flux density as a function of frequency (vertical) and spatial scale

(horizontal). Data that has been flagged (left image) appears to behave smoothly in frequency. However, after subtracting a 5th-order

polynomial fit, some fine-scale fluctuations are visible. These are not present when no data has been flagged (right image).

100000

1e+06

1e+07

1e+08

1e+09

1e+10

Pow

er

(mK

²[M

pc/

h]³

)

0.2

0.5

1

2

k∥ (

h/M

pc)

0.2

0.5

1

2

5

η (

µs)

0.06 0.08 0.1 0.15 0.2k⊥ (h/Mpc)

60 80 100 150 200U (wavelengths)

No flagging, inverse variance weighting

100000

1e+06

1e+07

1e+08

1e+09

1e+10

Pow

er

(mK

²[M

pc/

h]³

)

0.2

0.5

1

2

k∥ (

h/M

pc)

0.2

0.5

1

2

5

η (

µs)

0.06 0.08 0.1 0.15 0.2k⊥ (h/Mpc)

60 80 100 150 200U (wavelengths)

Flagged, inverse variance weighting

100000

200000

500000

1e+06

2e+06

5e+06

1e+07

2e+07

5e+07

1e+08

Pow

er

(mK

²[M

pc/

h]³

)

0.2

0.5

1

2

k∥ (

h/M

pc)

0.2

0.5

1

2

5

η (

µs)

0.06 0.08 0.1 0.15 0.2k⊥ (h/Mpc)

60 80 100 150 200U (wavelengths)

Difference between flagged and unflagged (inv var weighted)

100000

1e+06

1e+07

1e+08

1e+09

1e+10

Pow

er

(mK

²[M

pc/

h]³

)

0.2

0.5

1

2

k∥ (

h/M

pc)

0.2

0.5

1

2

5

η (

µs)

0.06 0.08 0.1 0.15 0.2k⊥ (h/Mpc)

60 80 100 150 200U (wavelengths)

Flagged, unit weighting

Figure 3. Cylindrically-averaged power spectra from simulated noise-less data. Top-left: from simulated data without any flags; Top-right: from simulated data after applying the RFI flags from a real observation and using inverse-variance weighting; Bottom-left:difference between non-flagged (top-left) and flagged data (top-right); bottom-right: same as top-right, but giving visibilities equal

weight, independent of how many visibilities were flagged before averaging. The red dashed line indicates the k-modes that correspondto the horizon; the blue dashed line corresponds to a 5 field of view.

MNRAS 000, 1–11 (2019)

6 A. R. Offringa et al.

data analysis, cosmological 21-cm power spectrum analysesare extremely sensitive to spectral fluctuations, and can benegatively affected by this effect.

To form images from visibilities, samples are gridded ona regular uv plane. Similar to averaging, data is also binnedduring data gridding, and samples that are missing due toRFI will effect the imaging in a similar way. Flagged samplescan therefore cause spectral fluctuations both during dataaveraging as well as during gridding.

We also find that another technical detail is relevant tothe excess flagging power. In standard radio interferometricdata, each visibility has an associated weight stored withit. This weight is taken into account during calibration andimaging, which results in inverse-variance weighted, least-squares calibration solutions and inverse-variance weightedimages, and therefore in an inverse-variance weighted powerspectrum. When averaging data, the visibility weights arenormally updated: when half the input samples in an aver-aging bin are flagged, the averaged visibility will get half theweight of a fully averaged sample. Unfortunately, weight-ing samples based on the RFI flags during imaging willcause differences between the uv-coverage at different fre-quencies. Similar to flagging and averaging, having differ-ent uv-coverages at different frequencies can lead to spec-tral fluctuations. The effect of non-smooth weights over fre-quency will, when visibilities are averaged or binned ontoa uv-grid, lead to effects that are similar to the effect ofmissing samples. On first order, it changes the centroid ofa uv-cell with unevenly distributed weights, causing fluctu-ations over frequency. We will compare the inverse-varianceweighting method to unitary weighting of the visibilities, inwhich all visibilities are given equal weight independently ofhow many samples were flagged before averaging.

3.5 Improved interpolation during averaging andgridding

As discussed in a previous section, during standard radio-interferometric data averaging, flagged samples are interpo-lated and replaced by the mean of the time-frequency binthey are in. This binning interpolation method contributesto the flagging excess power, because a bin with flagged sam-ples can have a biased average that results in spectral fluc-tuations. In this paper we test an improved interpolationscheme.

For our improved interpolation scheme, before averag-ing we replace flagged samples by the windowed, Gaussian-weighted average of unflagged samples:

V ′(i, j) =∑

k,l W(k, l)F(i + k, j + l)V(i + k, j + l)∑k,l W(k, l)F(i + k, j + l) , (11)

with W(k, l) = exp(−0.5(k2 + l2)/σ2), the two-dimensionalGaussian function with width parameter σ; V(i, j) the com-plex visibility for timestep i and channel j; and F(i, j) theflag status for sample (i, j): 0 if it is flagged and 1 otherwise.The sums in Eq. (11) are over l, k ∈ [− 1

2 (N − 1) : 12 (N − 1)],

with N the (odd) size of the window. We have chosen σ to bethe width of one timestep/channel (corresponding to a tem-poral σ of 2s and a spectral σ of 3 kHz), which we find islarge enough to calculate a representable visibility for miss-ing samples, and at the same time small enough to avoid theneed of a computationally-expensive large window.

100

1000

10000

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2

Δ(k)

(mK)

k∥ (h/Mpc)

Inv variance weightingUnit Weighting (UW)UW (even timesteps)

UW + interpolationUW + low res fwdm

Without flags

Figure 4. Power at k⊥ = 0.1 h/Mpc without applying GPR,

showing power as function of k‖ with and without flagging. Solidlines are from simulated data that includes flagged RFI, the

dashed line is without any flagged samples.

3.6 Forward modelling

As long as the high-resolution RFI detection flags are storedalong with the averaged data, it is possible to forward predictthe sky model, including the effect of data flagging and av-eraging. This requires storing the high-resolution flags, pre-dicting the model at high resolution and propagating thehigh resolution model data through the same flagging andaveraging steps. This would cancel the excess flagging powerassociated with the modelled sky sources. However, doingso requires prediction of the sky model at the time and fre-quency resolution at which flagging is performed, which iscomputationally expensive. For LOFAR, predicting the fore-ground model at low resolution costs already approximatelyhalf of our total computational budget. Predicting at highresolution (> 100 times more visibilities) is therefore expen-sive with current techniques, although optimizations mightbe possible. Prediction at low resolution is however alreadyfeasible. A low-resolution prediction will not fully take intoaccount the loss of flagged high-resolution samples, but willremove most of the foreground power before gridding, andthis will therefore reduce the excess power associated withthe gridding step. We will assess to what level low-resolutionmodelling mitigates the flagging excess power.

4 RESULTS

Fig. 3 shows cylindrically-averaged power spectra from thesimulated data. Because no foreground removal strategy hasbeen applied yet, the power spectra are dominated by powerfrom the spectrally-smooth foregrounds. The power in theplot decreases rapidly with increasing k ‖ . There is a hori-zontal line of high power visible at k ‖ ≈ 2.5 h/Mpc, whichis caused by missing channels at the 200 kHz LOFAR sub-band edges. The poly-phase filter that forms the subbandsaliases the signals from other channels into these channels,and these channels have to be removed in observations. We

MNRAS 000, 1–11 (2019)

RFI excision effects in 21-cm power spectra 7

1

10

100

1000

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2

Δ(k)

(mK)

k∥ (h/Mpc)

Inv variance weightingUnit weighting (UW)

UW (even timesteps)

UW + interpolationUW + low res fwdm

UW + interp + low res fwdm

Figure 5. Difference power at k⊥ = 0.1 h/Mpc showing the excess

power introduced by data flagging for the same cases as in Fig. 4.

have therefore also removed these edge channels in the sim-ulations.

The top-right panel in Fig. 3 shows the cylindrically-averaged power spectra from data that includes RFI flags.It can be seen that the spectral fluctuations caused by flag-ging result in excess power at high k ‖-values. As can be seenin the bottom-left image of Fig. 3, which shows the powerspectrum of the data cube difference with and without flag-ging, the flagging excess power affects all k ‖-values with ap-proximately equal power. It therefore also contaminates thepower spectrum window in which the epoch of reionizationsignals are most easily detectable.

Certain effects can be absorbed in calibration solutions.To validate whether calibration affects the excess power, wehave also performed calibration of the simulated data. Thisresults in fact in a slightly increased excess power: the excesspower is not suppressed by calibration.

Visibility weighting results: So far, the visibilities wereinverse-variance weighted, i.e., the number of unflagged vis-ibilities before averaging is propagated into the weight ofan individual averaged visibility. The bottom-right panel ofFig. 3 shows the power spectrum from the same data af-ter giving all visibilities the same weight. The excess flag-ging power has decreased quite significantly compared to theinverse-variance visibility weighted plot, but excess power isstill visible. Similar to the inverse-variance weighting case,the power spectrum from unitary-weighted visibilities is af-fected with approximately similar power at all modes. Thereason that inverse variance increases flagging excess power,is that the inverse-variance weights are dependent on thenumber of flagged samples, and therefore cause the uv-coverage to be different for different frequencies. On firstorder, having spectrally fluctuating weights causes the cen-troid of uv-cells to be different for different frequencies andtherefore results in small fluctuations in the gridded visibil-ities.

Fig. 4 shows k⊥ = 0.1 h/Mpc slices through thecylindrically-averaged power spectra of Fig. 3, converted to

0.0 0.5 1.0 1.5 2.0 2.5

k [h cMpc−1]

100

101

P(k

)[K

2h−3cMpc3]

excess_odd

excess_even

cross spectra

Figure 6. 2D power-spectra averaged over all kx and ky , compar-ing the odd and even timestep data, as well as its cross spectrum.

At small k‖-values, the odd and even sets show correlated excesspower. At k‖ & 0.3 h/Mpc, the cross spectra is about a factor of 2

below the individual spectra, implying that the flagging artefacts

correlate partially between the odd and even sets.

dimensionless power ∆(k) =√

Pk3/(2π2), with k =√

k2‖ + k2

⊥.

Similarly, Fig. 5 shows the difference between the non-flagged “ground truth” data and various cases that includeflagging. The difference plot is constructed by subtractingthe affected visibilities from the ground-truth visibilities ingridded uv-space before calculating the power spectrum.The values in these plots can be compared to the expectedvalue of the power in the 21-cm signal fluctuations, which istypically predicted to be several to tens of mK at these red-shifts and k-values (e.g. Greig & Mesinger 2015). Because ofthe conversion to dimensionless power, the power (and ex-cess flagging power) increases toward higher |k ‖ | values. Thepower levels are still relatively high because no foregroundswere yet subtracted. With inverse variance weighting, theexcess flagging amplitude at k⊥ = 0.1 h/Mpc ranges fromapproximately 100 to 3000 mK at low and high k ‖ , respec-tively. This decreases to 10 to 400 mK when using unitaryvisibility weighting.

Another correction we try is to grid the visibilities ontheir true centroid during the gridding process. The LO-FAR DPPP software stores metadata in the measurementset from which the true visibility centroid position can beinferred. However, we find that such a correction results inno improvement.

Gaussian interpolation results: In §3.5 we have de-scribed an interpolation method to replace flagged samplesby a Gaussian-weighted average of unflagged samples. Thistype of interpolation was implemented in the dppp soft-ware. To test this method, we construct a simulated high-resolution LOFAR data cube including flags as before. Sub-sequently, we interpolate flagged values with a window sizeN = 15 and a Gaussian kernel size of σ = 1. If for a particu-lar sample all samples within the corresponding window areflagged, the sample is not interpolated and the correspond-ing output sample is flagged.

Even though the interpolation step is performed at highresolution, it is computationally insignificant compared tothe RFI detection step and reading of the data.

MNRAS 000, 1–11 (2019)

8 A. R. Offringa et al.

114 116 118 120 122 124 126 128Frequency [MHZ]

0.2

0.0

0.2

0.4

0.6

0.8

1.0

Even

/Odd

Cor

rela

tion

coef

f. (m

agen

ta)

10-8

10-7

10-6

10-5

10-4

10-3

10-2

Varia

nce

(arb

itrar

y un

it, o

rang

e)Figure 7. Correlation between the odd and even cases (magentaline) drawn together with the variance (orange line). Bands with

higher RFI excess power, and therefore higher RFI occupancy,show a stronger correlation between odd and even. This implies

that stronger sources are more consistently present.

The power spectrum results for this interpolationscheme are visualized in Fig. 4 and Fig. 5. Compared tonormal averaging with unit weighted visibilities, Gaussianinterpolation reduces the excess flagging power by over anorder of magnitude. At small k ‖ values, the excess noisedecreases to a level of approximately one mK. With this de-crease in excess power, together with further decompositionof the data by GPR and the decrease caused by averagingmultiple nights together, the flagging excess power will nolonger prevent detection of 21-cm signals from the EoR.

Forward modelling results: Fig. 5 includes the result ofsubtracting the predicted sky model from the low resolu-tion data. After low-resolution prediction, the residual excesspower varies from 0.3 to 19 mK. The residual excess power iscaused by the fact that a low-resolution prediction does notmatch the high-resolution predicted data precisely, becauseof the combination of flagging and averaging. Compared tohigh-resolution interpolation, the low-resolution sky modelsubtraction reduces the excess power by approximately afactor of 2 to 5 at low and high k ‖ , respectively. Consideringthat a low-resolution predict removes power before the grid-ding operation, and can not model the excess fluctuationsthat are arising from averaging high-resolution data, this re-sult implies that most of the flagging excess power arisesduring gridding of the low-resolution data on the uv-plane,and not during the first stage of high-resolution data averag-ing. Although the low-resolution data has considerably fewerflags because of data averaging, in which flagged values arereplaced by the bin average, a number of samples remainsflagged after averaging. Those flags arise when an averagingbin (or interpolation kernel) was completely flagged in thehigh resolution data. It is those flags that cause most of theexcess power.

This result also implies that flagging at lower resolu-tion to skip the time-frequency averaging, will not avoid theexcess noise: although the averaged visibilities do no longerhave biases due to the high-resolution flagging, the griddingof visibilities in bins would still be affected by missing low-

resolution samples. Moreover, flagging at a lower resolutionis less effective, causing more false negative.

Low-resolution prediction cannot solve inaccuraciesthat were petrified into the data during high-resolution pro-cessing. However, by using Gaussian interpolation of thehigh-resolution data instead of data averaging, it is possi-ble to reduce the excess power further. As shown in Fig. 5,the combination of uniform weighting, high-resolution inter-polation and low-resolution forward modelling results in thelowest excess power, being < 1 mK at k ‖ < 1 h/Mpc.

4.1 Systematic nature of excess flagging power

So far, we have simulated a single night of observation. Thesingle-night results indicate that, if not mitigated, the excessflagging power can be above likely predictions of the 21-cmsignal power. However, if this power would decrease withtime similar to the system noise, it will be an order of magni-tude lower for a 100-night data analysis. When using unitaryvisibility weighting with no further mitigation, the resultsshow that the excess power will in that case (mostly) notprevent detection of the 21-cm signals. We investigate there-fore whether excess flagging power indeed behaves noise-likeor has a systematic nature.

Certain transmitting sources of interference will occupythe same frequencies consistently, while other sources of in-terference might behave more erratic, occupying randomtimeslots or channels. Examples of sources for such inter-ference are lightning, solar flares and sparking devices. Toassess whether the excess flagging power is systematicallypresent or whether it averages down with time, we split ourdata in even and odd timesteps. This approach is taken in-stead of constructing a second night of simulated data, be-cause simulating a second high-resolution observation withreal RFI flags from a second night is practically difficult,mainly because storing (another) large volume of data onour EoR cluster would interfere with the running LOFAREoR observations. Another approach would be to analysethe time-dependence by splitting the observation halfway intime. This would be slightly more representative, becausethe even and odd timesteps are more likely to correlate thanthe first and second half of the observation. However, do-ing so resulted in excess noise that is difficult to interpret,because the uv-coverage of the first and second halves aredifferent.

If the excess power is not systematic, using half the datashould increase the power (in units of mK) by approximately

a factor of√

2, comparable with the system noise. However,as shown in Figs. 4 and 5, the data do not agree with thishypothesis: the excess power in the even-timestep set is atthe same level as in the full data. This implies that thissource of excess power is systematic and could therefore beproblematic for EoR experiments if not mitigated.

To further analyse the correlation of the flagging ex-cess power over time, we construct a cross power spectrumbetween odd and even timestep sets. If the excess power isuncorrelated in time, a cross spectrum between the two setsshould decrease the excess power substantially. In Fig. 6 it isshown that for k & 0.5 h/Mpc, the cross-spectrum is approx-imately a factor of two below the individual spectra. Fig. 7shows the correlation over frequency, which indicates that an

MNRAS 000, 1–11 (2019)

RFI excision effects in 21-cm power spectra 9

10-1 100

k [h cMpc−1]

100

101

102

103

104

105

106

∆2(k

)[m

K2]

0.0 0.5 1.0 1.5 2.0 2.5k [h cMpc−1]

10-3

10-2

10-1

100

101

102

103

P(k

)[K

2h−

3cM

pc3

]

excessexcess after GPRexcess UW

excess UW after GPRexcess UW+inter.excess UW+inter after GPR

Thermal noise (100 nights, SEFD=4000 Jy)Simulated 21-cm signal (21cmFast)

Figure 8. Impact on the excess power of the GPR procedure. Left: Spherically averaged power-spectra. Right: 2D power-spectra averaged

over all kx and ky . The excess power is shown for three different processing setups, before (solid line) and after applying GPR (dashedline). For comparison, we also show the thermal noise power spectra corresponding to about 100 nights of LOFAR-HBA observation

(assuming an SEFD=4000 Jy), and the power spectra of a simulated 21-cm signal.

increase in excess power at a particular frequency also causesan increased correlation. A possible explanation for this isthat stronger sources of interference are consistently presentat the same frequency, while weaker, transient sources of in-terference and false positive flags do not transmit/occur atone specific frequency.

4.2 Impact of GPR on the excess power

By applying GPR on simulated data sets, one can anal-yse what would be the impact of running GPR on RFI af-fected observation, and if part of the flagging excess powercan be modelled by GPR as part of the foregrounds mode-mixing. In Fig. 8, we analyse excess image cubes that areconstructed by subtracting an unflagged from a flagged datacube. As before, noiseless simulations are used. GPR is ableto model part of the excess power at lower k-values. This isparticularity the case when using inverse-variance weightingand without interpolation, for which a reduction in excesspower of more than one order of magnitude at small k is ob-served. When using unit-weighting and with interpolation,the frequency-coherence of the excess power is reduced, andas is the impact of the GPR. Nevertheless at this stage theexcess power is considerably reduced and is at the level of the21-cm signal. Flagging the subbands which are most affectedby RFI (see e.g. Fig. 7) and subtracting the low-resolutionforward model before gridding could reduce it even more.

5 CONCLUSIONS

For the LOFAR EoR case we have shown that RFI flaggingfollowed by data averaging and gridding, with no furthermitigation, causes excess power that is significantly above

the expected 21-cm signals and does not considerably aver-age down over time. In order to achieve a detection of 21-cmsignals, the excess power can be significantly decreased by:i) using the same weight for all visibilities, instead of prop-agating the number of visibilities in the averaging bin (i.e.inverse-variance weighting); ii) forward modelling (“predic-tion”) of the data at low resolution; and iii) Gaussian inter-polation of missing samples prior to any data averaging, in-stead of the commonly-used method of replacing a sample bythe mean of the averaging bin. Furthermore, a part of the ex-cess power behaves statistically as normally-distributed fore-grounds, and Gaussian process regression can remove aboutanother factor of two of the excess power. Subbands that seea larger contamination of RFI show stronger excess flaggingpower, and a final mitigation strategy is to select bands thatare less contaminated by RFI. Together these techniques re-duce the excess flagging power by approximately three or-ders of magnitude. In particular, inside the EoR windowat k = 0.1 h/Mpc the power is reduced from about 200 to0.3 mK, where the 21-cm signals are expected to be on theorder of several mK.

The presented techniques have been implemented inthe pipeline of the LOFAR EoR project. Subtraction ofthe low-resolution forward model is part of the direction-dependent calibration scheme. This scheme uses Sagecal CO(Yatawatta 2016) to subtract the model from the averageddata including direction-dependent effects.

Because the excess power scales linearly with the fore-ground power, mitigation of the flagging excess power is evenmore critical for fields with higher sky temperature. One ofthe fields that is also targetted as part of the LOFAR EoRproject, is a field that is centred on 3C 196, a bright (∼ 80 Jy)quasar at corresponding frequencies (Scaife & Heald 2012),and the temperature of this field of view is therefore aboutan order of magnitude above more quiet fields.

MNRAS 000, 1–11 (2019)

10 A. R. Offringa et al.

Although this issue was analysed in the context of theLOFAR EoR project, it can be assumed that our conclusionsare equally applicable to other EoR experiments. Flaggingexcess power scales linearly with the number of flagged sam-ples, and although telescopes such as the Murchison Wide-field Array are in a more benign RFI environment comparedto LOFAR, the percentage of flagged samples due to RFI arecomparable: in Offringa et al. (2015), the overall percentageof RFI observed with the MWA is 1.1%. For the LOFARbandwidth used in this work this value is 1.12% (see Ta-ble 1). One of the reasons for this relatively low occupationof RFI compared to the MWA is the high time and frequencydata recording resolution of LOFAR (Offringa et al. 2013).Delay delay-rate filtering (Parsons & Backer 2009) to removeforegrounds as performed by the PAPER EoR team is sim-ilar to interpolation and/or forward modelling, and is likelyto decrease the flagging excess power as well, depending onthe resolution of the data at which it is applied.

These conclusions are also important for the SKA tele-scope, for which the data rate is so high that early aver-aging of data is mandatory, and the full resolution data atwhich the RFI is detected will not be kept. For a future SKAEoR experiment, it is probably necessary to implement ascheme similar to the Gaussian interpolation introduced inthis work. It is also important to store the high-resolutionflags when averaging the data, so that any residual flaggingexcess power from modelled sources can be forward modelledas accurately as is computationally allowed.

In some cases calibration can remove unmodelled ef-fects, such as cable reflections or beam changes. In the caseof flagging excess power, we found that the observed ex-cess power is not absorbed in calibration solutions. Thisis expected, because RFI flagging is inherently a baseline-based effect, and will be variable on time and frequencyscales smaller than the solution interval. This is particu-larly the case when the solutions are constrained to be spec-trally smooth, as is desirable to avoid suppression of theunmodelled 21-cm signals (Patil et al. 2016) and to avoidcalibration-induced excess power (Barry et al. 2016).

We have shown how interpolation using a Gaussian ker-nel before time and frequency averaging can help in reducingexcess power. It is worth mentioning that this interpolationtechnique does not only benefit EoR studies, but will in gen-eral result in more accurately averaged visibilities comparedto the de facto method of replacing missing samples by themean of the averaging bin. This is because for a particularsample, the Gaussian interpolated visibility will generallyrepresent the interpolated visibility more closely than themean of a time-frequency averaged bin. With the de factoaveraging method, an averaging bin is generally not centredon the interpolated visibility, and visibilities are weightedequally independent of their Euclidean time-frequency dis-tance to the interpolated visibility. Because of the small mag-nitude of the effect that averaging of flagged samples has,improved interpolation will likely be of inconsiderable smallmagnitude for most science cases. Nevertheless, it might stillbe relevant for reaching high dynamic ranges, for examplein continuum imaging. Because of the small computationalcost of the method, it is straightforward to implement.

ACKNOWLEDGEMENTS

A. R. Offringa acknowledges financial support from the Eu-ropean Research Council under ERC Advanced Grant LO-FARCORE 339743. FGM and LVEK would like to acknowl-edge support from a SKA-NL Roadmap grant from theDutch ministry of OCW. We thank M. Mevius for helpfuldiscussions.

REFERENCES

Barry N., Hazelton B., Sullivan I., Morales M. F., Pober J. C.,2016, MNRAS, 461, 3135

Beardsley A. P., et al., 2016, ApJ, 833, 102

Bernardi G., de Bruyn A. G., Harker G., et al., 2010, A&A, 522,

A67

Callingham J. R., et al., 2017, ApJ, 836, 174

Carroll P. A., et al., 2016, MNRAS, 461, 4151

Chapman E., et al., 2013, MNRAS, 429, 165

Datta A., Bowman J. D., Carilli C. L., 2010, ApJ, 724, 526

Ewall-Wice A., Dillon J. S., Liu A., Hewitt J., 2017, MNRAS,

470, 1849

Franzen T. M. O., Jackson C. A., Offringa A. R., et al., 2016,

MNRAS, 459, 3314

Furlanetto S. R., Oh S. P., Briggs F. H., 2006, Physics Reports,

433, 181

Ghosh A., Mertens F. G., Koopmans L. V. E., 2018, MNRAS,

474, 4552

Greig B., Mesinger A., 2015, MNRAS, 449, 4246

van Haarlem M. P., et al., 2013, A&A, 556, A2

Harker G., et al., 2009, MNRAS, 397, 1138

Hurley-Walker N., et al., 2017, MNRAS, 464, 1146

Iliev I. T., Shapiro P. R., Ferrara A., Martel H., 2002, ApJ, 572,

L123

Jacobs D. C., et al., 2016, ApJ, 825, 114

Jelic V., et al., 2008, MNRAS, 389, 1319

Kazemi S., Yatawatta S., Zaroubi S., Labropoulos P., de A. G.,Koopmans L. V. E., Noordam J., 2011, MNRAS, 414, 1656

Liu A., Tegmark M., Zaldarriaga M., 2009, MNRAS, 394, 1575

McQuinn M., Zahn O., Zaldarriaga M., Hernquist L., Furlanetto

S. R., 2006, ApJ, 653, 815

Mertens F. G., Ghosh A., Koopmans L. V. E., 2018, MNRAS,

478, 3640

Middelberg E., 2006, PASA, 23, 64

Morales M. F., 2005, ApJ, 619, 678

Morales M. F., Matejek M., 2009, MNRAS, 400, 1814

Morales M. F., Hazelton B., Sullivan I., Beardsley A., 2012, ApJ,752, 137

Nuttall A., 1981, IEEE Transactions on Acoustics, Speech, andSignal Processing, 29, 84

Offringa A. R., 2016, A&A, 595, A99

Offringa A. R., de Bruyn A. G., Biehl M., Zaroubi S., Bernardi

G., Pandey V. N., 2010, MNRAS, 405, 155

Offringa A. R., van de Gronde J. J., Roerdink J. B. T. M., 2012,

A&A, 539

Offringa A. R., et al., 2013, A&A, 549, A11

Offringa A. R., McKinley B., Hurley-Walker N., et al., 2014, MN-

RAS, 444, 606

Offringa A. R., et al., 2015, PASA, 32, e008

Offringa A. R., et al., 2016, MNRAS, 458, 1057

Parsons A. R., Backer D. C., 2009, AJ, 138, 219

Parsons A. R., Pober J., McQuinn M., Jacobs D., Aguirre J.,

2012, ApJ, 753, 81

Patil A. H., et al., 2016, MNRAS, 463, 4317

Patil A. H., Yatawatta S., Koopmans L. V. E., et al., 2017, ApJ,

838, 65

MNRAS 000, 1–11 (2019)

RFI excision effects in 21-cm power spectra 11

Peck L. W., Fenech D. M., 2013, Astronomy and Computing, 2,

54

Prasad J., Chengalur J., 2012, Experimental Astronomy, 33, 157Procopio P., et al., 2017, PASA, 34, e033

Rasmussen C. E., Williams C. K. I., 2005, Gaussian Processes

for Machine Learning (Adaptive Computation and MachineLearning). The MIT Press

Scaife A. M. M., Heald G. H., 2012, MNRAS Lett., 423, L30Trott C. M., Wayth R. B., 2017, PASA, 34, e061

Trott C. M., et al., 2016, ApJ, 818, 139

Vedantham H., Shankar N. U., Subrahmanyan R., 2012, ApJ, 745,176

Winkel B., Kerp J., Stanko S., 2006, AN, 88, 789

Yatawatta S., 2014, in XXXIth URSI General Assembly and Sci-entific Symposium.

Yatawatta S., 2016. EURASIP (arXiv:1605.09219)

Yatawatta S., de Bruyn A. G., Brentjens M. A., et al., 2013, A&A,550

This paper has been typeset from a TEX/LATEX file prepared by

the author.

MNRAS 000, 1–11 (2019)