NIR Transient Surveys

Nicholas CrossWFAU, Edinburgh

Nigel Hambly, Mike Read, Ross Collins, Eckhard Sutorius, Rob Blake, Mark Holliman

IAU S285, Oxford 2

NIR Variability Science Drivers

• NIR, smaller detectors, higher backgrounds and more expensive detectors than optical– Only do multi-epoch work where it is not practical for

optical detectors– Looking through the dense dusty regions of the MW

to the far side– Young Stellar Objects in star-forming regions– Low mass stars / brown dwarfs– High z galaxies / Snae– Can get better RR Lyrae / Cepheid distances in NIR

19/09/2011

IAU S285, Oxford 3

NIR Variability Surveys

• UKIRT WFCAM – UKIDSS – DXS/UDS (Deep surveys, multi-epoch), – WFCAM Transit Survey,– Calibration/Standard Stars, – Surveys of YSOs in Orion/Ophiuchus

• VISTA– VISTA Variables in Via-Lactea (VVV), (RR Lyrae,

Cepheids)– VISTA Magellanic Cloud (VMC), (RR Lyrae, Cepheids)– VIDEO (Deep Extragalactic – SNae)

19/09/2011

IAU S285, Oxford 4

WFCAM• 3.8 m UKIRT telescope

on Mauna Kea.• 4 2k x 2k Rockwell

Hawaii 2 detectors.• Spaced 94% apart. • 0.4” pixels.• 13.65’ across each side.• 60% of time on UKIRT in

2005b• 100% for 2009a

19/09/2011

IAU S285, Oxford 5

VIRCAM

19/09/2011

• 4.1m VISTA telescope at Cerro Paranel.• 16 2k x 2k Raytheon VIRGO detectors• Spaced 90% in x and 42.5% in y.• 0.34” pixels• Tile is 1.5° • VIRCAM has 100% of time. • > 3 times area WFCAM• 2 * QE

IAU S285, Oxford 6

VISTA Public Surveys

19/09/2011

IAU S285, Oxford 7

VISTA Variables in Via-Lactea (VVV)

• Very high density ~106 sources / sq. deg.– Issues with

deblending• 500 sq. deg• ~100 epochs

(currently ~10)• ~ few 1010 detections

19/09/2011

IAU S285, Oxford 8

Processing of WFCAM and VISTA data

• VDFS: VISTA Data Flow System (System for processing of UKIRT WFCAM and VISTA data.– CASU (Cambridge): Data reduction, processing of

observing blocks, photometric and astrometric calibration

– WFAU (Edinburgh): Archive, processing of multiple observing blocks – deep stacks, multi-band tables, links to external tables, MULTI-EPOCH.

– For VISTA, data goes through ESO and final products go to ESO too.

19/09/2011

IAU S285, Oxford 9

Constraints from VDFS• >=6 week time lag before data at WFAU– Data needs to be transferred to Cambridge (with VISTA

this includes disk drive to Garching and then to Cambridge)

– Accurate photometric calibration (including scattered light corrections uses 1 month of data.

– VoEvent alerts are too late from WFAU• Reprocessing of OB data requires retransfer between

CASU and WFAU and reingest of data at WFAU. – Detection tables are used by many curation processes –

reingestion into these slows later stages.

19/09/2011

IAU S285, Oxford 10

Stages of multi-epoch processing

• Stack epochs to create deep images and extract catalogues

• Create master list (Source table) from band-merged catalogues from deep images.

• Recalibrate each epoch image compared to the deep image in that filter and pointing.

• Create table linking sources to each observation • Calculate the noise properties of each pointing and filter• Calculate astrometric and photometric statistics for

each source.

19/09/2011

IAU S285, Oxford 11

Analysing Variables

• Calculate mean, rms of magnitudes.

• Bin in magnitude and calculate clipped median

• Fit empirical noise model • (m)=a+b10-0.4m+c10-0.8m

• Classify as variable or non-variable

19/09/2011

IAU S285, Oxford 12

Archival Databases• Curation of WFCAM and VISTA data occurs in a

RDBMS using Microsoft SQL Server. – Dynamic database, updated with new data,

improved calibrations and reprocessed data when necessary.

– Static releases to the science teams and world for science purposes.

• Curation controlled by comparing current state of DB with requirements

19/09/2011

IAU S285, Oxford 13

Programme Requirements• Pointing, filter and table requirements are setup by

grouping the metadata and using specifications for each survey.

• Schema updated if necessary• Stack / tile products made for a particular release

number• Source table created for particular pointings• Each stage of multi-epoch processing checks the

whether the previous table has changed in that pointing – higher curation event ID.

19/09/2011

IAU S285, Oxford 14

VISTA tiles• Most surveys require tiles to reach expected depth, and

tiles are standard ESO product.• PSF and sky vary on short time scales < integration time• Images filtered to remove large spatial variations (>30”)• Tile catalogues are inferior to pawprints:

– Not as accurate astrometry– Do not deal with saturation correctly– Extended (>30”) sources are missing or have incorrect

photometry• Catalogues from tiles and pawprints

– Need to be able to compare – multiple layers and linking tables.

19/09/2011

IAU S285, Oxford 15

Problems / bottlenecks /solutions• Reprocessing of OB data.

– 1st year of VISTA – 2 sets of full reprocessing• Ingesting new data while curating later products

– Put VVV on separate server and synchronise metadata tables• BUT foreign key constraints to vvvDetection cause major holdups if

metadata is deleted.

– Split vvvDetection into semesters / months so new data can be ingested into new semester.• Has not been implemented yet

• Users want to use both tile and pawprint detections– Produce linking tables

• BUT some queries that join these can join several tens of tables and SQL does not handle these joins well.

• Enhancements to user interface allow users to save intermediate results

19/09/2011

IAU S285, Oxford 16

Problems / bottlenecks /solutions• Checking non-detections of sources

– Using half-space method of Budavari, major improvement• Dealing with very long processing times of VVV

– Break curation into chunks with software testing to see what has already been done

– Make sure memory never exceeds ~40% – BUT this adds additional overheads at beginning of each run

• Variability table curation is dominated by DB reads (85% for VVV)– Use Query Analyser and other tools to optimise queries [OPTION

(MAXDOP 1)], adding removing indexes.– Split detection tables into parts?

• I/O limited between servers and disks– SQL Server “cluster” linked by infiniband 10Gbs-1

19/09/2011

IAU S285, Oxford 17

Other issues• Classification

– DB has simple classification (variable or not) and some other statistical quantities. VVV will have ~106 variables

– Chilean teams working on NIR templates for different types of variables

– Trend analysis (Istvan Dekany)• Accuracy

– VSA/WSA, simple ZP recalibration – rms ~0.005mag• Good enough for most variables• Planetary Transits require (prefer) ~0.001 mag.

• Confusion– Difference Imaging Analysis (Eamonn Kerins), will probably be

applied to densest 40 sq. deg of VVV bulge.

19/09/2011