SAS development, maintenance and future | Carlos Gabriel | XMM-Newton Users Group meeting - ESAC, 7/8 June 2016
Carlos GABRIEL (*)
XMM-Newton Science Operations Centre – ESAC / ESA
*on behalf of a lot of people (SAS & PPS Team, SAS WG, ...)
SAS development, maintenance and future
SAS development, maintenance and future | Carlos Gabriel | XMM-Newton Users Group meeting - ESAC, 7/8 June 2016
SAS & PPS development and maintenance
MPE: PN s/w - (2) AIP: src det s/w - (1) GOF: esas - (2) IRAP: pps s/w - (1) Strasbourg: ACDS s/w - (1) IFCA: screening s/w - still providing support CEA: MOS s/w - only support since 2009 LUX: ended June 2013 (1)MSSL: OM s/w ended March 2014
SAS CCBSOC (5) + SSC (3)
SAS CAL filesPPS
SAS WGCal Teams
SOC SAS&PPS
SSC SAS&PPS
XSA
Team of 8 (~ 5FTEs) SAS ~ 3 FTEs PPS ~ 2 FTEs
SAS development, maintenance and future | Carlos Gabriel | XMM-Newton Users Group meeting - ESAC, 7/8 June 2016
SAS development & maintenance
Main SAS upgrades contain• CAL changes - improvements in instrument calibration• algorithmic changes - and/or better analysis • graphical I/F changes - better I/Fs (or eg. changes in parameters)• changes due to OS / libs / compiler - forced evolution
A typical yearly release implies• many new task versions (~ 102) - despite age of SAS• few new tasks (~ 100) - not growing much anymore • scientific validation - quality assurance always needed• preparation and distribution of SAS - making it as easy and secure as possible to users
Further work• analysis of Helpdesk questions - continuous maintenance, evtl. bugs found• documentation maintenance & upgrade - threads, watch-outs, ...• follow-up measures - patches, watch-outs, individual help
SAS development, maintenance and future | Carlos Gabriel | XMM-Newton Users Group meeting - ESAC, 7/8 June 2016
SAS 15.0 released on 4/2/16 - Main changes
GENERAL- correction of the conversion between celestial and detector coordinate systems.
Euler Ψ angle with wrong sign in attcalc >> correctedSeveral tasks affected + new CCF (XMM_BORESIGHT_0026.CCF)
EPIC- new task ebkgreg for determination of optimal background region for a given source (PN)- new task eupper for fast calculation of upper limits- upgraded eboxdetect, accepting now also input in detector coordinates- the metatask eslewchain is now producing .png files for each subimage
RGS- upgraded task rgsrmfgen, able to deliver responses in separated RMF and ARF files
+ many fixes and small improvements: 72 tasks upgraded, with 129 sub-version changes
SAS development, maintenance and future | Carlos Gabriel | XMM-Newton Users Group meeting - ESAC, 7/8 June 2016
SAS 15.0 released on 4/2/16 - validation
Thorough validation necessary (“(4+1)way”):
• standard set through procs & chains > IA• X-cal DB fully reduced > comparisons• standard set through test PPS (SAS 15 based)• strict data reduction with the Threads+• dedicated data reduction for new / special S/W
>> several people involved in 4 weeks process+ specific time for validation of new attcalc, special effort by the instrument teams
>> Final report still to be produced
>> CAL-SRN 332 on BORESIGHT_0026
Extra this time: - tests of single released 32bit version in different OSs, both automatic and interactively
SAS development, maintenance and future | Carlos Gabriel | XMM-Newton Users Group meeting - ESAC, 7/8 June 2016
SAS validation - conclusions
Everything else validated but:
• use of modified embadpixfind for RGS bad pixel / column finding NOT validated (it needs
parameter space assessment) >> withdrawn from SAS 15
• Cross check of 32bit “single” version from other 32bits OSs OK ... but later Perl problems arising >> diverse 32bit versions released
SAS development, maintenance and future | Carlos Gabriel | XMM-Newton Users Group meeting - ESAC, 7/8 June 2016
SAS validation - rgsembadpixfind
Processing of
AB Dor: 35 observationsHR1099: 33 observationsCapella: 27 observationsMrk 421: 64 observations
Default rgsproc reduction with a) rgsbadpixfind b) rgsembadpixfind
Fit model, compare valid bins and fit goodness...
Mrk 421- Continuum - No differences at all
Motivation: SPR on wrong filtering of RGS bright emission lines
SAS development, maintenance and future | Carlos Gabriel | XMM-Newton Users Group meeting - ESAC, 7/8 June 2016
SAS validation - rgsembadpixfind
Fitting simultaneously RGS1 and RGS2, 1st and 2nd order in all 35 observations
• In almost all cases, less columns (typically 100) discarded• In almost all cases, smaller Cash/DoF, ie. good fitting of non-discarded columns ... BUT 3 cases with non-discarded hot columns
>> need assessment of full parameter space of new task rgsembadpixfind
☐ ☐
AB Dor data: use of APED model with 3 T’s and free abundances - fitting with ISIS (based on Sanz-Forcada, J.; Micela, G.; Maggio, A. 2007)
SAS development, maintenance and future | Carlos Gabriel | XMM-Newton Users Group meeting - ESAC, 7/8 June 2016
Patch 15.0.1 released in May
• preqgti had been kept (wrongly!) as “SOC only”, therefore not part of the public SAS 15 release- also necessary for explaining to users how to deal with multi-pointing mode in a watchout
• esas (v0.9.34) - including missing task merge_comp_xmm
• xmmextractor (v1.4.1) solving various small incompatibilities with SAS 15 tasks
• emosaicproc (v0.4.1) corrects typo in a parameter of emldetect call
• ommergelists (v1.0.5) solves problems with duplicated entries from mosaiced sources
• omsrclistcomb (v2.22.2) OM windows assignment fixed
SAS development, maintenance and future | Carlos Gabriel | XMM-Newton Users Group meeting - ESAC, 7/8 June 2016
SAS 15 release - binaries
Linux 32:
Linux 64:
MacOS:
RHEL 5.8 2.6.18 2.5
Ubuntu 14.04 LTS 3.13.0 2.19
Fedora 20 3.19.8 2.18
SuSE 13.1 3.11.6 2.18
MacOS 10.8.3 (M. Lion) Darwin 12.6.0 169.3.0
MacOS 10.10.5 (Yosemite) Darwin 14.5.0 1213.0.0
+ 2 universal SAS-VMs (32 & 64bits) - Ubuntu14.04.1
Already in SAS 14
New in SAS 15
RHEL 5.8 2.6.18 2.5
Ubuntu 14.04 LTS 3.13.0 2.19
Fedora 20 3.11.10 2.18
SuSE 13.1 3.11.6 2.18
RHEL 6.6 2.6.32 2.1
OS Kernel libc
SAS development, maintenance and future | Carlos Gabriel | XMM-Newton Users Group meeting - ESAC, 7/8 June 2016
Collecting information with every SAS release - SAS 15.0
SAS development, maintenance and future | Carlos Gabriel | XMM-Newton Users Group meeting - ESAC, 7/8 June 2016
GENERAL- migration to gfortran > ending dependency from NAG- graphical I/F for xmmextractor > new type of I/F due to special characteristics of task- study on replacement of PGPLOT by modern package > better output quality (same plotting as Chandra?)
EPIC- esas general refactoring > re-written in F90 + CAL-DB into normal CCFs- task for EPIC internal background event files > image and spectral correction- task for thumbnails incl. src & bck extraction reg. > fast diagnosis- diagnostic X-ray loading images from epproc > assess X-ray loading level
- refinement of pile-up corrections > optimisation / depends on study going on- source detection in slew data processing > slew pipeline providing source detection
- detector map to correct for bad areas in ext. sources > extended source analysis - time dependent PN quiescent background estimation > better gain correction- emosaicproc and emosaicprep upgrades > make data combination more user friendly - alternative stacked source detection task > better src detection on stacked data - time dependent PN quiescent background gain correction > use of instantaneous discarded line rates- different PSFs for different observing modes > to achieve calibration goals for timing / burst modes
Future main SAS upgrades
SAS development, maintenance and future | Carlos Gabriel | XMM-Newton Users Group meeting - ESAC, 7/8 June 2016
Future main SAS upgrades
RGS- finish / validate new bad column finding task > avoiding bright lines loses due to misidentification- new RGS1/RGS2 eff. area correction > cross calibration improvement- correct treatment of multi-pointing observations > solve problems of single observations in multipointing- spatial imaging of emission lines from ext sources > narrow energy range images > less background than EPIC
OM- NONE by default ... but some flags should be extracted and added to the catalogue file, output of omcat
SAS development, maintenance and future | Carlos Gabriel | XMM-Newton Users Group meeting - ESAC, 7/8 June 2016
Compilers
Migrating to gfortran
> major change, will free us from NAG - main change for SAS 16, it will require substantial tests
Integration tests are running - several tasks already adapted >> our goal is to be ready by September
SAS development, maintenance and future | Carlos Gabriel | XMM-Newton Users Group meeting - ESAC, 7/8 June 2016
Remote Interface for Science Analysis (RISA)
JOBS
1
2
4
5
6
7
8
9
SERVICE
REQUEST
SIAP and CDS query
VO
ESP
AC
E
GRID
XSA REQUES
STORAGE REQUEST
SERVICE INFO
RESULTS
VOSpec
10
•Process SOAP (VOTable)•Unique REQID•Perl Script•Grid Template•Job Submission & Status Info
3
Aladin
( )
UNIVA Grid Engine
RISA is ready to run...
SAS development, maintenance and future | Carlos Gabriel | XMM-Newton Users Group meeting - ESAC, 7/8 June 2016
RISA upgraded to process “light-services” from XSA:★ Spectra ★ Light curves ★ Images★ Source detection (>> Spectra, Light curves, ...)
RESTful syntax: http://<InternalServer>/RISA_sl? product=spectra& username=<XSAUserName>& obsid=0650560401& inst=EPN& exp=S003& target=Mkn501& srcreg=((X,Y) IN circle(25339.5,23998.5,328.9302))& bkgreg=((X,Y) IN circle(22491.5,26702.5,647.65702))& mail=<XSAMailAddress>
ds9 has been upgraded (there were security problems impeding its usage in XSA)
RISA web services to be used from XSA
SAS development, maintenance and future | Carlos Gabriel | XMM-Newton Users Group meeting - ESAC, 7/8 June 2016
Futuredepending on feedback...
• New functionalities• User´s defined filtering• RGS and OM
• New “light services”• Combine ODFs
• Open RESTful services to the world
•…
Reprocessing capabilities
Top Related