Modeling, Fitting and Statistics - Chandra X-ray...
Transcript of Modeling, Fitting and Statistics - Chandra X-ray...
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics1
Modeling, Fitting and Statistics
Aneta SiemiginowskaCXC Science Data System
http://cxc.harvard.edu/sherpa
CXC DS Sherpa Team:Stephen Doe, Dan Nguyan, Brian Refsdal
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics2
Outline
♦ What is Sherpa?♦ Observations and Models♦ Statistics♦ Fitting, optimization and results♦ Summary and Conclusion
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics3
CIAO’s Modeling and Fitting packageo Generalized package with a powerful model language to fit 1D and 2D datao Forward fitting technique - a model is evaluated, compared to the actual data, and then the
parameters are changed to improve the match. This is repeated until convergence occurs.o Beta version in CIAO 4.0 with Python and S-lang scripting languageo New release planned for Dec.2008 in CIAO4.1o Significant re-write to modularize the code for future improvementso Walkthrough with a few examples
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics4
Modeling: what can I learn from new observations?
• Data:o Write proposal, win and obtain new data
• Models:o model library that can describe the physical process in the sourceo typical functional forms or tables, derived more complex models - plasma
emission models etc.o parameterized approach - models have parameters
• Optimization Methods:o to apply model to the data and adjust model parameterso obtain the model description of your datao constrain model parameters etc. search of the parameter space
• Statistics:o a measure of the model deviations from the data
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics5
Observations: Chandra Data and more…
• X-ray Spectra typically PHA files with the RMF/ARF calibration files
• X-ray Images FITS images, exposure maps, PSF files
• Lightcurves FITS tables, ASCII files
• Derived functional description of the source:• Radial profile• Temperatures of stars• Source fluxes
• Concepts of Source and Background data
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics6
Observations: Data I/O in Sherpa
• Load functions (PyCrates) to input the data:data: load_data, load_pha, load_arrays, load_asciicalibration: load_arf, load_rmf load_multi_arfs, load_multi_rmfsbackground: load_bkg, load_bkg_arf , load_bkg_rmf2D image: load_image, load_psfGeneral type: load_table, load_table_model, load_user_model
• Multiple Datasets - data id
• Filtering of the dataload_data expressionsnotice/ignore commands in Sherpa
Help file:load_data( [id=1], filename, [options] )load_image( [id=1], filename|IMAGECrate,[coord="logical"] )
Examples:load_data("src", "data.txt", ncols=3)
load_data("rprofile_mid.fits[cols RMID,SUR_BRI,SUR_BRI_ERR]")load_data(“image.fits”)load_image(“image.fits”, coord=“world”))
Default data id =1load_data(2, “data2.dat”, ncols=3)
Examples:notice(0.3,8)notice2d("circle(275,275,50)")
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics7
Observations: Data I/O in Sherpa• Information about the data: show_data(), show_bkg()sherpa> show_all()Optimization Method: LevMarStatistic: Chi2Gehrels
Data Set: 1name = 3c273.pichannel = Int32[1024]counts = Int32[1024]staterror = Nonesyserror = Nonebin_lo = Nonebin_hi = Nonegrouping = Int16[1024]quality = Int16[1024]exposure = 38564.6089269backscal = 2.52643646989e-06areascal = 1.0grouped = Truesubtracted = Falseunits = energyresponse_ids = [1]background_ids = [1]
RMF Data Set: 1:1name = 3c273.rmfdetchans = 1024
sherpa> load_pha("3c273.pi")statistical errors were found in fi le '3c273.pi'but not used; to use them, re-read with use_errors=Trueread ARF file 3c273.arfread RMF file 3c273.rmfstatistical errors were found in fi le '3c273_bg.pi'but not used; to use them, re-read with use_errors=Trueread background fi le 3c273_bg.pi
sherpa> load_arf("3c273.arf")sherpa> load_arf("3c273.rmf")sherpa> load_bkg("3c273_bg.pi")
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics8
• Parameterized models: f(E,Θi) or f(xi,Θi)absorption - NH
photon index of a power law function - Γblackbody temperature kT
• Composite models: combined individual models in the library into a model that describes the observation
• Source models, Background models:
set_model(“xsphabs.abs1*powlaw1d.p1”)set_model(“const2d.c0+gauss2d.g2”)
Modeling: Model Concept in Sherpa
set_source(2,"bbody.bb+powlaw1d.pl+gauss1d.line1+gauss1d.line2")set_bkg_model(2,”const1d.bkg2”)
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics9
Modeling: Sherpa Models
• Model Library that includes XSPEC models
• User Models:• Python or Slang Functionsload_user_model, add_user_pars
• Python and Slang interface toC/C++ or Fortran code/functions
sherpa-11> list_models()['atten', 'bbody', 'bbodyfreq', 'beta1d', 'beta2d', 'box1d',…
Example Function myline:def myline(pars, x): return pars[0] * x + pars[1]
In sherpa:from myline import *
load_data(1, "foo.dat")load_user_model(myline, "myl")add_user_pars("myl", ["m","b"])set_model(myl)myl.m=30myl.b=20
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics10
Modeling: Parameter Space
sherpa-21> set_model(xsphabs.abs1*xszphabs.zabs1*powlaw1d.p1)sherpa-22> abs1.nH = 0.041sherpa-23> freeze(abs1.nH)sherpa-24> zabs1.redshift=0.312
sherpa-25> show_model()Model: 1apply_rmf(apply_arf((106080.244442 * ((xsphabs.abs1 * xszphabs.zabs1) * powlaw1d.p1)))) Param Type Value Min Max Units ----- ---- ----- --- --- ----- abs1.nh frozen 0.041 0 100000 10^22 atoms / cm^2 zabs1.nh thawed 1 0 100000 10^22 atoms / cm^2 zabs1.redshift frozen 0.312 0 10 p1.gamma thawed 1 -10 10 p1.ref frozen 1 -3.40282e+38 3.40282e+38 p1.ampl thawed 1 0 3.40282e+38
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics11
Standard PHA based analysis in Sherpa:
• Source data:• can be modeled in energy/wavelength space.• multiple data sets can be modeled with the same or different models in one Sherpa session.• data can be filtered on the command line, or from filter file.
• Instrument responses (RMF/ARF):• are entered independently from the source data.• one set of instrument responses can be read once and applied to multiple data sets.• several instrument responses used in analysis of one source model or multiple data sets.• multiple response files can be used in one source model expression.
• Background files:• are entered independently from the source data.• multiple background files can be used for one data set,e.g. grating analysis• the same background can be applied to multiple data sets.• background can be modeled independently of the source data, and have its separate
instrument responses.• background can be modeled simultaneously with the source data.• background can be subtracted from the source data (subtract/unsubtract).
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics12
What do we really do?Example:
I've observed my source, reduce the data and finally got my X-rayspectrum – what do I do now? How can I find out what does thespectrum tell me about the physics of my source?
Run Sherpa! But what does this program really do?
Chandra ACIS-S
Fit the data => C(h)=∫R(E,h) A(E) M(E,θ)dE
Assume a model and look for the best modelparameters which describes the observedspectrum.
Need a Parameter Estimator - Statistics
Counts Response Effective Area
Model
h- detector channelsE- Energyθ- model parameters
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics13
Parameter Estimators - Statistics
Large variance
Best
Biased
θ0S
tatis
tic
Requirements on Statistics:
• Unbiased- converge to true value withrepeated measurements
• Robust– less affected by outliers
• Consistent– true value for a large samplesize (Example: rms and Gaussiandistribution)
• Closeness- smallest variations from thetruth
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics14
One can use the Poisson distribution to assess the probability of sampling data Di given a predicted(convolved) model amplitude Mi. Thus to assess the quality of a fit, it is natural to maximize the product ofPoisson probabilities in each data bin, i.e., to maximize the Poisson likelihood:
In practice, what is often maximized is the log-likelihood,
L = logℒ. A well-known statistic in X-ray astronomy which is related to L is the so-called “Cash statistic”:
Maximum Likelihood:Assessing the Quality of Fit
∝
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics15
(Non-) Use of the Poisson Likelihood
In model fits, the Poisson likelihood is not as commonly used as it should be. Some reasonswhy include:
• a historical aversion to computing factorials;
• the fact the likelihood cannot be used to fit “background subtracted” spectra;
• the fact that negative amplitudes are not allowed (not a bad thing physics abhors negativefluxes!);
• the fact that there is no “goodness of fit" criterion, i.e. there is no easy way to interpret ℒmax(however, cf. the CSTAT statistic); and
• the fact that there is an alternative in the Gaussian limit: the χ2 statistic.
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics16
χ2 Statistic
Definition: χ2= ∑i (Di-Mi)2 / Mi
The χ2 statistics is minimized in the fitting the data,varying the model parameters until the best-fit modelparameters are found for the minimum value of theχ2 statistic
Degrees-of-freedom = k-1- N
N – number of parametersK – number of spectral bins
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics17
The version of χ2 derived above is called “data variance” χ2 because of the presence of D in the denominator.Generally, the χ2 statistic is written as:
where represents the (unknown!) variance of the Poisson distribution from which Di is sampled.
Sherpa χ2 Statisticnamechi2datavar Data Variance Di
chi2modvar Model Variance Mi
chi2gehrels Gehrels [1+(Di+0.75)1/2]2
chi2constvar “Parent”
leastsq Least Squares 1
Note that some X-ray data analysis routines may estimate σi during data reduction.In PHA files, such estimates are recorded in the STAT_ERR column.
“Versions” of the χ2 Statistic in Sherpa
22
2
( ),
N
i i
i i
D M
!
"# $ %
2
i!
2
i!
1
N
iiD
N
=!
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics18
Statistic in Sherpa
• χ2 statistics with different weights• Cash and Cstat based on Poisson likelihood
sherpa-12> list_stats()['leastsq', 'chi2constvar', 'chi2modvar', 'cash', 'chi2gehrels', 'chi2datavar', 'chi2xspecvar', 'cstat']sherpa-13> set_stat(“chi2datavar”)sherpa-14> set_stat(“cstat”)
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics19
Statistics - Example of Bias
• The χ2 bias can affect the results of the X-ray spectral fitting
• Simulate Chandra spectrum given RMF/ARF and the Poissonnoise - using fake_pha().
• The resulting simulated X-ray spectrum contains the modelpredicted counts with the Poisson noise. This spectrum is then fitwith the absorbed power law model to get the best fit parametervalue.
• Simulated 1000 spectra and fit each of them using differentstatistics: chi2datavar, chi2modvar and Cash.
•Plot the distribution of the photon index in the simulations withΓ=1.267.
underestimated overestimated
Very High S/N data!
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics20
Fitting: Search in the Parameter Space
sherpa-28> fit()Dataset = 1Method = levmarStatistic = chi2datavarInitial fit statistic = 644.136Final fit statistic = 632.106 at function evaluation 13Data points = 460Degrees of freedom = 457Probability [Q-value] = 9.71144e-08Reduced statistic = 1.38316Change in statistic = 12.0305 zabs1.nh 0.0960949 p1.gamma 1.29086 p1.ampl 0.000707365
sherpa-29> print get_fit_results()datasets = (1,)methodname = levmarstatname = chi2datavarsucceeded = Trueparnames = ('zabs1.nh', 'p1.gamma', 'p1.ampl')parvals = (0.0960948525609, 1.29085977295, 0.000707365006941)covarerr = Nonestatval = 632.10587995istatval = 644.136341045dstatval = 12.0304610958numpoints = 460dof = 457qval = 9.71144259004e-08rstat = 1.38316385109message = both actual and predicted relative reductions in the sum ofsquares are at mostftol=1.19209e-07nfev = 13
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics21
Fitting: Sherpa Optimization Methods• Optimization - a minimization of a function:
“A general function f(x) may have many isolated local minima, non-isolated minimumhypersurfaces, or even more complicated topologies. No finite minimization routinecan guarantee to locate the unique, global, minimum of f(x) without being fed intimateknowledge about the function by the user.”
• Therefore:1. Never accept the result using a single optimization run; always test the minimum using a different method.2. Check that the result of the minimization does not have parameter values at the edges of the parameter
space. If this happens, then the fit must be disregarded since the minimum lies outside the space thathas been searched, or the minimization missed the minimum.
3. Get a feel for the range of values of the fit statistic, and the stability of the solution, by starting theminimization from several different parameter values.
4. Always check that the minimum "looks right" using a plotting tool.
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics22
Fitting: Optimization Methods in Sherpa
• “Single - shot” routines: Simplex and Levenberg-Marquardt start from a guessed set of parameters, and then try to improve the
parameters in a continuous fashion:• Very Quick• Depend critically on the initial parameter values• Investigate a local behaviour of the statistics near the guessed parameters, and then
make another guess at the best direction and distance to move to find a betterminimum.
• Continue until all directions result in increase of the statistics or a number of steps hasbeen reached
• “Scatter-shot” routines: Monte Carlo try to look at parameters over the entire permitted parameter space to
see if there are better minima than near the starting guessed set ofparameters.
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics23
Final Analysis Steps
• How well are the model parameters constrained by the data?• Is this a correct model?• Is this the only model?• Do we have definite results?• What have we learned, discovered?• How our source compares to the other sources?• Do we need to obtain a new observation?
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics24
Confidence Limits
Essential issue = after the bets-fit parameters are found estimate theconfidence limits for them. The region of confidence is given by(Avni 1976):
χ2α = χ2
min +Δ(ν,α)
ν - degrees of freedom α - significance χ2
min - minimum
Δ depends only on the number of parameters involved not on goodness of fit
Significance Number of parameters α 1 2 3
0.68 1.00 2.30 3.50 0.90 2.71 4.61 6.25 0.99 6.63 9.21 11.30
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics25
Example of a “well-behaved” statistical surface inparameter space, viewed as a multi-dimensionalparaboloid (χ2, top), and as a multi-dimensionalGaussian (exp(-χ2 /2) ≈ L, bottom).
Calculating Confidence Limits means
Exploring the Parameter Space -Statistical Surface
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics26
sherpa-39> covariance()Dataset = 1Confidence Method = covarianceFitting Method = levmarStatistic = chi2datavarcovariance 1-sigma (68.2689%) bounds: Param Best-Fit Lower Bound Upper Bound ----- -------- ----------- ----------- zabs1.nh 0.0960949 -0.00436915 0.00436915 p1.gamma 1.29086 -0.00981129 0.00981129 p1.ampl 0.000707365 -6.70421e-06 6.70421e-06
sherpa-40> projection()Dataset = 1Confidence Method = projectionFitting Method = levmarStatistic = chi2datavarprojection 1-sigma (68.2689%) bounds: Param Best-Fit Lower Bound Upper Bound ----- -------- ----------- ----------- zabs1.nh 0.0960949 -0.00435835 0.00439259 p1.gamma 1.29086 -0.00981461 0.00983253 p1.ampl 0.000707365 -6.68862e-06 6.7351e-06
sherpa-48> print get_proj_results()datasets = (1,)methodname = projectionfitname = levmarstatname = chi2datavarsigma = 1percent = 68.2689492137parnames = ('zabs1.nh', 'p1.gamma', 'p1.ampl')parvals = (0.0960948525609, 1.29085977295, 0.000707365006941)parmins = (-0.00435834667074, -0.00981460960484, -6.68861977704e-06)parmaxes = (0.0043925901652, 0.00983253275984, 6.73510303179e-06)nfits = 46
Confidence Intervals
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics27
sherpa-61> reg_proj(p1.gamma,zabs1.nh,nloop=[20,20])sherpa-62> print get_reg_proj()min = [ 1.2516146 0.07861824]max = [ 1.33010494 0.11357147]nloop = [20, 20]fac = 4delv = Nonelog = [False False]sigma = (1, 2, 3)parval0 = 1.29085977295parval1 = 0.0960948525609levels = [ 634.40162888 638.28595426 643.93503803]
Confidence Regions
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics28
Behaviour of Statistics for One Parameter
Comparison of Two methods in Sherpa
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics29
Confidence Limits for Two Parameters
1σ, 2σ, 3σ contours+ Best fit parameters
Comparison of Two methods in Sherpa
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics30
Chandra Plotting Package
CIAO4 Infrastructure changed SM replaced with the VTK Modern graphics package Publication quality plots
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics31
A Simple ProblemFit Chandra 2D Image data in Sherpa
using Command Line Interface in Python
• Read the data• Choose statistics and optimization method• Define the model• Minimize to find the best fit parameters for the model• Evaluate the best fit - display model, residuals, calculate uncertainties
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics32
A Simple ProblemList of Sherpa Commands
Read Image data and Display in ds9
Set Statistics andOptimization Method
Define Model and SetModel parameters
Fit, DisplayGet Confidence Range
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics33
A Simple Problem
List of Sherpa Commands
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics34
List of Sherpa CommandsCommand Line View
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics35
Sherpa Scripts
Old New
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics36
Setup Environment
Import Sherpaand Chips
Set the System
Define directories
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics37
Loops
Model Parameters
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics38
A Complex ExampleFit Chandra and HST Spectra with Python script
• Setup the environment• Define functions• Run script and save results in nice format.• Evaluate results - do plots, check uncertainties, derive data and do
analysis of the derived data.
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics39
Setup
Optical spectra
X-ray spectra
UnitsConversion
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics40
Fit ResultsX-ray data with RMF/ARF and Optical Spectra in ASCII
Log ν
Log νF
ν
Quasar SED
Wavelength
Optical
X-ray
Energy
Flux
Optical
X-ray
CXC 6th CIAO WorkshopOctober 20-22, 2008
Aneta SiemiginowskaModeling Fitting and Statistics41
Learn more on Sherpa Web Pages
Freeman, P., Doe, S., & Siemiginowska, A.\ 2001, SPIE 4477, 76Doe, S., et al. 2007, Astronomical Data Analysis Software and Systems XVI, 376, 543