Re: Chi squared statistic and Error measurements
Posted by
karenacollins on
Jun 28, 2020; 8:49am
URL: http://astroimagej.170.s1.nabble.com/Chi-squared-statistic-and-Error-measurements-tp1379p1380.html
HI Bradley,
I just reviewed your result. I think the scatter in the light curve
makes chi2 almost irrelevant here since the transit is very shallow
and probably undetectable from the ground at 0.37 ppt.
I'm not sure why your error bars are so small. I assume you've made
sure your gain, dark current, and read noise are set properly on the
Aperture Settings page. If there isn't a problem there, it could
just be that although you have plenty of photons from the V=10 star,
maybe you are trying to drive below the systematics limit of the
system.
From a fitting perspective, AIJ does a simple best fit based on your
starting parameters, which are largely set by the placement of the
"left" and "right" markers. If you move those around, you may see
the transit model fit to different shallow transits aross the light
curve. MCMC analysis is more robust, but is not implemented in AIJ
because run times can be long.
The formal error calculation is described in the AIJ paper on arXiv.
Degrees of freedom are 7 for the transit model, plus 1 for each
detrend parameter includes, minus 1 for each locked transit model
parameter minus 1.
Karen
On 6/28/2020 12:03 AM, bswalter [via
AstroImageJ] wrote:
First my two specific questions about the reduced Chi-squared
statistic (Chi2r)
1. How are the σ_i calculated.
2. How are the degrees of freedom determined; specifically what
are the calculated parameters/dependencies that are subtracted
from the number of images to arrive at dof
What prompted my questions about Chi2r was a conclusion I reached
analyzing TFOP-SG1 observations of TIC180695581.01 made on
2020-06-25. 271 x 15 sec EXPTIME observations were made with the
0.61m telescope at PJMO through through a Bessel I filter (not Ic)
using a Princeton Instruments Pixis 2048 eXcelon camera. Plate
scale is 0.5072 arcsec/pixel. Gain, read noise and dark current
were correctly entered into the multi-aperture tool using values
measured for this specific camera, operating temperature and
personality settings.
A transit seemed to be clearly identified but the Ch2r value for
the fit at any bin size from 1 to 9 was large. Lowest Ch2r
occurred with 9 observations per bin but Ch12r was still
approximately 5.17 and with dof = 25 that gave probability
smaller than 0.001 of getting that Ch2r value or higher if the
model and data represent the same event. The error bars seemed
much too small for the scatter in the data. It has been my
experience that analytically calculated error values almost always
significantly to drastically underestimate errors depending on how
many sources of error have been properly included in the
analytical error calculation.
I suspected underestimation of error was the cause of a large Ch2r
and the average of single observation errors calculated for the
target star as the standard deviations of the 9 observations per
bin is approximately 2.7 times the error value given by AIJ. The
average of the AIJ calculate values for the target is 1.443 ppt.
Standard deviations of the of the 9 observation bins average 3.88
ppt, 2.718 times the AIJ value for the dn LC. The average
empirically calculated standard error of bin means is 2.549 times
the average of the dfn bin means. If observation sigmas are equal,
Chi2r will scale by the inverse of the square of this ratio which
would make Chi2r 0.79. They are reasonably closely grouped but
certainly not equal. I would have to calculate Chi2r in a
spreadsheet to get the exact value, but it will be much closer to
1.0 the 5.17 value calculated by AIJ. I had not read the thread
mentioned above when I did my initial comparison of single
observation errors and was encouraged to learn that the ratio of
my empirically determined errors to AIJ errors was close to the 3X
ratio that Luca found with his detailed comparison of analytical
error calculations.
Another thing that became clear was that the ratio of empirically
determined error to analytically determined error increases with
brightness of the star in a roughly linear way. An Excel
spreadsheet containing the measurements table for the run is
attached. The averages of empirically determined standard
deviations of obs in 9-image bins and error values of the bin
means calculated from AIJ single observation errors have been
added as have observing run averages for these various error
calculations. Also attached is a spreadsheet That compares the
various error calculations and analyzes the relationship between
star brightness and the ratio of empirical error to AIJ analytical
error.
If my analysis is out of whack in some way, I would appreciate
corrections.
Brad Walter.
TIC180695581-01_20200625_PJMO_I_bin1-Lightcurve-.pngTIC180695581-01_20200625_PJMO_I_bin9-Lightcurve.pngTIC180695581-01_20200625_PJMO_I_bin9-fitpanel02_rel_flux_T1.pngTIC180695581-01_20200625_PJMO_I_bin9-Measurements.xlsxTIC180695581.xlsx
Brad Walter