Chi squared statistic and Error measurements

[bswalter]

First my two specific questions about the reduced Chi-squared statistic (Chi2r)
1. How are the σ_i calculated.  
2. How are the degrees of freedom determined; specifically what are the calculated parameters/dependencies that are subtracted from the number of images to arrive at dof
What prompted my questions about Chi2r was a conclusion I reached analyzing TFOP-SG1 observations of TIC180695581.01 made on 2020-06-25. 271 x 15 sec EXPTIME observations were made with the 0.61m telescope at PJMO through through a Bessel I filter (not Ic) using a Princeton Instruments Pixis 2048 eXcelon camera. Plate scale is 0.5072 arcsec/pixel. Gain, read noise and dark current were correctly entered into the multi-aperture tool  using values measured for this specific camera, operating temperature and personality settings.  
A transit seemed to be clearly identified but the Ch2r value for the fit at any bin size from 1 to 9 was large. Lowest Ch2r occurred with 9 observations per bin but Ch12r was still approximately 5.17 and with dof = 25 that gave  probability  smaller than 0.001 of getting that Ch2r value or higher if the model and data represent the same event. The error bars seemed much too small for the scatter in the data. It has been my experience that analytically calculated error values almost always significantly to drastically underestimate errors depending on how many sources of error have been properly included in the analytical error calculation.  
I suspected underestimation of error was the cause of a large Ch2r and the average of single observation errors calculated for the target star as the standard deviations of the 9 observations per bin is approximately 2.7 times the error value given by AIJ. The average of the AIJ calculate values for the target is 1.443 ppt. Standard deviations of the of the 9 observation bins average 3.88 ppt, 2.718 times the AIJ value for the dn LC. The average empirically calculated standard error of bin means is  2.549 times the average of the dfn bin means. If observation sigmas are equal,  Chi2r will scale by the inverse of the square of this ratio which would make Chi2r 0.79. They are reasonably closely grouped but certainly not equal.  I would have to calculate Chi2r in a spreadsheet to get the exact value, but it will be much closer to 1.0 the 5.17 value calculated by AIJ.   I had not read the thread mentioned above when I did my initial comparison of single observation errors and was encouraged to learn that the ratio of my empirically determined errors to AIJ errors was close to the 3X ratio that Luca found with his detailed comparison of analytical error calculations.
Another thing that became clear was that the ratio of empirically determined error to analytically determined error increases with brightness of the star in a roughly linear way. An Excel spreadsheet containing the measurements table for the run is attached. The averages of  empirically determined standard deviations of obs in 9-image bins  and error values of the bin means calculated from  AIJ single observation errors  have been added as have observing run averages for these various error calculations. Also attached is a spreadsheet That compares the various error calculations and analyzes the relationship between star brightness and the ratio of empirical error to AIJ analytical error.
If my analysis is out of whack in some way, I would appreciate corrections.
Brad Walter.

TIC180695581-01_20200625_PJMO_I_bin1-Lightcurve-.pngTIC180695581-01_20200625_PJMO_I_bin9-Lightcurve.pngTIC180695581-01_20200625_PJMO_I_bin9-fitpanel02_rel_flux_T1.pngTIC180695581-01_20200625_PJMO_I_bin9-Measurements.xlsxTIC180695581.xlsx

[karenacollins]

HI Bradley,

I just reviewed your result. I think the scatter in the light curve makes chi2 almost irrelevant here since the transit is very shallow and probably undetectable from the ground at 0.37 ppt. 

I'm not sure why your error bars are so small. I assume you've made sure your gain, dark current, and read noise are set properly on the Aperture Settings page. If there isn't a problem there, it could just be that although you have plenty of photons from the V=10 star, maybe you are trying to drive below the systematics limit of the system.

From a fitting perspective, AIJ does a simple best fit based on your starting parameters, which are largely set by the placement of the "left" and "right" markers. If you move those around, you may see the transit model fit to different shallow transits aross the light curve. MCMC analysis is more robust, but is not implemented in AIJ because run times can be long.

The formal error calculation is described in the AIJ paper on arXiv. Degrees of freedom are 7 for the transit model, plus 1 for each detrend parameter includes, minus 1 for each locked transit model parameter minus 1.

Karen

On 6/28/2020 12:03 AM, bswalter [via AstroImageJ] wrote:

First my two specific questions about the reduced Chi-squared statistic (Chi2r)
1. How are the σ_i calculated.  
2. How are the degrees of freedom determined; specifically what are the calculated parameters/dependencies that are subtracted from the number of images to arrive at dof
What prompted my questions about Chi2r was a conclusion I reached analyzing TFOP-SG1 observations of TIC180695581.01 made on 2020-06-25. 271 x 15 sec EXPTIME observations were made with the 0.61m telescope at PJMO through through a Bessel I filter (not Ic) using a Princeton Instruments Pixis 2048 eXcelon camera. Plate scale is 0.5072 arcsec/pixel. Gain, read noise and dark current were correctly entered into the multi-aperture tool  using values measured for this specific camera, operating temperature and personality settings.  
A transit seemed to be clearly identified but the Ch2r value for the fit at any bin size from 1 to 9 was large. Lowest Ch2r occurred with 9 observations per bin but Ch12r was still approximately 5.17 and with dof = 25 that gave  probability  smaller than 0.001 of getting that Ch2r value or higher if the model and data represent the same event. The error bars seemed much too small for the scatter in the data. It has been my experience that analytically calculated error values almost always significantly to drastically underestimate errors depending on how many sources of error have been properly included in the analytical error calculation.  
I suspected underestimation of error was the cause of a large Ch2r and the average of single observation errors calculated for the target star as the standard deviations of the 9 observations per bin is approximately 2.7 times the error value given by AIJ. The average of the AIJ calculate values for the target is 1.443 ppt. Standard deviations of the of the 9 observation bins average 3.88 ppt, 2.718 times the AIJ value for the dn LC. The average empirically calculated standard error of bin means is  2.549 times the average of the dfn bin means. If observation sigmas are equal,  Chi2r will scale by the inverse of the square of this ratio which would make Chi2r 0.79. They are reasonably closely grouped but certainly not equal.  I would have to calculate Chi2r in a spreadsheet to get the exact value, but it will be much closer to 1.0 the 5.17 value calculated by AIJ.   I had not read the thread mentioned above when I did my initial comparison of single observation errors and was encouraged to learn that the ratio of my empirically determined errors to AIJ errors was close to the 3X ratio that Luca found with his detailed comparison of analytical error calculations.
Another thing that became clear was that the ratio of empirically determined error to analytically determined error increases with brightness of the star in a roughly linear way. An Excel spreadsheet containing the measurements table for the run is attached. The averages of  empirically determined standard deviations of obs in 9-image bins  and error values of the bin means calculated from  AIJ single observation errors  have been added as have observing run averages for these various error calculations. Also attached is a spreadsheet That compares the various error calculations and analyzes the relationship between star brightness and the ratio of empirical error to AIJ analytical error.
If my analysis is out of whack in some way, I would appreciate corrections.
Brad Walter.

TIC180695581-01_20200625_PJMO_I_bin1-Lightcurve-.pngTIC180695581-01_20200625_PJMO_I_bin9-Lightcurve.pngTIC180695581-01_20200625_PJMO_I_bin9-fitpanel02_rel_flux_T1.pngTIC180695581-01_20200625_PJMO_I_bin9-Measurements.xlsxTIC180695581.xlsx

---

If you reply to this email, your message will be added to the discussion below:

http://astroimagej.1065399.n5.nabble.com/Chi-squared-statistic-and-Error-measurements-tp1379.html

To start a new topic under AstroImageJ, email [hidden email]
To unsubscribe from AstroImageJ, click here.
NAML

[bswalter]

Karen,
Thanks for the response. I expected to get nothing but noise from this run. From previous transit LCs with this telescope I judged the limit for td detection to be somewhere between 1 and 2 ppt with optimum sky conditions. Looking at the data I expected the model to give me a flat line fit. I was surprised to get a fit so close to the predicted transit values when I set the ingress and egress markers at the predicted values. I tried other marker locations, and the model "flat lined" if I moved them very much including to the ingress and egress locations of the model in the initial fit.  I left ingress and egress markers at the predicted locations simply because the data gave no clue where they should be.

Gain, dark current and read noise were set correctly the aperture settings page. I read the AIJ paper a a couple of years ago and forgot appendix B. I have found that measurement errors determined empirically from standard deviations of binned objects and standard errors of their means compared to the corresponding values determined from the CCD error equation are often two to three times as large. Aside from the scintillation effects Luca mentioned, our skies are not photometric. There are constantly changing inhomogeneities within the field of view. Finally, many stars that are considered non-variable have low level variability at the fractional ppt level and we can't separate variability at that level from noise. In retrospect I think that may be true of the C4 star I chose. It has more empirically determined noise than it should given its average source-sky counts compared to the target and other comps.

Thanks again for the helpful response

Brad Walter