Measuring a huge amount of images

[PetriKehusmaa]

Hi Group,

I'm having a situation where I have 20 000 images of same exoplanet target taken during 2 years time period.

Intention is to make an extensive analysis of the system. Initial measurement could be done with AIJ and further analysis (numerical data) with other programs like IDL etc.

What would be the best practise to measure, normalize and detrend all images with AIJ? This data is including all phases of the system, not only transits and secondary ecplises.

Approx. 15 000 images are taken with same filter and exposure time. All images are calibrated.

Even thinking of measuring every night separately is quite demotivating :-)

Cheers,
Petri

[karenacollins]

Hi Petri,

I am working with a team publishing a large number of observations of the same star. I think the best answer for you depends on your telescopes pointing precision, whether the images are plate solved, how stable the atmospheric transparency is, and consistency of focus, seeing, comp star availability, and comp star longterm brightness.

If you have the same set of comp stars available for all observations, and the same aperture size will work reasonably well for all nights, you should be able to run photometry on all images in one run. If the pointing is off from night-to-night, you can plate solve the first image of the night and select "use RA/Dec to place apertures" to ensure the apertures track any large night-to-night jumps.

However, I think you may find that the comp stars may have long term brightness trends that may cause the night-to-night observations to not be directly comparable, unless you normalize each night independently.

If you do the photometry in one run, and then chop the measurements table on night boundaries and copy the header line to each file, you can then load them into AIJ as if they were done separately. Then normalization and detrending of each night could be done separately in AIJ.

If you are willing, the normalization and detrending could be done with custom code in IDL, python, etc., but you'd have to evaluate which would be most time efficient, since I don't the number of different nights of observations. If you have 20 nights of 1000 images, my bet is that the custom coding might not be worth the effort, but if you have 200 nights, of 100 images, then it might be worth considering custom detrend and normalization code.

We had 300 nights of light curves, and still used AIJ to so that we could review the quality of each night of observations before including it our analysis.

Karen

[PetriKehusmaa]

Hi Karen,

Thank you for your reply. I'll try be more specific this time :-)

Telescope pointing precision is good, pointing error is mainly +/- 1 arcmin. All images are plate solved and calibrated.

Then the hard part.

Atmospheric transparency is pretty stable. Our observatory is at 2500 meters above sea level in Atacama Chile. But of course there are some fluctuation because these images have been taken during two years time period.

The telescope is keeping its focus pretty well even if temperature is changing during the night. Focus routine is done by FocusMax for the filter been used for imaging so no filter offsets are being used. Focus run is done every time before starting the time series.

I'm currently using 8 comp stars which have approx. the same brightness as target. All of these stars are available for the whole stack of images. One of the problem seem to be longterm brightness and its fluctuation. Also meridian flips are causing these fluctuations.

--"If you do the photometry in one run, and then chop the measurements table on night boundaries and copy the header line to each file, you can then load them into AIJ as if they were done separately. Then normalization and detrending of each night could be done separately in AIJ."

How do I do normalization and detrending afterwards? And should I do the one run photometry without detrending and normalization (so 'Fit Mode' and 'Norm/Mag Ref' off) for all stars (target and comp stars)?

And then I need to chop individual nights off from the big file and for each night I need to detrend meridian flips and for transits/eclipses choose fit mode and normalization regions?

For target I have used AIRMASS, tot_C_cnts, Width_T1, BJD_TDB and Meridian_Flip detrend parameters. For comps stars I have used AIRMASS and BJD_TDB.

Is this about right? And after detrending/normalizing all individual nights separately, I should then have T1 data normalized through out the data sets and can then rely on those numbers to do further analysis of the whole data?

Even though I have a pretty stable imaging system, the end result still seem to suffer some systematics which I don't seem to be able to even out. I've been playing with aperture, number of comp stars and detrending parameters without success.

I have data from about 200 nights, so chopping data into individual files for each night is a mighty task.

Petri

[karenacollins]

Hi Petri,

The detrend parameters are applied separate from the photometry, so those setting are not critical during the photometry run. In fact, I would turn them all off so that the photometry will run faster. In fact, if you run the entire set at once, you should set the multi-aperture option to not update the table and plot while photometry is running, because the table gets huge and slows down significantly the longer it gets, if the table has to be updated after each photometric measurement. With that option off, it will update one time at the end of the run. Still you may be better off to break the images into ~1000 image chunks.

Now that I see you have meridian flips, the problem is even harder. There is a "meridian flip" detrend parameter, as you mentioned, but that will realign the baseline at only one flip point per light curve. As I mentioned, in our project of 300 light curves, we reduced each one separately, and honestly that is what I would do with your dataset too. Since all of your fields are nearly identical, all of the desired settings will be retained from one time-series to the next (including aperture positions), so I think all of your data could be processed relatively quickly (although I realize it would be a repetitive boring task). That way you can keep a quality check on each night of data, plus correct for the meridian flip offsets in the baseline. Also, you wouldn't have to manually split the table.

Explaining how to set the fit and detrend options is fairly involved. I'd suggest starting with the tutorial offer by Dennis Conti here (if you haven't already):
http://www.astrodennis.com/Guide.pdf
at his website here
http://www.astrodennis.com

Just skip to the relevant sections of the document. If you have specific questions that are not answered there, I'd be happy to help with any remaining areas of concern.

In general, for a light curve with a transit, you would want to choose the transit fit mode option  (the all green icon with a red line that looks like a transit) and normalize using the two OOT chunks of data using that normalize option (the normalize icon with green, white, green). For an OOT light curve, fit using the full green mode with the flat red line and normalize with the all green option. Note that the fitted/detrended/normalized data are just plotted by default. If you want to same them, you have to use the button on the left hand side of the plot options to add the data to the measurements table. Then save the measurements table. I suggest the "save all" option in the Multi-Plot file menu. Dennis' document covers the details of all of that.

Karen

On 1/23/2018 6:18 AM, PetriKehusmaa [via AstroImageJ] wrote:

Hi Karen,

Thank you for your reply.

Telescope pointing precision is good, pointing error is mainly +/- 1 arcmin. All images are plate solved and calibrated.

Then the hard part.

Atmospheric transparency is pretty stable. Our observatory is at 2500 meters above sea level in Atacama Chile. But of course there are some fluctuation because these images have been taken during two years time period.

The telescope is keeping its focus pretty well even if temperature is changing during the night. Focus routine is done by FocusMax for the filter been used for imaging so no filter offsets are being used. Focus run is done every time before starting the time series.

I'm currently using 8 comp stars which have approx. the same brightness as target. All of these stars are available for the whole stack of images. One of the problem seem to be longterm brightness and its fluctuation. Also meridian flips are causing these fluctuations.

--"If you do the photometry in one run, and then chop the measurements table on night boundaries and copy the header line to each file, you can then load them into AIJ as if they were done separately. Then normalization and detrending of each night could be done separately in AIJ."

How do I do normalization and detrending afterwards? And should I do the one run photometry without detrending and normalization (so 'Fit Mode' and 'Norm/Mag Ref' off) for all stars (target and comp stars)?

And then I need to chop individual nights off from the big file and for each night I need to detrend meridian flips and for transits/eclipses choose fit mode and normalization regions?

For target I have used AIRMASS, tot_C_cnts, Width_T1, BJD_TDB and Meridian_Flip detrend parameters. For comps stars I have used AIRMASS and BJD_TDB.

Is this about right?

I have data from about 200 nights, so chopping data into individual files for each night is a mighty task.

Petri

---

If you reply to this email, your message will be added to the discussion below:

http://astroimagej.1065399.n5.nabble.com/Measuring-a-huge-amount-of-images-tp959p961.html

To start a new topic under AstroImageJ, email [hidden email]
To unsubscribe from AstroImageJ, click here.
NAML