Reducing data failing

[Peter]

Hi,

The CCD Data Processor is failing to produce the master bias & flat.

It appears to stop loading after a few images (on windows, 64 bit java).

I notice on Task manager that it is taking a large proportion of my memory but isn't using any of my CPU at this point.

Has anyone experienced similar problems?  

(Also, if I'm allowed to ask here, is this the best software to bulk reduce data on windows?)

Thank you in advance,

Peter

****
EDIT: the files were of different resolution (I had 100s of images, with a few being different)! Maybe a error message should be in place.
****

[karenacollins]

Hi Peter,

How much memory have you made available to AstroImageJ? The default is
only 640 KB and the process may be running out of memory. Assuming you
are new to AIJ, the memory setting is at
AIJ_Toolbar->Edit->Options->Memory and Threads. You can set the value as
high as 60-75% of your total system memory. Restart AIJ after changing
the setting and the new memory will be available.

Click on the lower part of the toolbar after restarting AIJ to verify
the total memory available. Also, make sure it says that 64-bit Java is
being used. If not, reinstall the option from the website that includes
64-bit java for windows.

If neither of these are your issue, I need to know more since this issue
hasn't been previously reported. You could start with letting me know
the total amount of memory allocated to AIJ, and the number of
calibration images being combined and the file size of each. Also, click
on the lower part of the toolbar and let me know the percent memory
being used. A screen shot of the DP might also be useful.

I'll let others comment on what the best software is since I am highly
biased!

Karen

[Peter]

Hi,

I figured out the problem! Some of the files were of 4k resolution while the rest were in 2k, so the individual bias and flats where conflicted in dimensions.


While you're here, is there a way to bulk crop the processed images? Perhaps using a post processing macro? Also, is the CCD Data Processor available in command line?
(I have about 60GB of data to get through!)


Thanks!

Peter

[karenacollins]

Hi Peter,

The underlying ImageJ has a cropping capability that should do what you
need. That can probably be implemented in a macro as well. Search the
ImageJ documentation for cropping instructions. AIJ itself does not have
a separate cropping feature as part of DP, but if you write a macro, you
could add that as a post macro to run after each image reduction. If you
do write a cropping macro, others might find it useful too if you post
the macro code on the user forum.

Unfortunately, DP can not be ran in command line mode. But, I will note
that you can drag and drop files from the OS into the file name/pattern
boxes to save typing.

Karen

[Peter]

My cropping feature is rather simple for my case:
//setTool("rectangle");
makeRectangle(185, 71, 1674, 957);
run("Crop");

But, the RA/DEC coordinates aren't correctly adjusted as a result, which I need.
Do you have any recommendations on how to rectify this?

(Also, which are the best settings do you think to use for removing Cosmic rays? My images come out quite blurry using radius 2, threshold 10%.)

Thank you!

Peter

[karenacollins]

Hi Peter,

As you have discovered, cropping unfortunately does not adjust the WCS
coordinates to match the new image. The only straight forward work
around at this time is to plate solve the image again. AIJ's plate solve
will remove the WCS solution and update the headers with a new solution.
However, this process could be slow since I think you are processing
lots of images.

Here is another possibility: If you can find a way to adjust data in the
image's "info" (see Toobar->Image->Show Info...), the values that need
to be corrected are CRPIX1 and CRPIX2. The image "info" contains the
FITS header information for FITS images. You would need to adjust CRPIX1
by the number of pixels you crop from the left side of the image, and
CRPIX2 would need to be adjusted by the number of pixels you crop from
the bottom of the image (in FITS coordinate space). Maybe this post in
the ImageJ forum is useful:
http://imagej.1557.x6.nabble.com/add-edit-a-comment-and-store-it-with-an-image-td3685672.html

Since a crop with WCS correction would be generally useful, I have added
it to the feature request list. However, it will probably be a month or
two before I can find time to publish a new version of AIJ the supports
a WCS aware cropping feature.

Regarding cosmic ray removal, are you using the "Remove outliers"
feature on the Data Processor (DP) panel, or something else? I wouldn't
expect the "Remove outliers" feature to blur the image. It should only
replace a pixel with the median value of the surrounding pixels if its
actual value differs from the calculated median by more than the
"Threshold" setting (in ADU). If you are using that feature and entering
"10" as the threshold, it is likely that most or all of the pixels in
the image are being replaced with the neighboring pixel's median value,
resulting in the blurring effect. In that case, I would recommend
experimenting with threshold values in the range of 100-1000 ADU to see
if cosmic rays are corrected without replacing any of the other pixels.
This feature also exists in the AIJ toolbar at Process->Noise->Remove
Outliers if you want to include it to the cropping routine you mentioned.

Hopefully some of these ideas will be helpful,
Karen

[Peter]

Hi Karen,

Thank you for your detailed reply! I'm currently an undergraduate doing an internship with ESO, hence all the (maybe, less intelligent) questions.

I managed to manually adjust CRPIX1/2 to provide the correct the DEC/RA values, but my attempts to replace the values in a macro doesn't seem to work... (I attempted using .bat files, but it's pretty horrid)

The forum you link me to doesn't explain that clearly how to adapt an existing header, in particular in FITS format.  Might you have a better idea?


Thank you so much, I am truly grateful.

Peter

[karenacollins]

Hi Peter,

These questions are all good. This is a feature that AIJ should support.
It is probably too late for your work, but I'll do my best to get this
implemented in AIJ in a few weeks.

Unfortunately that post was the closest I could find related to what you
need to do. Have you used Python? If so, John Kielkopf offers a set of
python compatible routines here:
http://www.astro.louisville.edu/software/alsvid/index.html

There is a program included in Alsvid (that runs in the Python
environment)  which allows you to edit a FITS header value
(fits_edit_head.py). You could read the header first with
fits_list_head.py. Using those functions, I think a few lines of python
code could update the headers for you after the AIJ crop. Unfortunately,
you would need to run those commands outside AIJ and then reload the
updated files for further processing in AIJ.

You may be able to do the cropping in python as well. The PyFITS module
also offers many FITS file manipulation capabilities, but I'm not sure
about cropping.
http://www.stsci.edu/institute/software_hardware/pyfits

I wish I could offer an easier AIJ solution in the near term.
Karen