Data Subset Option request

[mattwood]

First, thanks for all the work you've all done to make AIJ such a useful tool.

For my data analysis for CBA data, I like to have text files in the format:

  HJD  Mag  Mag_Err  Airmass  JD_UTC  OBSCODE  Star_Name

where OBSCODE is the AAVSO observer code, and Star_Name is for example "AM CVn"

It would be very helpful if AIJ could have option in "Save Data Subset to File" for text string entries, e.g., "Comment 1", "Comment 2", etc.  I only need 2, but it might be useful to have 5.

I have no idea how difficult this would be to implement, and I don't know Java so can't volunteer to take it on myself.  

I can add these columns with a simple Perl or Python script after the fact, but it would be a time saver if this was an option.

Cheers,
Matt Wood

[mattwood]

Followup issue: When saving subset, my HJD_MOBS and JD_UTC columns are blank (with the second heading coming up as FJD_UTC).

I suppose I can write a Python/pandas script to read in the measurements.xls file and output what I need.

Cheers,
Matt

[karenacollins]

Hi Matt,

We can look at this, but I want to make sure I understand what you need. For the example you provided, is it correct that you want to be able to add the constant value of your AAVSO observer code in all rows under a column labelled "OBSCODE" and "AM CVn" for all rows of a new column labeled "Star_Name"? That would be much easier to handle than giving the user the capability to place unique values in each row of the table.

Karen

[karenacollins]

Hi Matt,

We'll have to think of a way around this. For most of our work, when a time column with JD in the name is not in the left-most column, we assume it is detrend parameter, so we prepend the name with F for fractional and remove the constant integer part. Maybe we'll add an option to allow that functionality to be disabled.

Are your HJD_MOBS and JD_UTC columns integer values? If not, I'm not sure why they would end up being blank. Could you send me an example measurements table and the resulting data subset table to my email on the main page?

Karen

[mattwood]

Yes, I was hoping for a feature that would allow the same text entry on every line, as you describe.  It's not a big deal if it's difficult to put out two different times in the subset file - I can spin up a translator program that will read the Measurements file and output in the format I use.  

One issue I found while trying to output to AAVSO format is that EXPTIME doesn't seem to be in the Measurements file as a column, and the interface that Dennis Conti wrote isn't asking for it as his instructions indicate.  

Again - none of these are big issues, so if you all want to 'punt' that's quite understandable.  If you have this as a GitHub project I could try to fork it and see if I can work up an option that would output the format I use, but not knowing Java the timescale for this is likely long (to never).

I sent the files you requested.

Thanks so much,
Matt

[karenacollins]

EXPTIME can be read from the fits headers and placed in the measurements table by specifying the header name in the Aperture Photometry Settings panel. To extract the values from the header, make sure the option "List the following FITS keyword decimal values in measurements table" is enabled and also make sure EXPTIME is part of the comma separated list in the "Keywords" box. That is the measurements table part.

If EXPTIME is needed in the AAVSO formatting, but not asked for, I'm sure Dennis will want to look into it.

Karen

[karenacollins]

Thank you for sending the measurements table and resulting data subset file. From those files I can tell that only the HJD_MOBS column is blank. The FJD_UTC column is there in place of the JD_UTC column since it is not the left-most column, and it contains the fractional part of the JD_UTC values in the output table. HJD_MOBS values are blank because the values are all NaNs in the measurements table. Did you extract HJD_MOBS from the headers or calculate it using the "Add Data" module of Multi-Plot? If so, something possibly went wrong in the extraction. If you can't track down why the values are NaNs in the table, let us know. Once tracked down, I think the HJD_MOBS values should be written out correctly. Of course handling the JD_UTC values correctly in a non-leftmost column will require a new option setting. We'll look at adding that capability, but it may take a bit of time for it to bubble up in priority.

Karen

[mattwood]

Thanks so much, Karen!  End of semester crush this week, but hoping to get this all straightened out next week.

[karenacollins]

Hi Matt, both of these issues have been addressed in v5.5.1.00 and later. The full time data are now output in ay data column selected. New entries in the label selection dropdown menus have been added with the 'F' prefix to all users to select the fractional (or integer selected version). We have also added the capability to output user defined constant column labels and values. The related release notes are:

--Multi-Aperture > File > ‘Save data subset to file’ now supports outputting full JD-related data in all data columns. Previously, outputting any time-related column (containing JD in the label), other than in the first (leftmost) column would subtract the integer value for purposes of detrending and avoiding fitting the large JD numbers. To now output the fractional version of the JD number, select the version of the time column label in the dropdown mention that is prefixed with ‘F’ (for fractional). For example, to output the decimal part of BJD_UTC, select FBJD_UTC in the dropdown menu.

-- Multi-Aperture > File > ‘Save data subset to file’ now allows the user to define custom constant column values and column label names for output in the data subset file. For example, to output a column labelled ‘Target’ with the value ‘Sirius’, open ‘Save data subset to file’, select the ‘Constant Col’ button at the bottom and define the label, value pairs that you desire to output. Click Okay to return to the Save Data Subset main panel. Then select the desired label names in the dropdown menus in the order that you want them in the output file.

Karen