IPAC 2MASS Working Group Meeting #103 Minutes

IPAC 2MASS Working Group Meeting #103 Minutes 9/10/96

Attendees: R. Beck, T. Evans, J. Fowler, L. Fullmer, T. Jarrett, G. Kopan, B. Light, C. Lonsdale, H. McCallon, S. Wheelock, J. White


  1. FDD Conformance
  2. NAMELIST File Delivery and Utilization
  4. Cumulative File Handling
  5. 2MAPPS 0.1 Status


  1. FDD Conformance -- In studying the 2MAPPS documentation (FDD, certain SDS's and SIS's), S. Wheelock discovered two inconsistencies between the FDD and other documents. Section 2.6 of the FDD states "...file formats representing position para- meters must be accurate to three decimal places when represented in seconds of time, two decimal places when represented in seconds of arc or camera pixels, and six decimal places when represented in floating-point degrees." Since it is clear that the intent is not accuracy but precision, this section will be reworded when the next version of the FDD is prepared, with "must provide precision of" replacing "must be accurate to". The main point is that some SIS's indicate position degree formats of F10.5, which provides a precision of only 0.036 arcseconds where 0.02 is required. Subsystem cognizant engineers are requested to check their SIS's and correct them in accordance with the six-decimal-place requirement where necessary, being sure that all users of the interface are aware of the changes.

    The other inconsistency involves the interpretation of subsystem exit codes. When a subsystem terminates, an exit code is passed to the calling script (EXEC, PCP, or a subsystem wrapper-script). The FORTRAN default is zero, and the value can be changed by using CALL EXIT(N) instead of STOP (other programming languages have similar control over the exit code), where N is set to the value desired for the exit code. When section 2.5 of the FDD was written, the intent was to allow for expansion of non-critical exit code meanings, and the convention was established that lower-numbered nonzero codes were the most severe. The PCP SDS and most of the subsystem coding has adopted higher-numbered codes as more critical. The less expensive resolu- tion of this conflict is to change the FDD, since the intent is to ensure uniformity rather than any particular convention. The cognizant engineers should employ the conventions stated in the PCP SDS, which are as follows (quoted from the SDS).


    Each subsystem will return to the PCP a completion code by executing an exit (code). This code will be written to the "log" file and can effect continued processing of the pipe. The return codes have the following meaning:

                Value   Effect
                  0     AOK       : SS Processing complete with no problems
                  2     Alert     : proceed nominally; note possible problem
                  4     Stop scan : Abort PCP but proceed with the next scan
                  6     Stop EXEC : Abort THIS scan and start no NEW ones
                  7     Abort EXEC: Abort ALL Scans ASAP
    Setting any status code other than 0 will force the "no Delete" PCP option at EOS.

    Possible non-zero return code examples:

          code 2 : Posman computes extracted src positions with large uncertainties
          code 4 : Rdframe found no PAR file for this scan.
          code 6 : the output disk is full.         

  2. NAMELIST File Delivery and Utilization -- R. Beck, who has taken over software delivery duties from the departing G. Laughlin, requested a clarification of NAMELIST file delivery require- ments with the following issues to be addressed:

    1. Must NAMELIST files be delivered with each delivery of executable software?

    2. Can NAMELIST file deliveries be made without delivering executable software at the same time?

    3. How will hemisphere dependence be handled?

    4. How will time dependence be handled?

    The last two issues are also involved in the question of how EXEC/PCP is supposed to get the right versions of NAMELIST files to the subsystems in the context of specific observation dates (ObsDates), scans, and hemi- spheres. Specifically regarding delivery by a subsystem cognizant engineer to the system, the following standards were adopted.

    1. There will be two dedicated hemisphere-specific subdirectories for all data, including NAMELIST files, above the subsystem level. For delivery purposes, cognizant engineers will place a NAMELIST file in each subdirectory, one for each hemisphere (identical if there is no hemisphere dependence; but see item D below) named /2massc/del/datan and /2massc/del/datas for the northern and southern hemispheres, respec- tively. To avoid naming conflicts, the NAMELIST files will be named nl., where is the name of the subsystem or module that will read that file; for example, BANDMERGE will read nl.bandmerge, and the POSMAN module PFPREP will read nl.pfprep. The names will be the same for both hemispheres, but the files will reside in the separate directories as stated above.

    2. NAMELIST files will be delivered as part of any software delivery; when only the NAMELIST files have changed, they may be delivered to the directories above without redelivering the rest of the software in its dedicated delivery directory (e.g., /2massc/del/src/pixcal).

    3. If there is a possibility of time dependence (e.g., the northern telescope had to be recollimated on 980123, and POSFRM's distortion model had to be recalculated), then the NAMELIST should contain para- meters indicating the valid date range for the other data, and the software should compare the ObsDate to this range before accepting the NAMELIST (or other data file) input. When multiple date ranges must be available, the different sets of data should be concatenated in a single file for EXEC/PCP to handle, and the input file should be reread as necessary to obtain a valid date range. Note that EXEC/PCP provides a special environment variable that can be used to override the standard setup of data directories for a subsystem; this may be useful for system- level testing or other purposes, so any cognizant engineer who feels a need for more flexible data setup should consult with R. Beck and J. White to learn how to use this option.

    4. The EXEC/PCP cognizant engineers request that every file existing in separate hemisphere-dependent locations contain either the text hemis='n' or hemis='s' (lower case, no embedded blanks) to tag its corresponding hemisphere, even if there is no other real hemisphere dependence. This is to allow certain scripts to grep on this text and perform standardized operations on files containing it. Note that such text may be placed in a table-file header; it may also be placed in NAMELIST files as part of the NAMELIST dictionary or as a comment between NAMELIST groups (e.g., after the &end and before the next input group, if any, begins with another &).

    5. If a subsystem requires script operations above the scan level, the cognizant engineer may arrange with EXEC/PCP to include those opera- tions in the system level initialization or termination scripts, but not via a general-purpose calling of subsystem scripts; the system- level scripts must contain the code and be maintained at the system level. For example, the system-level initialization script will handle copying the NAMELIST (and other) files for the appropriate hemisphere to the appropriate scan directories for the subsystems to use; similar sorts of operations for subsystem-peculiar requirements can be included via understandings worked out with the EXEC/PCP cognizant engineers.

    These conventions will be put into effect during the 2MAPPS 0.1 delivery exercise, not before, since two subsystems have already been delivered prior to the adoption of these conventions. Many changes will be found necessary during the tests about to start, and conforming to these standards can be included among them as work progresses. It should also be noted that the descriptions given above are hereby being presented to the entire team for the first time in such specific terms and reflect the author's interpretation of the working group discussion; if errors are found, a full corrected state- ment of all of the above aspects will be included in future meeting minutes. Team members are requested to make known any problems they may see in these conventions as described above.

  3. QUALITY Q&A -- L. Fullmer answered questions regarding how subsystems interact with the QUALITY subsystem, such as what directories files should be written in and what information is needed for QUALITY to make use of the information. The answer to the first question is that the directory structure will include a scan subdirectory named qual, i.e., just as the northern-hemisphere scan number 123 for the observation date 980204 will involve a directory named 980204n/s123/flat for flattened-frame data, it will involve a directory named 980204n/s123/qual to which data intended as input to the QUALITY subsystem should be written. In addition, there will be a higher-level directory for the night, 980204n/datequal, where the QUALITY subsystem itself will operate.

    It was pointed out that the best way for a subsystem to get its QUALITY output to the right directory is to write it to the current working directory and then have the subsystem wrapper script move it to the proper directory. This will be possible via simple relative-address directory specification (e.g., "../qual") or via environment variables provided by EXEC/PCP.

    As far as how to interpret its input, QUALITY will rely on the expertise of the subsystem engineers to provide thresholds for defining out-of-limits conditions, warning levels, guidance on what sort of graphical representa- tions would be useful for judging quality, etc. It is recognized that such expertise is just now being accumulated, and that considerable work with survey shakedown data will be needed before the vast range of possibilities can be honed down to the truly useful diagnostics. Thus this exercise of developing quality assessment tools will be an ongoing activity for which the primary individuals in charge will be R. Cutri, L. Fullmer, and J. Mazzarella.

  4. Cumulative File Handling -- Several (if not most) subsystems need to have scan-independent cumula- tive data files, and the question of how to handle this need arose. For example, the CALMON subsystem needs a cumulative data base for each hemi- sphere in which calibration data can be accumulated over time, thereby reducing parameter uncertainties as more information is gathered. A similar need exists for the POSMAN and BANDMERGE subsystems.

    Because of the asynchronous parallel processing of many scans, one cannot assume that a subsystem can simply open a cumulative file and write to it, because that file could be in use by another executing copy of the same subsystem. File-locking techniques exist, but all simple methods tried so far have been found to have certain quirks that would be best avoided. For this reason a different approach is to be taken: updates to cumulative files should be written to the current working directory, and then the EXEC subsystem will run a termination script that cleans up the entire run, including moving/appending these updates to their cumulative directories- /files. This appears to be workable for all the cases where a need is currently recognized, because only ASCII-file updates appear to be needed. Cognizant engineers who need this service should schedule a discussion with J. White and R. Beck to arrange for the proper script coding to be included.

  5. 2MAPPS 0.1 Status -- Software deliveries for the version 0.1 2MAPPS integration are on sched- ule for September 18. A plan for bridging certain gaps has been formulated. The key gaps are: DARKS, POSFRM, and POSPTS. A program that can strip out FITS files from the scan data has been written and will be used to set up twilight flat and dark frames for hand analysis by R. Cutri to produce respon- sivity images, masks, and darks; this replaces the DARKS subsystem. The POSMAN program PFPREP will be enhanced in several ways, the relevant one here being to expand the content of its erstwhile POS09 file to match the format and content of the POS01 file but with preliminary estimates for all quantities (it already contained such estimates for the frame offsets and several other parameters in common with POS01). The key element is the connection to J2000 coordinates, which at this early stage is made exclusively via the a priori position information passed from the observatory input. Since no associations with astrometric catalogs will be made until the real POSFRM comes online in a few months, tying extractions to external astrometric information will not be possible in version 0.1, nor will attenuation of the random walk in the frame-to-frame offset solutions, which will themselves be only the preliminary J-band-only estimates (since the first processing will be with replicated, K-band data, the limitation to single-band information will not be a real degradation at this stage).

    A POSFRM stand-in program denoted as version -0.9 will use the PFPREP POS01-like output and the FREXAS extractions to create a file in the format of POS02 that should be suitable for testing MAPCOR, and the POS01 substitute should suffice for BANDMERGE and POSPTS version -0.8, where the latter is another stand-in program that will provide the POSPTS function of converting the BANDMERGE output point-source list to J2000; the associations field of each record, however, will indicate that no association has been made, since no catalog or capability to use it has been set up yet.

    The output of 2MAPPS 0.1 should resemble the real output in format and general content, but with position and photometric accuracies not meeting the project requirements, although not so erroneous as to be unrecognizable. As stated in the minutes of previous meetings, many functions will not be per- formed, and associated files will not be generated. For more information on these limitations, see the minutes to meeting number 101, 8/20/96.