7.5.1: DEIMOS primary image storage format

Crudely, the DEIMOS CCD detector array looks like this
	<1-----2><1-----2><1-----2><1-----2>
	|   !   ||   !   ||   !   ||   !   |    North
	|       ||       ||       ||       |    ^
	|   !   ||   !   ||   !   ||   !   |    |   handedness of sky
	| CCD 1 || CCD 2 || CCD 3 || CCD 4 |    |
	|   !   ||   !   ||   !   ||   !   |    +-----> East
	|       ||       ||       ||       |
	|   !   ||   !   ||   !   ||   !   |
	+=======++=======++=======++=======+
	|   !   ||   !   ||   !   ||   !   |
	|       ||       ||       ||       |
	|   !   ||   !   ||   !   ||   !   |    +-----> +X direction
	| CCD 5 || CCD 6 || CCD 7 || CCD 8 |    |
	|   !   ||   !   ||   !   ||   !   |    |   Focal plane coords
	|       ||       ||       ||       |    v
	|   !   ||   !   ||   !   ||   !   |     +Y direction
	<2-----1><2-----1><2-----1><2-----1>
(The orientation of X and Y here is chosen for conformance with video display devices. This diagram does not intend to specify how this mosaic is oriented in the dewar or on DEIMOS, whether the illumination is coming from the front or back, nor how the different amplifiers should be numbered.)

There are 8 2k x 4k CCDs, each with 2 amplifiers. Amplifiers and their direction of serial readout are denoted by `>' and `<'. Additionally, the two amplifiers are denoted 1 and 2 in the above graphic. The spatial direction of the spectrograph is horizontal, and the spectral dispersion direction is vertical.

The Keck I instruments LRIS and HIRES already handle the case where readout is done via two CCD amplifiers. In a like manner the pixel streams from the 16 CCD amplifiers in DEIMOS will be descrambled by the DEIMOS instrument computer. In addition to the actual image pixels there may be overscan columns (OSC), prescan columns (PSC), overscan rows (OSR), and prescan rows (PSR) which contain data useful for calibration. In the descrambled memory the data will be arrayed contiguously as indicated below. Each CCD's region of OSC and PSC consists of 2 halves from the 2 different amplifiers.

	+-+-------+-+-+-------+-+-+-------+-+-+-------+-+
	| |  PSR  | | |  PSR  | | |  PSR  | | |  PSR  | |
	| <-------> | <-------> | <-------> | <-------> |
	| |   !   | | |   !   | | |   !   | | |   !   | |
	|P|       |O|P|       |O|P|       |O|P|       |O|
	|S|   !   |S|S|   !   |S|S|   !   |S|S|   !   |S|
	|C|       |C|C|       |C|C|       |C|C|       |C|
	| |   !   | | |   !   | | |   !   | | |   !   | |
	| |-------| | |-------| | |-------| | |-------| |
	| |  OSR  | | |  OSR  | | |  OSR  | | |  OSR  | |
	+-+=======+-+-+=======+-+-+=======+-+-+=======+-+
	| |  OSR  | | |  OSR  | | |  OSR  | | |  OSR  | |
	| |-------| | |-------| | |-------| | |-------| |
	| |   !   | | |   !   | | |   !   | | |   !   | |
	|P|       |O|P|       |O|P|       |O|P|       |O|
	|S|   !   |S|S|   !   |S|S|   !   |S|S|   !   |S|
	|C|       |C|C|       |C|C|       |C|C|       |C|
	| |   !   | | |   !   | | |   !   | | |   !   | |
	| <-------> | <-------> | <-------> | <-------> |
	| |  PSR  | | |  PSR  | | |  PSR  | | |  PSR  | |
	+-+-------+-+-+-------+-+-+-------+-+-+-------+-+
The storage of the OSR in the middle of the image acts as a conceptual aid. It reminds the observer and data reducer that there is a spatial discontinuity between the top and bottom portions of the image. It also introduces new complexity for the quick look display tool.

PSC usually do not provide much calibration information and are often discarded entirely, but OSC are always retained. This will create images which have a "jail-bar" appearance. Again, this will serve as a conceptual aid to remind observers that there are spatial discontinuities in the image data.

It is evident that the instrument computer will require >~128Mbytes of RAM. The data streams from all 16 amplifiers will arrive together and must be distributed without significant delays due to paging of VM.

Note that the scheme for documenting the image sections (below) permits other layouts for the image and calibration data. The above scheme makes it easiest for another program to snip the mosaicked image into its component parts. Experience may demonstrate difficulties with storing or displaying calibration data in the midst of image data. If so, the calibration data can be moved to the outside of all the image data. The FITS keywords which document the layout can still preserve all documentary information about the nature of the data in each section of the mosaic.

7.5.1.1: Storage of this mosaicked image as a FITS file

The in-memory layout described above is suited to a variety of output formats. The image could be stored as one, two, four, or eight FITS files by writing out appropriate subregions of memory. For the case where images from the ``bottom'' only of the array are written to FITS files we need to determine whether their row order should be reversed.

FITS keywords which document how the OSC and PSC are stored have been in use for LRIS and HIRES images. New FITS keywords which document the OSR and PSR in these DEIMOS images have been created. These keywords document all of the characteristics of the image sections in the mosaic.

7.5.1.2: Documentary and cross-reference information in the FITS files

When portions of the CCD image array are stored in separate files it will be important to be able to associate these files with each other. This association will be accomplished in several ways. The files will be given names which clearly indicate the relation. The documentary information in the FITS headers will be adequate to determine the original file name and exposure situation. The FITS primary image array will be followed by another FITS HDU which contains a ``grouping table''.

Hierarchical grouping of FITS files using embedded tables was proposed in a document available as URL=http://adfwww.gsfc.nasa.gov/other/convert/group.html This document details a protocol by which FITS files can be associated with other FITS files.

Software which knows how to use these tables will understand all the relations and process the entire dataset automatically. Software which does not know about these tables can simply ignore the lists. It should be understood that FITS grouping tables are still at the proposal stage, but that operational software exists and uses them to interconvert between FITS and HDF files.

7.5.1.3: Disk storage of spectrally dispersed images

The slitmask frame contains support ribs whose positions image onto the gaps between the adjacent CCDs in the detector mosaic. The distortions of the DEIMOS lens are not so large as to cause any portion of a spectrum in one section of the slit mask to fall onto the CCDs of another section. Therefore, in the case of dispersed images the DEIMOS frames will be stored as 4 separate FITS files of 2k x 8k pixels. This ensures that all scientific data from a single set of slits remain associated in the same file. Each such image will be 32Mbytes when stored as 16-bit integers.

There are several advantages of storing the spectral images in 4 separate pieces. Such images will be only 32MB rather than 128MB. This size difference will make it significantly easier to reduce the data on machines with limited amounts of RAM. Storing the images as 4 pieces allows each piece to be sent off to a different parallel processor. With the images in separate files it will not be necessary to have a central data management scheme which re-integrates the results from each CPU.

7.5.1.4: Disk storage of direct images

In most cases a direct image will be obtained using less than one half of the CCD detector array. If at some later date DEIMOS is used for drift scanning, those images will also be obtained using only one half of the CCD detector array.

In typical operation the number of OSC is such that their spatial extent in such an image will very nearly match the physical gaps between the CCDs. This fortuitous situation means that the appearance of these images on the display will roughly match the spatial coordinates on the sky.

In some cases a direct image may extend onto both halves of the CCD array. Such images can be stored using the obvious combination of mosaic keywords. Typical sizes of direct images should be approximately 8k x 3k pixels (~48Mbytes). Drift scan images, however, could be 8k x N pixels where N is determined by the length of the drift scan.

7.5.1.5: Disk storage of mask alignment images

When a new field of view is acquired by DEIMOS it will be necessary to go through a mask alignment procedure. This procedure will ensure that the light from each object goes through the appropriate slit. During the mask alignment procedure the DEIMOS CCD array may be reading out up to 16 sub-regions of the detectors. These sub-regions will typically be disjoint sections which are located at arbitrary positions on the detector. We assert the constraint that each of these sub-regions must be rectangular and identical in size. They will be stored in memory such that they form a horizontal mosaic which is sorted by x-position along the slitmask and CCDs.

7.5.1.6: Disk storage of non-contiguous subsections

In any case where the readout regions of the CCDs are physically contiguous the mosaicking schemes detailed above can be used. It may also be possible to command the CCD controllers to read out multiple non-contiguous regions of different sizes. In this case the data from each CCD should be written to a separate FITS file. Such FITS files will contain documentary and cross-reference information in the same manner as the files which combine information from multiple CCDs.
Steve Allen <sla@ucolick.org>
$Date: 1996/03/19 00:43:51 $