DEIMOS PDR Chapter 9

9. SOFTWARE AND DATA HANDLING SYSTEM

9.1 Introduction

The preliminary design of the DEIMOS software and data handling system is still in progress and will not be reviewed at this November 15, 1994 DEIMOS PDR. A separate software PDR will be held when the preliminary software design is complete, and that is currently scheduled for mid-May 1995. There are several reasons for this:

First, the UCO/Lick software manpower available to work on DEIMOS has been constrained by demands of other UCO/Lick and Keck instrumentation projects, which are only now starting to wind down. As a result, the DEIMOS software design effort to date has been extremely limited, although this effort will be ramping up as current projects ramp down.

Second, several pivotal software design choices cannot yet be decided. For example, CARA has proposed that the EPICS system (a distributed instrument control system in widespread use in high energy physics and planned for use by several major telescopes, including Gemini) be used for the Keck II telescope Drive and Control System (DCS). CARA are suggesting, although not requiring, that EPICS also be used for control of Keck II instruments such as DEIMOS. To promote the use of EPICS for Keck II instruments, the Project has invited the instrument software development teams to a CARA-sponsored EPICS training session the week of December 12. The decision on whether to use EPICS for DEIMOS awaits that session.

Third, based on the experience of software development for both HIRES and LRIS, it is important to avoid ramping up the software development effort too quickly, before the rest of the instrument design has stabilized to the point where the software functional requirements can be defined, and to conserve software manpower for the later phases of the project, where it will be most needed. This is and has been a fundamental assumption on which the current software budget and schedule is based.

Accordingly, rather than present an incomplete skeletal design for the DEIMOS software and data handling system, this chapter will provide a progress report on the current design efforts, and will:

Review the data rates and data storage, display, and handling requirements.
Identify major areas requiring further research/evaluation.
Provide a more detailed but still preliminary outline for the functional requirements document for the software PDR.
Summarize the planned software effort leading up to the software PDR.
Introduce a preliminary schedule for the period following the software PDR.

9.2 Data Flow and Data Rates

The default data system described in Chapter 4 is essentially a scaled-up of the HIRES system, but using second-generation UCSD Controllers. The following discussion assumes a 2 x 4 mosaic of three-side-buttable 2K x 4K CCDs. Each chip has 2 readout amplifiers located at opposite ends of a single 2 kilopixel serial register, which is split in the middle. During image readout, 1024 image pixels per row will be clocked into each readout amplifier. If prescan and overscan pixels are included, there will be about 1060 pixels per amplifier per row. Thus, for full-frame readout, each readout amplifier will see 4096 rows x 1060 pixels per row, or 4.34 megapixels.

For reference, the per-pixel readout time for the HIRES CCD system is 33.5 microseconds in single-amplifier readout mode. This is adequate to ensure our minimum performance spec of 120 sec to read all chips on DEIMOS. However, for the purpose of calculating the required bandwidth for the component data paths, we assume a per-pixel readout time of only 10 microseconds to allow for the possibility of reduced multiplexing and for improvements in the performance characteristics of the CCDs and/or the UCSD system, which seem probable. As noted in Chapter 4, 10 microseconds is our new readout goal.

There are 16 amplifiers per dewar. For a full-frame readout, each amplifier will produce a data stream of 4.34 megapixels, yielding 69.44 megapixels per image. Since each digitized pixel produces 2 bytes of data, the image from each dewar will contain 138.88 megabytes (MB). Given the 10 microsecond per-pixel read time, each amplifier has a data rate of 100 kilopixels sec-1, or 200 kilobytes sec-1. Since all 16 amplifiers in each dewar will be read simultaneously, the aggregate data rate per dewar is 3.2 MB sec-1, or 6.4 MB sec-1 from both dewars.

A block diagram of the data system was shown in Figure 4.1. As noted there, the data rates reflect a 20 microsecond readtime per pixel and hence should be multiplied by two to meet our more stringent current goal of 10 microseconds. The exact number of analog boards and other components will also differ, since Figure 4.1 refers to the first-generation UCSD Controller whereas our plan is to use the second-generation Controller. Regardless, the major system components will include:

2 preamplifier boxes, one per dewar.
2 UCSD CCD controllers, one per dewar, containing analog boards, timing boards, and a utility board.
VME Crate in Control Room; accepts data and formats it for transfer to Instrument Control Computer.
Instrument Control Computer, keyboard, disk drives, video monitors, and storage media.
Data Analysis Computer; for further, detailed data analysis.

With the UCSD CCD Controllers, each timing board transmits its data to the Control Room VME Crate via a pair of fiber-optic cables contained in the telescope cable wrap. The timing board/fiber cable throughput of the current UCSD system is about 3.3 MB sec-1, while the aggregate data rate from each dewar is 3.2 MB sec-1, so capacity in this link is just adequate. With the second-generation UCSD system, the corresponding throughput will be higher, about 5 MB sec-1.

The VME Crate connects to the Instrument Computer via an industry-standard interface over a fiber link. There will have to be two such links rather than the one shown in Figure 4.1 to accommodate the goal of 10 microsecond readtime per pixel.

9.3 Data Storage, I/O, and Display

The video display will be able to display only one beam at a time. However, we plan enough video memory to keep pictures from both beams in the display memory at one time. Each 8K x 8K image contains 128 MB, for a total of 256 MB of video memory required. When pictures are stored in memory, the video display will be able to switch between them in 1 second.

Manipulation of images is a concern. If images are not stored in RAM but must be fetched from disk, processing speed will be limited by disk I/O time. Assuming current SCSI speeds of 2 MB sec-1, it will take 60 seconds to read an image off disk. This would make flatfielding and other image arithmetic time consuming. We will solve this problem first by including generous amounts of RAM in the Instrument Computer (512 Mbytes is budgeted). The second strategy is to rely on faster RAID I/O technology. This technology is 10 times faster than SCSI (20 MB sec-1) and will allow full DEIMOS images to be read off disk in only 15 seconds. RAID disks are discussed in the next section.

The video display of a mosaic of 8 large CCDs is challenging. Fortunately 2Kx2K monitors are just now becoming available. A full-up scheme might utilize a mosaic of 4 of these, with 2x2 pixel binning, plus a separate monitor that would show a blowup of a subregion at full scale. Or we might use 2 side-by-side monitors to show just the upper or lower portions of each detector, or possibly just one monitor with rapid zoom and pan. The video display problem is discussed further in the next section.

9.4 Major Areas Requiring Further Research and Evaluation

A number of aspects of the DEIMOS data handling software and hardware are pushing the limits of either our experience or current technology. As such, a vital part of our initial design work is to conduct adequate research into new and emerging technologies in disk drives, networks, and image display systems. We need an accurate understanding of the costs and capabilities of these systems in order to establish reasonable performance goals. In addition, we need to gain practical experience displaying and manipulating 8K by 8K tiled images. Accordingly, we have identified several areas of research on which we hope to make significant progress by the software PDR, and for which some limited results may be available for the November review.

9.4.1 High-Capacity

Disk Drives Image storage capacity and data rates required by DEIMOS exceed those of HIRES and LRIS by an order of magnitude. Based on our current, albeit limited, knowledge of high-speed/high-capacity disk technologies (including RAID), we believe these higher requirements can be met within the amount budgeted, which is $90K. At current prices, this would buy 60 GB of RAID disk, enough to store 250 complete exposures (both beams), or 125 exposures of both raw and flattened data.

As we do not yet have any direct experience with this class of disk drive, we propose to:

a) Survey a sample of reputable vendors for current product specifications and pricing, and if possible arrange a few product demonstrations.

b) Solicit comments from other observatories (e.g., CFHT) that have purchased RAID disks.

c) Conduct a limited literature search of relevant industry trade journals for evaluations of products and surveys of where such technology is headed.

Since this technology is changing very rapidly, it doesn't make sense to do an exhaustive amount of research this early in the project. However, prior to the software PDR, we need at least to be sure that our estimates of the costs and capabilities of these drives are valid, and will invest roughly a man-week of research effort in this area. More extensive research would then be conducted in early 1997, prior to purchase of the drives.

9.4.2 High-Bandwidth Network Technologies

The high data rates required by DEIMOS will also outstrip the capabilities of the existing ethernet networks with which we are most familiar. This affects the architecture of the DEIMOS data handling system in two areas. First, it impacts the data pathway between the CCD VME crate and the Instrument Computer (see Figure 4.1). Here, we have suggested the use of FDDI fiber links. Network technology would also impact the links between the various workstations on which DEIMOS images will be displayed and reduced.

There are several competing technologies (fast ethernet, FDDI, ATM, and others) that might address our bandwidth requirements. However, this is also a rapidly changing field, and while we are at least aware of the alternatives, we lack detailed knowledge or direct experience with any of them. In order to set reasonable bounds on our system architecture within the constraints of our budget, we need to better understand these technologies. We propose a limited research initiative, similar to what we proposed for disk drive technologies. Once again, more extensive research would be conducted in late 1996 to early 1997, prior to purchase of the final systems.

9.4.3 8K x 8K Tiled Images

One of the major challenges facing the DEIMOS software and data-handling system is the efficient display and manipulation of 8Kx 8K tiled images. Currently, most workstation screens and video display monitors have resolutions of order 1K x 1K, although 2K x 2K devices are just now becoming available. While 4x4 pixel binning could be used to display a DEIMOS image on a 2K x 2K monitor, at this level of binning faint features would become difficult to discern.

Above, we described a scheme that would utilize a 2 x 2 mosaic of four 2K x 2K video display monitors to display a full 8K x 8K image using 2x2 pixel binning. A fifth monitor would be used to display an unbinned sub-region of the image, and that sub-region could be rapidly panned within the full image. While we believe that such a system could be assembled and have budgeted accordingly, it is not clear that such an arrangement is optimal, or that an astronomer would find it convenient to interact with an image so displayed. In various discussions, it has become abundantly clear that it is extremely difficult to visualize in the abstract how one would interact with a DEIMOS image displayed on a mosaic of monitors (or on other configurations of monitors). Our current thinking is that a single 2K x 2K video display might actually suffice, provided that it had a rapid capability for panning and zooming within the full 8K x 8K image, but this is by no means certain.

To better understand the nature of the task, we are planning to conduct a limited set of experiments to display simulated 8K x 8K DEIMOS images using different types of video display resolutions and configurations (mosaiced versus non-mosaiced, binned versus panned), and obtain feedback from potential DEIMOS observers as to which configurations work best. As part of these experiments, we will contact vendors of 2K x 2K video displays and attempt to arrange demonstrations and loans of equipment. Since purchase of the final system will not occur for several years, we will limit these experiments to refine our functional requirements and will conduct a more extensive evaluation of the available hardware closer to the time of purchase. Given the critical importance of this component of the system, we believe it is worth investing several weeks of effort in these initial tests prior to the software PDR.

9.4.4 Instrument Control Environments: KTL/Music and KTL/EPICS

Until recently we have been assuming that the DEIMOS instrument control hardware and software would follow the HIRES model, since it has been shown to work reasonably well. As we are already thoroughly familiar with that model, this is the least expensive course.

It should be noted that the HIRES software model consists of a number of distinct layers, and that certain layers of the model might be replaced without changing the entire model. As noted earlier, CARA has proposed the use of the EPICS system for the Keck II Telescope control software (DCS) and is urging its use for the Keck II instruments. At present, we do not know enough about EPICS to determine whether it would be of major benefit to the DEIMOS software effort. Were EPICS to be used within DEIMOS, it would replace a layer of the messaging system (KTL/Music) that is relatively close to the low-level hardware. CARA has suggested that EPICS could overcome some of the known deficiencies and limitations of the KTL/Music system, that it would produce more reliable systems, and that it would be more maintainable since EPICS is already in use at dozens of sites while KTL/Music is (and will likely always be) used at only a few. Since the KTL/Music layer that EPICS would replace is rather removed from the level of the user interfaces, the switch would be relatively transparent to observers.

We plan to investigate EPICS further later this year, and to attend the training session provided by CARA. We anticipate spending several weeks to a month on this initial evaluation prior to the software PDR. If a clear and convincing case can be made that the benefits of switching to EPICS outweigh the costs (which are non-negligible), then we will likely switch from using KTL/Music to KTL/EPICS. Otherwise, we will proceed as originally planned in following the HIRES software model, including the use of KTL/Music.