Back to Volume
Paper: The Operation and Architecture of the Keck Observatory Archive
Volume: 485, Astronomical Data Analysis Software and Systems XXIII
Page: 123
Authors: Berriman, G. B.; Gelino, C. R.; Laity, A.; Kong, M.; Swain, M.; Holt, J.; Goodrich, R.; Mader, J.; Tran, H. D.
Abstract: The Infrared Processing and Analysis Center (IPAC) and the W. M. Keck Observatory (WMKO) are collaborating to build an archive for the twin 10-m Keck Telescopes, located near the summit of Mauna Kea. The Keck Observatory Archive (KOA) takes advantage of IPAC's long experience with managing and archiving large and complex data sets from active missions and serving them to the community; and of the Observatory's knowledge of the operation of its sophisticated instrumentation and the organization of the data products. By the end of 2013, KOA will contain data from all eight active observatory instruments, with an anticipated volume of 28 TB. The data include raw science and observations, quick look products, weather information, and, for some instruments, reduced and calibrated products. The goal of including data from all instruments is the cumulation of a rapid expansion of the archive's holdings, and already data from four new instruments have been added since October 2012. One more active instrument, the integral field spectrograph OSIRIS, is scheduled for ingestion in December 2013. After preparation for ingestion into the archive, the data are transmitted electronically from WMKO to IPAC for curation in the physical archive. This process includes validation of the science and content of the data and verification that data were not corrupted in transmission. The archived data include both newly-acquired observations and all previously acquired observations. The older data extends back to the date of instrument commissioning; for some instruments, such as HIRES, these data can extend as far back as 1994. KOA will continue to ingest all newly obtained observations, at an anticipated volume of 4 TB per year, and plans to ingest data from two decommissioned instruments. Access to these data is governed by a data use policy that guarantees Principal Investigators (PI) exclusive access to their data for at least 18 months, and allows for extensions as granted by institutional Selecting Officials. Approximately one-half of the data in the archive are public. The archive architecture takes advantage of existing software and is designed for sustainability. The data preparation and quality assurance software exploits the software infrastructure at WMKO, and the physical archive at IPAC re-uses the portable component based architecture developed originally for the Infrared Science Archive, with extensions custom to KOA as needed. We will discuss the science services available to end-users. These include web and program query interfaces, interactive tabulation of data and metadata, association of files with science files, and interactive visualization of data products. We will discuss how the growth in the archive holdings has led to to a growth in usage and published science results. Finally, we will discuss the future of KOA, including the provision of data reduction pipelines and interoperability with world-wide archives and data centers, including VO-compliant services.
Back to Volume