| |
 |
| Paper: |
Rapid Large Scale Reprocessing of the ODI Archive using the QuickReduce Pipeline |
| Volume: |
495, Astronomical Data Analysis Software and Systems XXIV (ADASS XXIV) |
| Page: |
53 |
| Authors: |
Gopu, A.; Kotulla, R.; Young, M. D.; Hayashi, S.; Harbeck, D.; Liu, W.; Henschel, R. |
| Abstract: |
The traditional model of astronomers collecting their observations as raw
instrument data is being increasingly replaced by astronomical observatories
serving standard calibrated data products to observers and to the public at
large once proprietary restrictions are lifted. For this model to be effective,
observatories need the ability to periodically re-calibrate archival data
products as improved master calibration products or pipeline improvements become
available, and also to allow users to rapidly calibrate their data
on-the-fly. Traditional astronomy pipelines are heavily I/O dependent and do not
scale with increasing data volumes. In this paper, we present the One Degree
Imager - Portal, Pipeline and Archive (ODI-PPA) calibration pipeline framework which
integrates the efficient and parallelized QuickReduce pipeline
to enable a large number of simultaneous, parallel data reduction jobs - initiated by
operators AND/OR users - while also ensuring rapid processing times and full
data provenance. Our integrated pipeline system allows re-processing
of the entire ODI archive (∼15,000 raw science frames, ∼3.0 TB compressed)
within ∼18 hours using twelve 32-core compute nodes on the Big Red II
supercomputer. Our flexible, fast, easy to operate, and highly scalable
framework improves access to ODI data, in particular when data rates double with an
upgraded focal plane (scheduled for 2015), and also serve as a template for
future data processing infrastructure across the astronomical community and
beyond. |
|
|
|
|
|
