| |
 |
| Paper: |
A globally distributed and scalable data post-processing framework for WALLABY science |
| Volume: |
535, Astronomical Data Analysis Software and Systems XXXI |
| Page: |
65 |
| Authors: |
Shen, A. X.; Reynolds, T. |
| Abstract: |
WALLABY is the ASKAP all-sky neutral hydrogen (HI) survey. The post-processing involves mosaicking of spectral line data cubes, source finding, crossmatching, computing moment maps for the detected galaxies, kinematics and ad-hoc interactions with the data. The aforementioned processes are provided by a number of collaborating institutions, which are distributed internationally and utilize different computing facilities. Over the course of the survey, these institutions will process data on the scale of petabytes. The existing post-processing approach is mostly manual,
leading to unnecessary effort by scientists. As such, a new framework is required to efficiently process large ASKAP data across international borders for the full survey.
We have developed a new framework for distributed, scalable and portable WALLABY data post-processing. The technology stack includes PostgreSQL for relational databases, replicated across institutions with Bucardo, to provide a central location for WALLABY survey data products. Interactive web interfaces such as Open OnDemand
portals, Jupyter notebooks and VO services provide user access to the data. Computational pipelines, composed in Nextflow, allow HPC resource agnostic and parallel execution of containerized applications. In this discussion we provide an overview of the framework, system architecture, and how it will be utilized to help WALLABY scientists process ASKAP spectral-line data. |
|
|
|
|
|
