As current- and next-generation astronomical instruments come online, they
will generate an unprecedented deluge of data. Analyzing these data in real
time presents unique conceptual and computational challenges, and their
long-term storage and archiving is scientifically essential for generating
reliable, reproducible results. We present here the real-time processing (RTP)
system for the Hydrogen Epoch of Reionization Array (HERA), a radio
interferometer endeavoring to provide the first detection of the highly
redshifted 21 cm signal from Cosmic Dawn and the Epoch of Reionization by an
interferometer. The RTP system consists of analysis routines run on raw data
shortly after they are acquired, such as calibration and detection of
radio-frequency interference (RFI) events. RTP works closely with the
Librarian, the HERA data storage and transfer manager which automatically
ingests data and transfers copies to other clusters for post-processing
analysis. Both the RTP system and the Librarian are public and open source
software, which allows for them to be modified for use in other scientific
collaborations. When fully constructed, HERA is projected to generate over 50
terabytes (TB) of data each night, and the RTP system enables the successful
scientific analysis of these data.
Description
A Real Time Processing System for Big Data in Astronomy: Applications to HERA
%0 Generic
%1 laplante2021processing
%A La Plante, Paul
%A Williams, Peter K. G.
%A Kolopanis, Matthew
%A Dillon, Joshua S.
%A Beardsley, Adam P.
%A Kern, Nicholas S.
%A Wilensky, Michael
%A Ali, Zaki S.
%A Abdurashidova, Zara
%A Aguirre, James E.
%A Alexander, Paul
%A Balfour, Yanga
%A Bernardi, Gianni
%A Billings, Tashalee S.
%A Bowman, Judd D.
%A Bradley, Richard F.
%A Bull, Phil
%A Burba, Jacob
%A Carey, Steve
%A Carilli, Chris L.
%A Cheng, Carina
%A DeBoer, David R.
%A Dexter, Matt
%A Acedo, Eloy de Lera
%A Ely, John
%A Ewall-Wice, Aaron
%A Fagnoni, Nicolas
%A Fritz, Randall
%A Furlanetto, Steven R.
%A Gale-Sides, Kingsley
%A Glendenning, Brian
%A Gorthi, Deepthi
%A Greig, Bradley
%A Grobbelaar, Jaspar
%A Halday, Ziyaad
%A Hazelton, Bryna J.
%A Hewitt, Jacqueline N.
%A Hickish, Jack
%A Jacobs, Daniel C.
%A Julius, Austin
%A Kerrigan, Joshua
%A Kittiwisit, Piyanat
%A Kohn, Saul A.
%A Lanman, Adam
%A Lekalake, Telalo
%A Lewis, David
%A Liu, Adrian
%A MacMahon, David
%A Malan, Lourence
%A Malgas, Cresshim
%A Maree, Matthys
%A Martinot, Zachary E.
%A Matsetela, Eunice
%A Mesinger, Andrei
%A Molewa, Mathakane
%A Morales, Miguel F.
%A Mosiane, Tshegofalang
%A Murray, Steven
%A Neben, Abraham R.
%A Nikolic, Bojan
%A Parsons, Aaron R.
%A Pascua, Robert
%A Patra, Nipanjana
%A Pieterse, Samantha
%A Pober, Jonathan C.
%A Razavi-Ghods, Nima
%A Ringuette, Jon
%A Robnett, James
%A Rosie, Kathryn
%A Santos, Mario G.
%A Sims, Peter
%A Smith, Craig
%A Syce, Angelo
%A Thyagarajan, Nithyanandan
%A Zheng, Haoxuan
%D 2021
%K library
%T A Real Time Processing System for Big Data in Astronomy: Applications to
HERA
%U http://arxiv.org/abs/2104.03990
%X As current- and next-generation astronomical instruments come online, they
will generate an unprecedented deluge of data. Analyzing these data in real
time presents unique conceptual and computational challenges, and their
long-term storage and archiving is scientifically essential for generating
reliable, reproducible results. We present here the real-time processing (RTP)
system for the Hydrogen Epoch of Reionization Array (HERA), a radio
interferometer endeavoring to provide the first detection of the highly
redshifted 21 cm signal from Cosmic Dawn and the Epoch of Reionization by an
interferometer. The RTP system consists of analysis routines run on raw data
shortly after they are acquired, such as calibration and detection of
radio-frequency interference (RFI) events. RTP works closely with the
Librarian, the HERA data storage and transfer manager which automatically
ingests data and transfers copies to other clusters for post-processing
analysis. Both the RTP system and the Librarian are public and open source
software, which allows for them to be modified for use in other scientific
collaborations. When fully constructed, HERA is projected to generate over 50
terabytes (TB) of data each night, and the RTP system enables the successful
scientific analysis of these data.
@misc{laplante2021processing,
abstract = {As current- and next-generation astronomical instruments come online, they
will generate an unprecedented deluge of data. Analyzing these data in real
time presents unique conceptual and computational challenges, and their
long-term storage and archiving is scientifically essential for generating
reliable, reproducible results. We present here the real-time processing (RTP)
system for the Hydrogen Epoch of Reionization Array (HERA), a radio
interferometer endeavoring to provide the first detection of the highly
redshifted 21 cm signal from Cosmic Dawn and the Epoch of Reionization by an
interferometer. The RTP system consists of analysis routines run on raw data
shortly after they are acquired, such as calibration and detection of
radio-frequency interference (RFI) events. RTP works closely with the
Librarian, the HERA data storage and transfer manager which automatically
ingests data and transfers copies to other clusters for post-processing
analysis. Both the RTP system and the Librarian are public and open source
software, which allows for them to be modified for use in other scientific
collaborations. When fully constructed, HERA is projected to generate over 50
terabytes (TB) of data each night, and the RTP system enables the successful
scientific analysis of these data.},
added-at = {2021-04-12T12:13:54.000+0200},
author = {La Plante, Paul and Williams, Peter K. G. and Kolopanis, Matthew and Dillon, Joshua S. and Beardsley, Adam P. and Kern, Nicholas S. and Wilensky, Michael and Ali, Zaki S. and Abdurashidova, Zara and Aguirre, James E. and Alexander, Paul and Balfour, Yanga and Bernardi, Gianni and Billings, Tashalee S. and Bowman, Judd D. and Bradley, Richard F. and Bull, Phil and Burba, Jacob and Carey, Steve and Carilli, Chris L. and Cheng, Carina and DeBoer, David R. and Dexter, Matt and Acedo, Eloy de Lera and Ely, John and Ewall-Wice, Aaron and Fagnoni, Nicolas and Fritz, Randall and Furlanetto, Steven R. and Gale-Sides, Kingsley and Glendenning, Brian and Gorthi, Deepthi and Greig, Bradley and Grobbelaar, Jaspar and Halday, Ziyaad and Hazelton, Bryna J. and Hewitt, Jacqueline N. and Hickish, Jack and Jacobs, Daniel C. and Julius, Austin and Kerrigan, Joshua and Kittiwisit, Piyanat and Kohn, Saul A. and Lanman, Adam and Lekalake, Telalo and Lewis, David and Liu, Adrian and MacMahon, David and Malan, Lourence and Malgas, Cresshim and Maree, Matthys and Martinot, Zachary E. and Matsetela, Eunice and Mesinger, Andrei and Molewa, Mathakane and Morales, Miguel F. and Mosiane, Tshegofalang and Murray, Steven and Neben, Abraham R. and Nikolic, Bojan and Parsons, Aaron R. and Pascua, Robert and Patra, Nipanjana and Pieterse, Samantha and Pober, Jonathan C. and Razavi-Ghods, Nima and Ringuette, Jon and Robnett, James and Rosie, Kathryn and Santos, Mario G. and Sims, Peter and Smith, Craig and Syce, Angelo and Thyagarajan, Nithyanandan and Zheng, Haoxuan},
biburl = {https://www.bibsonomy.org/bibtex/2901c68daf79e4e71f9ddaef84ada3838/gpkulkarni},
description = {A Real Time Processing System for Big Data in Astronomy: Applications to HERA},
interhash = {e33783a0aa1f682f1237e40fb84d7499},
intrahash = {901c68daf79e4e71f9ddaef84ada3838},
keywords = {library},
note = {cite arxiv:2104.03990Comment: 14 pages, 3 figures, submitted to Astronomy and Computing},
timestamp = {2021-04-12T12:13:54.000+0200},
title = {A Real Time Processing System for Big Data in Astronomy: Applications to
HERA},
url = {http://arxiv.org/abs/2104.03990},
year = 2021
}