VIRTUAL RESEARCH ENVIRONMENT GOES LIVE: BIG DATA SHARED ACROSS SCIENTIFIC ANALYSES

The EOSC (European Open Science Cloud) Future Project is aimed at bringing together diverse analysis workflows from Astrophysics and High Energy Physics communities by providing a reliable AAI framework, federated storage services with the required exabyte-scale data management needs, and a large computing infrastructure on which to run analyses. The project aims at demonstrating how a common platform could support the goals of Dark Matter and Extreme Universe Science Projects in the respect of FAIR data policies. Such a platform is being implemented as a Virtual Research Environment (VRE), an ecosystem with different complexity levels enhanced by a Web UI interface; it already provides functionalities for data injection, replication and processing through a containerized jupyter notebook extension of Rucio, a Data Management framework developed at CERN. The backend infrastructure of the VRE is the ESCAPE Data Lake, a Kubernetes (K8s) cluster for data management and orchestration including a main server, an authorization server and an UI interface. As the reproducibility of scientific results is becoming more and more crucial for the advancement of science and for testing new theories, as well as being useful to spark scientific collaboration, the project’s aim is to expand and adapt the current ecosystem’s capabilities to enable scientists to not only interact with experimental data locally during a workflow run, but also to flexibly spawn complete re-analyses and interact with large data sets from anywhere. For this, the integration with the VRE of Reana [1], a re-analysis platform developed at CERN supporting various computing backends (Kubernetes, HTCondor, Slurm), is a key step of the project. The power of Reana relies on the fact that it takes advantage of declarative (.yaml files), instead of imperative, data analysis approach and achieves extended portability thanks to recent advances of container technology in the general IT industry. Before the start of the project, Reana takes input files from the ...