A continuación procedemos a mostrar cómo usar de forma básica sistema de contenerización para entornos HPC apptainer.
Procedemos a loguearnos en Drago.
# ssh usuario@drago.csic.es
Podemos hacer uso del mismo de la siguiente forma:
[usuario@drago31010015 ~]$ module av
------------------------------------------------------------------------------------------------ /lustre/opt/rama -------------------------------------------------------------------------------------------------
rama0.2 rama0.3
------------------------------------------------------------------------------------------ ⚛⚛⚛ FOSS 0.2 ⚛⚛⚛ -------------------------------------------------------------------------------------------
APBS/3.0.0 GCC/10.2.0 Java/13.0.2 (D:13) SAGE/6.4.2 flex/2.6.4 (D) gettext/0.21 (D) picard/2.25.1-Java-11
BLAST/2.11.0-Linux_x86_64 GCC/10.3.0 Julia/1.7.2-linux-x86_64 Trimmomatic/0.39-Java-11 foss/2020a gompi/2020a pkg-config/0.29.2 (D)
Bison/3.3.2 GCC/11.2.0 (D) M4/1.4.18 ant/1.10.7-Java-11 foss/2020b gompi/2020b pkgconf/1.8.0
Bison/3.8.2 (D) GCCcore/8.3.0 M4/1.4.19 (D) ant/1.10.11-Java-11 (D) foss/2021b (D) gompi/2021a pplacer/1.1.alpha19
CUDA/11.4.1 GCCcore/9.3.0 Miniconda3/4.9.2 binutils/2.32 fosscuda/2020a gompi/2021b (D) zlib/1.2.11 (D)
CUDAcore/11.0.2 GCCcore/10.2.0 NVHPC/21.11 binutils/2.34 fosscuda/2020b (D) gompic/2020a
CUDAcore/11.1.1 (D) GCCcore/10.3.0 Nextflow/21.03.0 binutils/2.35 gamess/2021 gompic/2020b (D)
FastQC/0.11.9-Java-11 GCCcore/11.2.0 (D) OpenSSL/1.1 (D) binutils/2.36.1 gcccuda/2020a ncurses/6.0
GCC/8.3.0 Java/11.0.2 QIIME2/2021.8 binutils/2.37 (D) gcccuda/2020b (D) ncurses/6.1
GCC/9.3.0 Java/12.0.2 RDP-Classifier/2.13-Java-11 cuDNN/8.0.4.30-CUDA-11.1.1 gettext/0.20.1 ncurses/6.2 (D)
----------------------------------------------------------------------------------------- ⚛⚛⚛ Campus 0.2 ⚛⚛⚛ ------------------------------------------------------------------------------------------
gaussian/09.gaussian gaussian/16.gaussian.C01 gaussian/16.gaussian (D)
--------------------------------------------------------------------------------------- ⚛⚛⚛ Restricted 0.2 ⚛⚛⚛ ----------------------------------------------------------------------------------------
Bison/3.0.4 GCC/10.2.0 M4/1.4.18 binutils/2.35 gcccuda/2020b ifort/2018.3.222-GCC-7.3.0-2.30 intel/2020b
Bison/3.8.2 GCC/11.2.0 M4/1.4.19 binutils/2.36.1 gettext/0.21 iimpi/2020b intel/2021b
COMSOL/6.0 GCCcore/7.3.0 MaterialsStudio/22.1.MaterialsStudio binutils/2.37 gompi/2020b iimpi/2021b intel/2022.00 (D)
COMSOL/6.1 (D) GCCcore/9.3.0 OpenSSL/1.1 flex/2.6.4 gompi/2021b iimpi/2022.00 (D) ncurses/6.2
COMSOL6/6.0 GCCcore/10.2.0 Schrodinger/2022-2 foss/2020b gompic/2020b imkl/2021.4.0 pkg-config/0.29.2
CUDAcore/11.0.2 GCCcore/10.3.0 Schrodinger/2022-4 (D) foss/2021b icc/2018.3.222-GCC-7.3.0-2.30 imkl/2022.0.1 (D) zlib/1.2.11
CUDAcore/11.1.1 GCCcore/11.2.0 binutils/2.30 fosscuda/2020b iccifort/2018.3.222-GCC-7.3.0-2.30 intel-compilers/2021.4.0
GCC/9.3.0 M4/1.4.17 binutils/2.34 gcccuda/2020a iccifort/2020.4.304 (D) intel-compilers/2022.0.1 (D)
Where:
Aliases: Aliases exist: foo/1.2.3 (1.2) means that "module load foo/1.2" will load foo/1.2.3
D: Default Module
Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
[usuario@drago31010015 ~]$ module load rama0.3
[usuario@drago31010015 ~]$ module av
-------------------------------------------------------------------------------------- /lustre/opt/foss/0.3/modules/all/Core --------------------------------------------------------------------------------------
APBS/3.0.0_Linux GCC/8.3.0 Julia/1.8.5-linux-x86_64 ant/1.10.12-Java-11 (D) flex/2.6.4 (D) irma/1.0.3
AutoDock-Vina/1.1.2-linux_x86 GCC/10.2.0 Julia/1.9.3-linux-x86_64 (D) apptainer/1.3.2 foss/2019b mothur/1.48.0
BLAST/2.11.0-Linux_x86_64 GCC/10.3.0 M4/1.4.18 binutils/2.31.1 foss/2020b ncurses/6.0
Bison/3.0.5 GCC/11.2.0 M4/1.4.19 (D) binutils/2.32 foss/2021a ncurses/6.1
Bison/3.3.2 GCC/11.3.0 Mamba/23.1.0-1 binutils/2.35 foss/2021b ncurses/6.2
Bison/3.8.2 (D) GCC/12.2.0 Miniconda3/22.11.1-1 binutils/2.36.1 foss/2022a ncurses/6.3 (D)
CMake/3.12.1 GCC/12.3.0 (D) NVHPC/22.7-CUDA-11.7.0 binutils/2.37 foss/2022b (D) picard/2.25.0-Java-11
CUDA/11.3.1 GCCcore/8.2.0 NVHPC/23.1-CUDA-12.0.0 (D) binutils/2.38 gettext/0.19.8.1 picard/2.25.1-Java-11 (D)
CUDA/11.4.1 GCCcore/8.3.0 Nextflow/22.10.6 binutils/2.39 gettext/0.21 pkgconf/1.8.0 (D)
CUDA/11.7.0 GCCcore/10.2.0 OpenSSL/1.1 (D) binutils/2.40 (D) gettext/0.21.1 (D) pplacer/1.1.alpha19
CUDA/12.0.0 GCCcore/10.3.0 QIIME2/2022.11 bwa-mem2/2.2.1_x64 gfbf/2022b tbl2asn/20220427-linux64
CUDA/12.1.0 (D) GCCcore/11.2.0 QIIME2/2023.2 (D) charliecloud/0.34c gompi/2019b zlib/1.2.11
CellRanger/7.2.0 GCCcore/11.3.0 RDP-Classifier/2.13-Java-11 cuDNN/8.2.1.32-CUDA-11.3.1 gompi/2020b zlib/1.2.12
Cereal/1.3.0 GCCcore/12.2.0 SAGE/6.4.2 cuDNN/8.2.2.26-CUDA-11.4.1 gompi/2021a zlib/1.2.13 (D)
Cytoscape/3.9.1-Java-11 GCCcore/12.3.0 (D) Trimmomatic/0.39-Java-11 cuDNN/8.4.1.50-CUDA-11.7.0 (D) gompi/2021b zsh/5.8
FEniCS/2019.1.0 IGV/2.16.2-Java-11 Trimmomatic/0.39-Java-17 (D) ecBuild/3.7.0 gompi/2022a
FastQC/0.11.9-Java-11 Java/11.0.18 (11) VarScan/2.4.4-Java-11 fenics-dolfinx/0.7.1 gompi/2022b
GCC/8.2.0-2.31.1 Java/17.0.6 (D:17) ant/1.10.9-Java-11 ffnvcodec/11.1.5.2 gompi/2023a (D)
------------------------------------------------------------------------------------- /lustre/opt/campus/0.3/modules/all/Core -------------------------------------------------------------------------------------
gaussian/09.gaussian gaussian/16.gaussian.C01 gaussian/16.gaussian.C02 gaussian/16.gaussian (D)
----------------------------------------------------------------------------------- /lustre/opt/restricted/0.3/modules/all/Core -----------------------------------------------------------------------------------
Bison/3.0.4 COMSOL/6.2 GCCcore/12.2.0 M4/1.4.18 OpenSSL/1.1 binutils/2.28 binutils/2.39 flex/2.6.3 iimpi/2022b intel-compilers/2022.2.1 pkgconf/1.8.0 zlib/1.2.12
Bison/3.8.2 GCCcore/7.3.0 M4/1.4.17 M4/1.4.19 Schrodinger/2023-4 binutils/2.30 binutils/2.40 flex/2.6.4 imkl/2022.2.1 intel/2022b zlib/1.2.11 zlib/1.2.13
------------------------------------------------------------------------------------------------ /lustre/opt/rama -------------------------------------------------------------------------------------------------
rama0.2 rama0.3 (L)
Where:
L: Module is loaded
Aliases: Aliases exist: foo/1.2.3 (1.2) means that "module load foo/1.2" will load foo/1.2.3
D: Default Module
Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
[usuario@drago31010015 ~]$ module spider apptainer
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
apptainer: apptainer/1.3.2
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Description:
Apptainer is a container platform. It allows you to create and run containers that package up pieces of software in a way that is portable and reproducible. You can build a container using Apptainer
on your laptop, and then run it on many of the largest HPC clusters in the world, local university or company clusters, a single server, in the cloud, or on a workstation down the hall. Your container
is a single file, and you don’t have to worry about how to install all the software you need on each different operating system.
You will need to load all module(s) on any one of the lines below before the "apptainer/1.3.2" module is available to load.
rama0.3
Help:
Description
===========
Apptainer is a container platform. It allows you to create
and run containers that package up pieces of software in a way that is portable
and reproducible. You can build a container using Apptainer on your laptop, and
then run it on many of the largest HPC clusters in the world, local university
or company clusters, a single server, in the cloud, or on a workstation down
the hall. Your container is a single file, and you don’t have to worry about
how to install all the software you need on each different operating system.
More information
================
- Homepage: https://apptainer.org/
[usaurio@drago31010015 ~]$
Procedemos a descargarnos imagen de hello-world desde el hub de singualarity, pero podría construirse una imagen a mano, o descargarse desde otros registros, esto es a modo prueba:
[usuario@drago31010015 ~]$ apptainer pull shub://singularityhub/hello-world
INFO: Downloading shub image
59.8MiB / 59.8MiB [============================================================================================================================================================================] 100 % 8.4 MiB/s 0s
[usuario@drago31010015 ~]$ ls -lh hello-world_latest.sif
-rwxr-x--- 1 usuario centro 44K Sep 11 10:47 hello-world_latest.sif
[usuario@drago31010015 ~]$ apptainer run hello-world_latest.sif
INFO: gocryptfs not found, will not be able to use gocryptfs
Tacotacotaco
[usuario@drago31010015 ~]$ apptainer exec hello-world_latest.sif cat /etc/os-release
INFO: gocryptfs not found, will not be able to use gocryptfs
NAME="Ubuntu"
VERSION="14.04.6 LTS, Trusty Tahr"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 14.04.6 LTS"
VERSION_ID="14.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
Procedemos a generar un script de slurm tal como se muestra a continuación haciendo uso del módulo apptainer.
# vim myapptainerjob.sh
#!/bin/bash
###################################################################################################################
############################################# DESCRIPCION ########################################################
###################################################################################################################
# Script de Prueba para Apptainer mediante Conda.
###################################################################################################################
################################# CONFIGURACION DE LA SOLICITUD AL CLUSTER #######################################
###################################################################################################################
#SBATCH -p generic # Particion elegida para el trabajo(cola)
#SBATCH -N 1 # Numero de nodods solicitados
#SBATCH -n 1 # Numero de cores(CPUs)
#SBATCH --mem=40G # memoria total solicitada
#SBATCH -t 0-01:00:00 # Duracion solicitada para el trabajo (D-HH:MM:SS)
#SBATCH --job-name=HelloWorld-Apptainer # Nombre del trabajo
#SBATCH -o HellowWord-Apptainer-%j.out # fichero de salida estandart
#SBATCH --mail-type=begin #Envia un correo cuando el trabajo inicia
#SBATCH --mail-type=end #Envia un correo cuando el trabajo finaliza
#SBATCH --mail-user=usuario@csic.es #Direccion de correo a la que enviar los mensajes
###################################################################################################################
################################# DESPLIEGUE DEL SOFTWARE A UTILIZAR #############################################
###################################################################################################################
module load rama0.3
module load apptainer/1.3.2
###################################################################################################################
########################################## COMANDO A EJECUTAR #####################################################
###################################################################################################################
# Fecha/Hora de inicio del trabajo
echo 'Trabajo iniciado en fecha:'
date
# COMANDOS
# Correr un contenedor
#apptainer run lolcow_latest.sif > salida.txt
# ejecutar una instruccion dentro de un contenedor
#apptainer exec lolcow_latest.sif cowsay moo > salida.txt
apptainer exec hello-world_latest.sif cat /etc/os-release > salida.txt
# Fecha/Hora de fin del trabajo
echo 'Trabajo finalizado en fecha:'
date
A continuación procedemos a ejectuar script con sbatch.
[usuario@drago31010015 ~]$ sbatch myapptainerjob.sh
[usuario@drago31010015 ~]$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
578039 generic HelloWor usuario PD 0:00 1 (Priority)