Compiling Yambo on Perlmutter with GPU

[milesj]

Hi all,

I'm trying to compile Yambo on the Perlmutter supercomputer cluster with GPU acceleration. My configuration and make steps seem to complete, but when I call p2y I get the following error:

Code: Select all

`MPIDI_CRAY_init: GPU_SUPPORT_ENABLED is requested, but GTL library is not linked`

From advice on this website, I tried adding the flag -target-accel=nvidia80 to my configure call, but it was not recognized. I also tried adding CRAY_ACCEL_TARGET=nvidia80, and it compiled, but the error above still happened.

Here's my configuration call:

Code: Select all

./configure CRAY_ACCEL_TARGET=nvidia80 FC=nvfortran F77=nvfortran CC=nvc CPP="gcc -E -P" FPP="gfortran -E -P -cpp" --enable-open-mp --enable-par-linalg --enable-mpi --enable-hdf5-par-io --enable-slepc-linalg --enable-cuda="cuda11.7,cc70" --enable-time-profile --enable-memory-profile

I've also attached my config.log and compile_p2y.log files below. Perhaps I need to add the CRAY_ACCEL_TARGET variable somewhere after compilation? I'm not really sure what else I can do, but I also don't know a ton about compilers and gpus.

Thanks for any help!
-Miles

[Nicola Spallanzani]

Dear Miles,
the variable CRAY_ACCEL_TARGET=nvidia80 shouldn't be needed. The only flag that you need is --enable-cuda="cuda11.7,cc70".
However in this flag you are selecting the wrong compute capablity (cc), Perlmutter is mounting Nvidia A100 gpus with cc80.
So the right flag is: --enable-cuda="cuda11.7,cc80".

Best,
Nicola

[milesj]

Hi Nicola,

Unfortunately changing it to cc80 did not help. It also seems I need the flags 'FC=nvfortran F77=nvfortran CC=nvc CPP="gcc -E -P" FPP="gfortran -E -P -cpp" ' for the configure step to work.

I've also tried adding 'MPICC="cc -target-accel=nvidia80 -shared" ' to my configure call, which didn't change anything, and I've tried adding -target-accel=nvidia80 to various flags in ./config/setup, which either did not change anything or caused an error in the make step.

Let me know if you have any other ideas! Is there anything special I need to do when calling p2y to avoid this error?

thanks,
miles

[Nicola Spallanzani]

Please try this configure starting from scratch:

Code: Select all

./configure FC=nvfortran F77=nvfortran CC=nvc CPP="cpp -E" FPP="nvfortran -Mpreprocess -E" \
    --enable-open-mp --enable-par-linalg --enable-mpi --enable-hdf5-par-io \
    --enable-slepc-linalg --enable-cuda="cuda11.7,cc80" --enable-time-profile --enable-memory-profile

From scratch I mean deleting the directory of yambo and unpacking a new one.

Best,
Nicola

[milesj]

Hi Nicola,

Unfortunately the same error ocurred:

`
MPICH ERROR [Rank 0] [job id ] [Fri Oct 13 13:14:49 2023] [nid001645] - Abort(-1) (rank 0 in comm 0): MPIDI_CRAY_init: GPU_SUPPORT_ENABLED is requested, but GTL library is not linked
(Other MPI error)

aborting job:
MPIDI_CRAY_init: GPU_SUPPORT_ENABLED is requested, but GTL library is not linked
`

I've attached my config.log file, but unfortunately the compile_p2y.log file is too large to attach here.

Thanks,
Miles

[milesj]

For anyone else trying, here's a procedure that ended up working for me:

Code: Select all

login

export LC_ALL=C
module load gpu ; module swap PrgEnv-gnu PrgEnv-nvidia ; module unload darshan ; module unload cray-libsci ; module load cray-fftw ; module load cray-hdf5-parallel

mkdir yambo
cd yambo/
git clone https://github.com/yambo-code/yambo.git ./


./configure FC=nvfortran  F77=nvfortran  CC=nvc CPP="cpp -E" FPP="nvfortran -Mpreprocess -E" --enable-cuda="cuda11.7,cc80" --enable-time-profile --enable-memory-profile --with-hdf5-path=/opt/cray/pe/hdf5-parallel/1.12.2.3/nvidia/20.7 --with-blas-libs=/opt/nvidia/hpc_sdk/Linux_x86_64/22.7/compilers/lib/libblas.so --with-lapack-libs="/opt/nvidia/hpc_sdk/Linux_x86_64/22.7/compilers/lib/liblapack.so /opt/nvidia/hpc_sdk/Linux_x86_64/22.7/compilers/lib/libblas.so" --with-fft-path=/opt/cray/pe/fftw/3.3.10.3/x86_milan --with-libxc-path=/global/common/software/nersc/pm-2021q4/sw/libxc/v5.2.2/alv-gpu MPIFC='ftn' MPIF77='ftn' MPICC='cc' MPICXX='CC' --enable-par-linalg --enable-hdf5-par-io --enable-open-mp

(then, in ./config/setup:
add -I/opt/nvidia/hpc_sdk/Linux_x86_64/22.5/cuda/11.7/include -acc -gpu=cc80,cuda11.7  -target-accel=nvidia80 to cflags
add  -target-accel=nvidia80 to fcflags and fcuflags)

As with the other post on compiling yambo for cpu, there seem to be weird errors incorporating slepc. Unfortunately this procedure only seemed to work for the newest yambo distribution from github - I tried applying it to yambo-5.1.0 or copying their slepc configurations over, but was not able to get it to work - there was always either a slepc-related compilation error or a cuda-related compilation error.

[Nicola Spallanzani]

Dear Miles,
thank you very much for your help. Have you tried to compile also the version 5.1.2?

Best,
Nicola