Page 1 of 1
Compiling Yambo on Perlmutter with GPU
Posted: Fri Oct 13, 2023 12:19 am
by milesj
Hi all,
I'm trying to compile Yambo on the Perlmutter supercomputer cluster with GPU acceleration. My configuration and make steps seem to complete, but when I call p2y I get the following error:
Code: Select all
`MPIDI_CRAY_init: GPU_SUPPORT_ENABLED is requested, but GTL library is not linked`
From advice on
this website, I tried adding the flag -target-accel=nvidia80 to my configure call, but it was not recognized. I also tried adding CRAY_ACCEL_TARGET=nvidia80, and it compiled, but the error above still happened.
Here's my configuration call:
Code: Select all
./configure CRAY_ACCEL_TARGET=nvidia80 FC=nvfortran F77=nvfortran CC=nvc CPP="gcc -E -P" FPP="gfortran -E -P -cpp" --enable-open-mp --enable-par-linalg --enable-mpi --enable-hdf5-par-io --enable-slepc-linalg --enable-cuda="cuda11.7,cc70" --enable-time-profile --enable-memory-profile
I've also attached my config.log and compile_p2y.log files below. Perhaps I need to add the CRAY_ACCEL_TARGET variable somewhere after compilation? I'm not really sure what else I can do, but I also don't know a ton about compilers and gpus.
Thanks for any help!
-Miles
Re: Compiling Yambo on Perlmutter with GPU
Posted: Fri Oct 13, 2023 7:36 am
by Nicola Spallanzani
Dear Miles,
the variable CRAY_ACCEL_TARGET=nvidia80 shouldn't be needed. The only flag that you need is --enable-cuda="cuda11.7,cc70".
However in this flag you are selecting the wrong compute capablity (cc), Perlmutter is mounting Nvidia A100 gpus with cc80.
So the right flag is: --enable-cuda="cuda11.7,cc80".
Best,
Nicola
Re: Compiling Yambo on Perlmutter with GPU
Posted: Fri Oct 13, 2023 9:30 am
by milesj
Hi Nicola,
Unfortunately changing it to cc80 did not help. It also seems I need the flags 'FC=nvfortran F77=nvfortran CC=nvc CPP="gcc -E -P" FPP="gfortran -E -P -cpp" ' for the configure step to work.
I've also tried adding 'MPICC="cc -target-accel=nvidia80 -shared" ' to my configure call, which didn't change anything, and I've tried adding -target-accel=nvidia80 to various flags in ./config/setup, which either did not change anything or caused an error in the make step.
Let me know if you have any other ideas! Is there anything special I need to do when calling p2y to avoid this error?
thanks,
miles
Re: Compiling Yambo on Perlmutter with GPU
Posted: Fri Oct 13, 2023 9:56 am
by Nicola Spallanzani
Please try this configure starting from scratch:
Code: Select all
./configure FC=nvfortran F77=nvfortran CC=nvc CPP="cpp -E" FPP="nvfortran -Mpreprocess -E" \
--enable-open-mp --enable-par-linalg --enable-mpi --enable-hdf5-par-io \
--enable-slepc-linalg --enable-cuda="cuda11.7,cc80" --enable-time-profile --enable-memory-profile
From scratch I mean deleting the directory of yambo and unpacking a new one.
Best,
Nicola
Re: Compiling Yambo on Perlmutter with GPU
Posted: Sun Oct 15, 2023 8:12 pm
by milesj
Hi Nicola,
Unfortunately the same error ocurred:
`
MPICH ERROR [Rank 0] [job id ] [Fri Oct 13 13:14:49 2023] [nid001645] - Abort(-1) (rank 0 in comm 0): MPIDI_CRAY_init: GPU_SUPPORT_ENABLED is requested, but GTL library is not linked
(Other MPI error)
aborting job:
MPIDI_CRAY_init: GPU_SUPPORT_ENABLED is requested, but GTL library is not linked
`
I've attached my config.log file, but unfortunately the compile_p2y.log file is too large to attach here.
Thanks,
Miles
Re: Compiling Yambo on Perlmutter with GPU
Posted: Wed Nov 15, 2023 11:04 pm
by milesj
For anyone else trying, here's a procedure that ended up working for me:
Code: Select all
login
export LC_ALL=C
module load gpu ; module swap PrgEnv-gnu PrgEnv-nvidia ; module unload darshan ; module unload cray-libsci ; module load cray-fftw ; module load cray-hdf5-parallel
mkdir yambo
cd yambo/
git clone https://github.com/yambo-code/yambo.git ./
./configure FC=nvfortran F77=nvfortran CC=nvc CPP="cpp -E" FPP="nvfortran -Mpreprocess -E" --enable-cuda="cuda11.7,cc80" --enable-time-profile --enable-memory-profile --with-hdf5-path=/opt/cray/pe/hdf5-parallel/1.12.2.3/nvidia/20.7 --with-blas-libs=/opt/nvidia/hpc_sdk/Linux_x86_64/22.7/compilers/lib/libblas.so --with-lapack-libs="/opt/nvidia/hpc_sdk/Linux_x86_64/22.7/compilers/lib/liblapack.so /opt/nvidia/hpc_sdk/Linux_x86_64/22.7/compilers/lib/libblas.so" --with-fft-path=/opt/cray/pe/fftw/3.3.10.3/x86_milan --with-libxc-path=/global/common/software/nersc/pm-2021q4/sw/libxc/v5.2.2/alv-gpu MPIFC='ftn' MPIF77='ftn' MPICC='cc' MPICXX='CC' --enable-par-linalg --enable-hdf5-par-io --enable-open-mp
(then, in ./config/setup:
add -I/opt/nvidia/hpc_sdk/Linux_x86_64/22.5/cuda/11.7/include -acc -gpu=cc80,cuda11.7 -target-accel=nvidia80 to cflags
add -target-accel=nvidia80 to fcflags and fcuflags)
As with the other post on compiling yambo for cpu, there seem to be weird errors incorporating slepc. Unfortunately this procedure only seemed to work for the newest yambo distribution from github - I tried applying it to yambo-5.1.0 or copying their slepc configurations over, but was not able to get it to work - there was always either a slepc-related compilation error or a cuda-related compilation error.
Re: Compiling Yambo on Perlmutter with GPU
Posted: Thu Nov 16, 2023 12:17 pm
by Nicola Spallanzani
Dear Miles,
thank you very much for your help. Have you tried to compile also the version 5.1.2?
Best,
Nicola