Yambo 5.3 on LEONARDO

[mrefiore]

Dear Developers,

I'm trying to compile Yambo 5.3 on LEONARDO with GPU enabled (I wish to have a local version that I may modify beside the regular module).
I'm loading these modules

Code: Select all

module load nvhpc/23.11 openmpi/4.1.6--nvhpc--23.11 fftw/3.3.10--openmpi--4.1.6--nvhpc--23.11 parallel-netcdf/1.12.3--openmpi--4.1.6--nvhpc--23.11 netcdf-c/4.9.2--openmpi--4.1.6--nvhpc--23.11 netcdf-fortran/4.6.1--openmpi--4.1.6--nvhpc--23.11 netlib-scalapack/2.2.0--openmpi--4.1.6--nvhpc--23.11 petsc/3.20.1--openmpi--4.1.6--nvhpc--23.11-mumps

and running configure as

Code: Select all

./configure FC=nvfortran F77=nvfortran MPIFC=mpif90 MPIF77=mpif90 'CPP=cpp -E' 'FPP=nvfortran -Mpreprocess -E' CC=nvc --with-cuda-runtime=11.8 --with-cuda-cc=80 --enable-open-mp --enable-cuda-fortran --enable-par-linalg --enable-time-profile --enable-memory-profile --with-fft-path=$FFTW_HOME --with-scalapack-libs=$NETLIB_SCALAPACK_LIB/libscalapack.so --with-blacs-libs=$NETLIB_SCALAPACK_LIB/libscalapack.so --with-hdf5-path=$HDF5_HOME --with-netcdf-path=$NETCDF_C_HOME --with-netcdff-path=$NETCDF_FORTRAN_HOME --with-petsc-path=$PETSC_HOME

However, at make, I first notice this

Code: Select all

...
\t[libxc-5.2.3] installation
\t[devicexlib-0.8.5] download devicexlib-0.8.5.tar.gz
\t[devicexlib-0.8.5] configuration
\t[devicexlib-0.8.5] all compilation
\t[devicexlib-0.8.5] installation
chmod: cannot access '/leonardo/pub/userexternal/mrefiore/yambo/lib/external/nvfortran/mpif90/cudaf/lib*/*devXlib*a': No such file or directory
\t[lib/qe_pseudo] qe_pseudo (setup)
\t[lib/qe_pseudo] kind
...

and then finally the compilation exits with this error

Code: Select all

make[1]: *** [/leonardo/pub/userexternal/mrefiore/yambo/config/mk/local/rules.mk:15: mod_gpu.o] Error 2
\t[driver] yambo (setup)
yambo linking failed. Check log/compile_yambo.log
make: *** [config/mk/global/actions/compile_yambo.mk:43: yambo] Error 1

The log I've attached indeed confirms that there are some problems with devxlib:

Code: Select all

NVFORTRAN-F-0004-Unable to open MODULE file devxlib_environment.mod (mod_gpu.f90: 12)
NVFORTRAN/x86-64 Linux 23.11-0: compilation aborted

Can I ask for your kind help in trying to solve this issue? I'm not very familiar with the new configure and this new library.
Thank you very much!!
Best,

Michele

[mrefiore]

Let me further add that the configure of devXlib itself, run during yambo compilation as

Code: Select all

./config/configure --prefix=/leonardo/pub/userexternal/mrefiore/yambo/lib/external/nvfortran/mpif90/cudaf --enable-openmp --enable-cuda-fortran --with-cuda-cc=80 --with-cuda-runtime=11.8 --with-blas-libs=-lblas --with-lapack-libs=-llapack F90=mpif90 MPIF90=mpif90

gives this error

Code: Select all

checking for libcuda from specified libcuda path, dynamic... checking static... not found.
checking for /usr/local/cuda/... no
checking for /usr/local/cuda/include... no
checking for /usr/local/cuda/lib64... no
checking for Fortran flag to compile .f90 files... (cached) none
checking whether Fortran compiler accepts -cuda -gpu=cuda11.8... yes
checking for nvcc... /leonardo/prod/spack/5.2/install/0.21/linux-rhel8-icelake/gcc-8.5.0/nvhpc-23.11-wnrvac5a7nx7cnbtaamktaaqljmgorva/Linux_x86_64/23.11/compilers/bin/nvcc
checking whether nvcc works... yes
checking for cuInit in -lcuda... no
configure: error: in `/leonardo/pub/userexternal/mrefiore/yambo/lib/devxlib/devicexlib-0.8.5':
configure: error: Couldn't find libcuda

and then of course the library cannot be compiled, and yambo also fails.
Why is it not finding libcuda? When configuring yambo it says it's using its internal cuda. Should I use an external cuda installed on the cluster?

Thank you for your help!

Michele

[Nicola Spallanzani]

Dear Michele,
you can simply solve the issue adding a flag in the makefile of the devicexlib installation:

- open the file lib/devxlib/Makefile.loc

- add the flag "--enable-cuda-env-check=no" to the variable CONFFLAGS:

Code: Select all

CONFFLAGS=--prefix=$(LIBPATH) $(devxlib_flgs) --enable-cuda-env-check=no \           
           --with-blas-libs="$(BLAS_LIBS)" --with-lapack-libs="$(LAPACK_LIBS)" \
           F90="$(fc)" MPIF90="$(fc)" 

- start from scratch launching a "make distclean" and then the configure line again.

In addition you can change the software stack using this one:

Code: Select all

module load profile/candidate
module load yambo/5.3.0--hpcx-mpi--2.19--nvhpc--24.5-cuda-12.2 
module load cuda/12.2
module load nvhpc/24.5
module load binutils/2.42

export FC=nvfortran 
export F77=nvfortran
export CC=nvc
export CPP="cpp -E"
export FPP="nvfortran -Mpreprocess -E"
export F90SUFFIX=".f90"
export MPIFC=mpifort
export MPIF77=mpifort
export MPICC=mpicc

./configure \
  --enable-mpi --enable-open-mp \
  --enable-msgs-comps --enable-time-profile --enable-memory-profile \
  --enable-par-linalg \
  --with-blas-libs="$OPENBLAS_LIB/libopenblas.so" \
  --with-lapack-libs="$OPENBLAS_LIB/libopenblas.so" \
  --with-hdf5-path=$HDF5_HOME \
  --with-netcdf-path=$NETCDF_C_HOME \
  --with-netcdff-path=$NETCDF_FORTRAN_HOME \
  --with-fft-path=$FFTW_HOME \
  --with-libxc-path=$LIBXC_HOME \
  --with-fft-libs=$FFTW_LIB/libfftw3.a \
  --with-fft-includedir=$FFTW_INC \
  --with-scalapack-libs=$NETLIB_SCALAPACK_LIB/libscalapack.so \
  --with-blacs-libs=$NETLIB_SCALAPACK_LIB/libscalapack.so \
  --enable-slepc-linalg \
  --with-petsc-path=$PETSC_HOME \
  --with-slepc-path=$SLEPC_HOME \
  --with-devxlib-path=$DEVICEXLIB_HOME \
  --with-cuda-cc=80 --with-cuda-runtime=11.8 \
  --enable-cuda-fortran 

The loading of the yambo module is used to automatically load all the useful dependencies instead of loading them one by one.

Let me know if it works for you.
Best,
Nicola

[mrefiore]

Dear Nicola,

Thank you very much for your help!
Now the internal devicexlib can be compiled without errors! Thank you again.

I still have two very minor issues:

1) using your suggested configure doesn't seem to find the external devicexlib and the internal is compiled (now without errors)
2) there seems to be a problem in the download of iotk:

Code: Select all

*** Unable to download iotk-y1.2.2. Test whether curl or wget is installed and working,
*** if you have direct access to the internet.
*** If not, copy iotk-y1.2.2.tar.gz into extlibs/archive/

gzip: stdin: unexpected end of file
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
gtar: This does not look like a tar archive
gtar: Exiting with failure status due to previous errors
WARNING: I can't seem to be able to run `tar' with the given arguments.
         You may want to install GNU tar or Free paxutils, or check the
         command line arguments. 

Other libraries that must be compiled (for instance the same devicexlib) are normally and correctly downloaded, extracted and compiled. Only iotk seems to have issues. I've temporarily solved it by copying the tar.gz in lib/archive (not extlibs/archive as mentioned in the error log).

Thank you very much once again.
Best,

Michele