Run Yambo initialization, but stuck there

[hongtang2023]

Dear Yambo team,

I used QE calculated a metallic system with about 18,000 k points. The system.save folder is 12T size data.
Then I used p2y to convert to SAVE folder, which is 5.7T data.

question 1, how can I know the conversion to the SAVE folder is complete and correct?
I checked the file in the system.save/LOG folder, they all finished with " == P2Y completed ==".
Does this means the conversion is ok?


question2, when I run yambo initialization, it was stuck and show the screen below.


Please help me!


Best,
Hong



"

__ __ ____ ___ ___ ____ ___
| | |/ | | | \ / \
| | | o | _ _ | o ) |
| ~ | | \_/ | | O |
|___, | _ | | | O | |
| | | | | | | |
|____/|__|__|___|___|_____|\___/


<07s> [01] MPI/OPENMP structure, Files & I/O Directories
<07s> MPI Cores-Threads : 1(CPU)-2(threads)
<07s> [02] CORE Variables Setup
<07s> [02.01] Unit cells
<07s> [02.02] Symmetries
<07s> [02.03] Reciprocal space
<07s> [02.04] K-grid lattice
<10s> Grid dimensions : 30 48 48
<10s> [02.05] Energies & Occupations
"
then suck here.

[Daniele Varsano]

Dear Hong,

I checked the file in the system.save/LOG folder, they all finished with " == P2Y completed ==".
Does this mean the conversion is ok?

Yes, this means that the calculation was successful!

question2, when I run yambo initialization, it was stuck and show the screen below.

Probably 18k k-points is too much and it needs a lot of time to build the k points indexing.

Are you sure you need such large grids?
You can have a look to this recent paper for a way to speed up calculations for metals:

Efficient full frequency GW for metals using a multipole approach for the dielectric screening
Dario A. Leon, Andrea Ferretti, Daniele Varsano, Elisa Molinari, and Claudia Cardoso
Phys. Rev. B 107, 155130 2023
https://journals.aps.org/prb/abstract/1 ... 107.155130

Best,
Daniele

[hongtang2023]

Dear Daniele,

Thank you so much for helping!
I think I need to do with many kpoints to converge the calculations.

I tried to run
yambo -fatlog

and wait for over 1 hr. But get the following error " invalid pointer
Aborted "


Please help me why " invalid pointer".



Thank you very much!

Best,
Hong





____ ____ ___ .___ ___. .______ ______
\ \ / / / \ | \/ | | _ \ / __ \
\ \/ / / ^ \ | \ / | | |_) | | | | |
\_ _/ / /_\ \ | |\/| | | _ < | | | |
| | / _____ \ | | | | | |_) | | `--" |
|__| /__/ \__\ |__| |__| |______/ \______/



<06s> [01] MPI/OPENMP structure, Files & I/O Directories
<06s> MPI Cores-Threads : 1(CPU)-2(threads)
<06s> [02] CORE Variables Setup
<06s> [02.01] Unit cells
<07s> [02.02] Symmetries
<07s> [02.03] Reciprocal space
<07s> [02.04] K-grid lattice
<09s> Grid dimensions : 30 48 48
<09s> [02.05] Energies & Occupations
<01h-12m> [WARNING][X] Metallic system
<01h-20m> [03] Transferred momenta grid and indexingmunmap_chunk(): invalid pointer
Aborted

[Daniele Varsano]

Dear Hong,

I have the impression that a 18k k points is too large to be managed by the standard algorithms. I really suggest you to reduce your k point sampling and check if you really need such a large grid that it is rather unusual.
Some years ago the inizialization was adapted to deal with large grids, I do not know the status of that branch, I will ask to the developers and let you know.

Best,
Daniele

[claudio]

Dear Hong

maybe you can use a small number of k-points and a double-grid for speed up convergence,

have a look here:

http://www.attaccalite.com/speed-up-die ... ith-yambo/

the same approach works with BSE with Haydock or Inversion solvers

best
Claudio

[hongtang2023]

Dear Daniele and Claudio,

Thank you so much for helping and great information!

As I understand it, some questions in my mind.

1. the double grid approach is great for converging the dielectric function. As in your example, 2-dimensional hexagonal boron nitride, in the step 6, "Now repeat the dielectric constant calculation of point 1. " Is this point 1 the first q point in the coarse grid? or in the fine grid?

The coarse grid is not shifted, I think, and the fine grid is shifted. But for the q grid, both for coarse and fine grid, there is nothing about " shifted or not". So the first q points in coarse and fine grids are (0, 0, 0). However, the second q points in the coarse and fine grid are different.

2. My question is : can we calculate the dielectric function (or EELS) for any q points (non zero q points) in the fine grid? not just in the coarse grid? with this double grid approach.


3. the reason I want to do the above 2 is that I want to get the plasmon dispersion curve, omega_pl vs. q.
So, I want to calculate the EELS at different q values. The system I am working on has a wired plasmon dispersion in experiment. I want to study it with calculation method. I want to make sure the dispersion curve, omega_pl vs. q I calculated has a fine enough resolution just around the q=0 area. That is why I want to very fine k (or q) grid.

4. update about I try with Yambo RPA calculation for EELS at q=0 with this fine k grid QE result. As I reported, the Yambo p2y looks ok, since all the files in LOG folder end with " == P2Y completed == ". But Yambo initialization Yambo -fatlog stops with " invalid pointer Aborted ". Interestingly, the Yambo RPA calculation for EELS at q=0 stops and aborts with "munmap_chunk(): invalid pointer ". the log file stops with "
...
<37s> P1-nid004216: [02.04] K-grid lattice
<38s> P1-nid004216: Grid dimensions : 30 48 48
<39s> P1-nid004216: [02.05] Energies & Occupations
<01h-30m> P1-nid004216: [WARNING][X] Metallic system
<01h-42m> P1-nid004216: [03] Transferred momenta grid and indexing
"

It looks to me the Yambo RPA calculation and the Yambo initialization have the same errors.



Please help!

Best,
Hong

[claudio]

Dear Hong

> 1. the double grid approach is great for converging the dielectric function. As in your example, 2-dimensional hexagonal boron nitride, in the step 6, "Now repeat the dielectric constant calculation of point 1. " Is this point 1 the first q point in the coarse grid? or in the fine grid?

I mean point 1 of this tutorial, namely you calculate again the dielectric constant but this time Yambo will use the double-grid to average the denominators
and result will be better

> 2. My question is : can we calculate the dielectric function (or EELS) for any q points (non zero q points) in the fine grid? not just in the coarse grid? with this double grid approach.

no, just on the coarse grid, but the idea is that you can find a compromise. Try to use the largest possible coarse grid with yambo and in addition include a double-grid
to speed up convergence of the spectra

> 3. the reason I want to do the above 2 is that I want to get the plasmon dispersion curve, omega_pl vs. q.
So, I want to calculate the EELS at different q values. The system I am working on has a wired plasmon dispersion in experiment. I want to study it with calculation method. I want to make sure the dispersion curve, omega_pl vs. q I calculated has a fine enough resolution just around the q=0 area. That is why I want to very fine k (or q) grid.

I understand, try to approach slowly to the maximum you can do with Yambo, you can also check if it is a memory problem and increase the memory
in your calculation using threads etc..

>4. update about I try with Yambo RPA calculation for EELS at q=0 with this fine k grid QE result. As I reported, the Yambo p2y looks ok, since all the files in LOG folder end with " == P2Y completed == ". But Yambo initialization >Yambo -fatlog stops with " invalid pointer Aborted ". Interestingly, the Yambo RPA calculation for EELS at q=0 stops and aborts with "munmap_chunk(): invalid pointer ". the log file stops with "

try with a slightly smaller grid, and let us know
Claudio

[hongtang2023]

Dear Claudio and Daniele,

Thank you so much for helping!
I guess I want to stick with my current QE results for the Yambo calculations. The current QE results cost me a huge computational time on NERSC cluster. and The current k grid in the current QE results is still not the finest one I want, it is already a compromised k grid 48x48x30. (The ideal k grid should be even finer than this.)

Please let me know how I can let the Yambo RPA optical calculation done with this QE results. The system has 24 atoms with 2 kinds of atoms, and 620 bands and 17666 irreducible k points.
In the last try, I used 2048 node on NERSC's Perlmutter cluster, but still got errors below, please also see attached submit script for this run. Are the errors due to memory issue? or intrinsic Yambo code issue?
Please help me find the issues and how to get the calculation done.

Errors:
munmap_chunk(): invalid pointer
malloc(): memory corruption (fast)
munmap_chunk(): invalid pointer
munmap_chunk(): invalid pointer
munmap_chunk(): invalid pointer
munmap_chunk(): invalid pointer
munmap_chunk(): invalid pointer
munmap_chunk(): invalid pointer
munmap_chunk(): invalid pointer
munmap_chunk(): invalid pointer
munmap_chunk(): invalid pointer
munmap_chunk(): invalid pointer
munmap_chunk(): invalid pointer
munmap_chunk(): invalid pointer
munmap_chunk(): invalid pointer
munmap_chunk(): invalid pointer
munmap_chunk(): invalid pointer
munmap_chunk(): invalid pointer
mremap_chunk(): invalid pointer
free(): invalid size
munmap_chunk(): invalid pointer
munmap_chunk(): invalid pointer
free(): invalid size
munmap_chunk(): invalid pointer
munmap_chunk(): invalid pointer
munmap_chunk(): invalid pointer
munmap_chunk(): invalid pointer
munmap_chunk(): invalid pointer
munmap_chunk(): invalid pointer
corrupted double-linked list
munmap_chunk(): invalid pointer
munmap_chunk(): invalid pointer
munmap_chunk(): invalid pointer
munmap_chunk(): invalid pointer
munmap_chunk(): invalid pointer
munmap_chunk(): invalid pointer
free(): invalid size
srun: error: nid006108: task 2702: Segmentation fault
srun: Terminating StepId=19357625.0
munmap_chunk(): invalid pointer
munmap_chunk(): invalid pointer
munmap_chunk(): invalid pointer
munmap_chunk(): invalid pointer
malloc(): memory corruption (fast)
munmap_chunk(): invalid pointer
munmap_chunk(): invalid pointer
munmap_chunk(): invalid pointer
munmap_chunk(): invalid pointer
munmap_chunk(): invalid pointer
munmap_chunk(): invalid pointer
free(): invalid size
munmap_chunk(): invalid pointer
munmap_chunk(): invalid pointer
munmap_chunk(): invalid pointer
free(): invalid size
munmap_chunk(): invalid pointer
munmap_chunk(): invalid pointer
munmap_chunk(): invalid pointer
munmap_chunk(): invalid pointer
...
...
...



Submit script for this run:
#!/bin/bash
#SBATCH -A m3356
#SBATCH -C cpu
#SBATCH --qos=regular
#SBATCH --time=8:00:00
#SBATCH --nodes=2048
## # #SBATCH --ntasks-per-node=16
#SBATCH --mail-type=begin,end,fail
#SBATCH --mail-user=htang5@tulane.edu

#
## # 128 cores/ node on Perlmutter CPU only
## # --ntasks-per-node=16 is the mpi per node
## # -c is 128/(mpi per node) X 2
## # -n is the total mpi
#
##OpenMP settings:
#export OMP_NUM_THREADS=4
#export OMP_PLACES=threads
#export OMP_PROC_BIND=spread




MPIRUN="srun -n 4096 -c 128 --cpu_bind=cores"

# bash energyrange.sh # energyrange.dat shouldbe one column, 2 rows
MPICH_GPU_SUPPORT_ENABLED=0

YAMBO="/global/u2/t/tang2017/YAMBO_Perlmutter/yambo-5.1.1/bin/yambo"


# YAMBO="/global/u2/t/tang2017/YAMBO_mcp07_coding/check_CODE_and_workingon/yambo-5.1.1_try_compile/yambo-5.1.1_mcp07_noSOC/bin/yambo"


$MPIRUN $YAMBO -F run_in_ALDA -J alda_q100 > run_in_ALDA.out


cp run_in_ALDA_no_Drude run_in_ALDA


$MPIRUN $YAMBO -F run_in_ALDA -J alda_q100 > run_in_ALDA.out



sed -i 's/#/%%/g' o-*

[hongtang2023]

Dear Claudio and Daniele,

I contacted NERSC cluster team. They told me the errors seems to be memory related issue. However, the Nersc team is not familar with Yambo code.

Please check my job submission script and my Yambo RPA input file. please help me how to set the parameters to avoid memory issues.

Please help!


Yambo RPA run input file
#
# ____ ____ _ ____ ____ ______ ___
# |_ _||_ _| / \ |_ \ / _||_ _ \ ." `.
# \ \ / / / _ \ | \/ | | |_) | / .-. \
# \ \/ / / ___ \ | |\ /| | | __". | | | |
# _| |_ _/ / \ \_ _| |_\/_| |_ _| |__) |\ `-" /
# |______||____| |____||_____||_____||_______/ `.___."
#
#
#
# Version 5.1.0 Revision 21422 Hash (prev commit) fde6e2a07
# Branch is
# MPI+OpenMP+HDF5_MPI_IO Build
# http://www.yambo-code.org
#
optics # [R] Linear Response optical properties
chi # [R][CHI] Dyson equation for Chi.
tddft # [R][K] Use TDDFT kernel
ElecTemp= 0.344694E-4 eV ## Electronic Temperature
FFTGvecs= 3 Ry # [FFT] Plane-waves
X_Threads=0 # [OPENMP/X] Number of threads for response functions
DIP_Threads=0 # [OPENMP/X] Number of threads for dipoles
Chimod= "Hartree" # [X] IP/Hartree/ALDA/LRC/PF/BSfxc
FxcGRLc= 2 Ry # [TDDFT] XC-kernel RL size
NGsBlkXd= 2 Ry # [Xd] Response block size
% QpntsRXd
1 | 1 | # [Xd] Transferred momenta
%
% BndsRnXd
1 | 530 | # [Xd] Polarization function bands
%
% EnRngeXd
0.000 | 1.300000 | eV # [Xd] Energy range
%
% DmRngeXd
0.100000 | 0.100000 | eV # [Xd] Damping range
%
ETStpsXd=350 # [Xd] Total Energy steps DrudeWXd= ( 0.5 , 0.1 ) eV # [Xd] Drude plasmon
% LongDrXd
1.000000 | 0.000000 | 0.000000 | # [Xd] [cc] Electric Field
%





job submission script
#!/bin/bash
#SBATCH -A mxxx
#SBATCH -C cpu
#SBATCH --qos=regular
#SBATCH --time=8:00:00
#SBATCH --nodes=2048
## # #SBATCH --ntasks-per-node=16
#SBATCH --mail-type=begin,end,fail
#SBATCH --mail-user=xxxxxxxxxxxx

#
## # 128 cores/ node on Perlmutter CPU only
## # --ntasks-per-node=16 is the mpi per node
## # -c is 128/(mpi per node) X 2
## # -n is the total mpi
#
##OpenMP settings:
#export OMP_NUM_THREADS=4
#export OMP_PLACES=threads
#export OMP_PROC_BIND=spread




MPIRUN="srun -n 4096 -c 128 --cpu_bind=cores"

# bash energyrange.sh # energyrange.dat shouldbe one column, 2 rows
MPICH_GPU_SUPPORT_ENABLED=0

YAMBO="/global/u2/t/tang2017/YAMBO_Perlmutter/yambo-5.1.1/bin/yambo"


$MPIRUN $YAMBO -F run_in_ALDA -J alda_q100 > run_in_ALDA.out




sed -i 's/#/%%/g' o-*

[Daniele Varsano]

Dear Hong Tang,

I'm not sure if this is a memory issue, anyway you can distribute the memory load among CPU by setting by hand the parallelization strategy in this way:

Code: Select all

DIP_CPU= "1 Nc Nv"                      # [PARALLEL] CPUs for each role
DIP_ROLEs= "k c v"                    # [PARALLEL] CPUs roles (k,c,v)

X_CPU= "1 1 1 Nc Nv"                        # [PARALLEL] CPUs for each role
X_ROLEs= "q g k c v"                      # [PARALLEL] CPUs roles (q,g,k,c,v)

where Nc and Nv are the MPI task you want to assign. Their product need to be the total number of cpu you are using (4096?).
Anyway, if you post also the report and one of the log file, we can inspect more deeply.

Best,
Daniele