Band gap convergence
Moderators: Davide Sangalli, andrea.ferretti, myrta gruning, andrea marini, Daniele Varsano
-
- Posts: 88
- Joined: Sun Apr 11, 2021 3:02 pm
Band gap convergence
Hello,
I am converging the empty bands for calculating the GW correction to the band gap. I have done this convergence study for different models obtaining convergence almost always when having >140/200 empty bands (using the BG terminator). However, a particular model is giving me problems when converging the bands (it seem to converge quite slowly). I have converged the response block size and bands at the same time, and at fixed NGsBlkXp the energy at different bands goes like this
100 4,22
150 3,7018
200 3,7112
250 3,7096
300 3,7128
350 3,7172
450 3,7172
The system is particularly big and I've also seen that it converges very slowly with k-point sampling so I need to make a compromise between the feasibility of the calculation and accuracy. In fact, considering 350 bands would request a lot of memory and I'm not sure I could finish the calculation (I've finished the calculation with 200 bands). What value could be safe to take to calculate the GW corrections?
I am converging the empty bands for calculating the GW correction to the band gap. I have done this convergence study for different models obtaining convergence almost always when having >140/200 empty bands (using the BG terminator). However, a particular model is giving me problems when converging the bands (it seem to converge quite slowly). I have converged the response block size and bands at the same time, and at fixed NGsBlkXp the energy at different bands goes like this
100 4,22
150 3,7018
200 3,7112
250 3,7096
300 3,7128
350 3,7172
450 3,7172
The system is particularly big and I've also seen that it converges very slowly with k-point sampling so I need to make a compromise between the feasibility of the calculation and accuracy. In fact, considering 350 bands would request a lot of memory and I'm not sure I could finish the calculation (I've finished the calculation with 200 bands). What value could be safe to take to calculate the GW corrections?
Laura Caputo
Ph.D. Student
Université Catholique de Louvain
https://uclouvain.be/fr/repertoires/laura.caputo
Ph.D. Student
Université Catholique de Louvain
https://uclouvain.be/fr/repertoires/laura.caputo
- Daniele Varsano
- Posts: 3875
- Joined: Tue Mar 17, 2009 2:23 pm
- Contact:
Re: Band gap convergence
Dear Laura,
from the number you posted, it seems to me that 200 bands are enough to have a converged gap with more than reasonable accuracy.
If you have memory problems (I presume in the calculation of the screening) these can be mitigated by choosing an optimal parallelization strategy for memory distribution, ue assigning CPU on the bands only balancing the number of CPU in valence and conduction bands according to their proportion (Nvalnce/cpu_v \sim Nconduct/cpu_c)
Best,
Daniele
from the number you posted, it seems to me that 200 bands are enough to have a converged gap with more than reasonable accuracy.
If you have memory problems (I presume in the calculation of the screening) these can be mitigated by choosing an optimal parallelization strategy for memory distribution, ue assigning CPU on the bands only balancing the number of CPU in valence and conduction bands according to their proportion (Nvalnce/cpu_v \sim Nconduct/cpu_c)
Best,
Daniele
Dr. Daniele Varsano
S3-CNR Institute of Nanoscience and MaX Center, Italy
MaX - Materials design at the Exascale
http://www.nano.cnr.it
http://www.max-centre.eu/
S3-CNR Institute of Nanoscience and MaX Center, Italy
MaX - Materials design at the Exascale
http://www.nano.cnr.it
http://www.max-centre.eu/
-
- Posts: 88
- Joined: Sun Apr 11, 2021 3:02 pm
Re: Band gap convergence
Dear Daniele,
Thanks. I had the doubt about the convergence since when I plotted the lines, 200 bands didn't seem converged but looking at the numbers directly could be easier.
As said, I've calculated the gap with 200 bands, without having too many memory problems. However, as you said, I have these kind of problems when calculating the screening with higher parameters. Just to fully understand your suggestion, in my input files I usually select something like:
PAR_def_mode= "balanced" # [PARALLEL] Default distribution mode ("balanced"/"memory"/"workload")
X_and_IO_CPU= "1 10 12" # [PARALLEL] CPUs for each role
X_and_IO_ROLEs= "k c v" # [PARALLEL] CPUs roles (q,g,k,c,v)
X_and_IO_nCPU_LinAlg_INV=-1 # [PARALLEL] CPUs for Linear Algebra (if -1 it is automatically set)
X_Threads=1 # [OPENMP/X] Number of threads for response functions
DIP_CPU= "1 10 12" # [PARALLEL] CPUs for each role
DIP_ROLEs= "k c v" # [PARALLEL] CPUs roles (k,c,v)
DIP_Threads=1 # [OPENMP/X] Number of threads for dipoles
SE_CPU= "1 1 97" # [PARALLEL] CPUs for each role
SE_ROLEs= "q qp b" # [PARALLEL] CPUs roles (q,qp,b)
for OpenMP/MPI calculation, depending on how demanding the calculation is. If I understood correctly, I should modify the PARALLEL options depending on how many filled/empty bands I have in proportion, right?
Thanks. I had the doubt about the convergence since when I plotted the lines, 200 bands didn't seem converged but looking at the numbers directly could be easier.
As said, I've calculated the gap with 200 bands, without having too many memory problems. However, as you said, I have these kind of problems when calculating the screening with higher parameters. Just to fully understand your suggestion, in my input files I usually select something like:
PAR_def_mode= "balanced" # [PARALLEL] Default distribution mode ("balanced"/"memory"/"workload")
X_and_IO_CPU= "1 10 12" # [PARALLEL] CPUs for each role
X_and_IO_ROLEs= "k c v" # [PARALLEL] CPUs roles (q,g,k,c,v)
X_and_IO_nCPU_LinAlg_INV=-1 # [PARALLEL] CPUs for Linear Algebra (if -1 it is automatically set)
X_Threads=1 # [OPENMP/X] Number of threads for response functions
DIP_CPU= "1 10 12" # [PARALLEL] CPUs for each role
DIP_ROLEs= "k c v" # [PARALLEL] CPUs roles (k,c,v)
DIP_Threads=1 # [OPENMP/X] Number of threads for dipoles
SE_CPU= "1 1 97" # [PARALLEL] CPUs for each role
SE_ROLEs= "q qp b" # [PARALLEL] CPUs roles (q,qp,b)
for OpenMP/MPI calculation, depending on how demanding the calculation is. If I understood correctly, I should modify the PARALLEL options depending on how many filled/empty bands I have in proportion, right?
Laura Caputo
Ph.D. Student
Université Catholique de Louvain
https://uclouvain.be/fr/repertoires/laura.caputo
Ph.D. Student
Université Catholique de Louvain
https://uclouvain.be/fr/repertoires/laura.caputo
- Daniele Varsano
- Posts: 3875
- Joined: Tue Mar 17, 2009 2:23 pm
- Contact:
Re: Band gap convergence
Dear Laura,
these are the parameter you should change depending on the filled/empty bands proportion.
and it is desirable to assign more CPUs to the conduction as they are much numerous.
Best,
Daniele
Yes,If I understood correctly, I should modify the PARALLEL options depending on how many filled/empty bands I have in proportion, right?
these are the parameter you should change depending on the filled/empty bands proportion.
Code: Select all
X_and_IO_CPU= "1 10 12"
X_and_IO_ROLEs= "k c v"
Best,
Daniele
Dr. Daniele Varsano
S3-CNR Institute of Nanoscience and MaX Center, Italy
MaX - Materials design at the Exascale
http://www.nano.cnr.it
http://www.max-centre.eu/
S3-CNR Institute of Nanoscience and MaX Center, Italy
MaX - Materials design at the Exascale
http://www.nano.cnr.it
http://www.max-centre.eu/
-
- Posts: 88
- Joined: Sun Apr 11, 2021 3:02 pm
Re: Band gap convergence
Dear Daniele,
After your suggestion I have been trying to run Yambo in parallel with different options, for example:
X_and_IO_CPU= "1 6 4" # [PARALLEL] CPUs for each role
X_and_IO_ROLEs= "k c v" # [PARALLEL] CPUs roles (q,g,k,c,v)
X_and_IO_nCPU_LinAlg_INV=-1 # [PARALLEL] CPUs for Linear Algebra (if -1 it is automatically set)
X_Threads=12 # [OPENMP/X] Number of threads for response functions
DIP_CPU= "1 6 4" # [PARALLEL] CPUs for each role
DIP_ROLEs= "k c v" # [PARALLEL] CPUs roles (k,c,v)
DIP_Threads=12 # [OPENMP/X] Number of threads for dipoles
SE_CPU= "1 1 24" # [PARALLEL] CPUs for each role
SE_ROLEs= "q qp b" # [PARALLEL] CPUs roles (q,qp,b)
SE_Threads=12 # [OPENMP/GW] Number of threads for self-energy
and in the file generated in the LOG, there isn't 'switching to default' so I guess the parallelization was working. However, to do each step, it takes more than 10h... am I doing something incorrectly or is just the calculation very demanding? On the cluster that I'm using, I can't use more than 24CPU but it seems that using more threads doesn't speed up the calculation a lot, is it true?
After your suggestion I have been trying to run Yambo in parallel with different options, for example:
X_and_IO_CPU= "1 6 4" # [PARALLEL] CPUs for each role
X_and_IO_ROLEs= "k c v" # [PARALLEL] CPUs roles (q,g,k,c,v)
X_and_IO_nCPU_LinAlg_INV=-1 # [PARALLEL] CPUs for Linear Algebra (if -1 it is automatically set)
X_Threads=12 # [OPENMP/X] Number of threads for response functions
DIP_CPU= "1 6 4" # [PARALLEL] CPUs for each role
DIP_ROLEs= "k c v" # [PARALLEL] CPUs roles (k,c,v)
DIP_Threads=12 # [OPENMP/X] Number of threads for dipoles
SE_CPU= "1 1 24" # [PARALLEL] CPUs for each role
SE_ROLEs= "q qp b" # [PARALLEL] CPUs roles (q,qp,b)
SE_Threads=12 # [OPENMP/GW] Number of threads for self-energy
and in the file generated in the LOG, there isn't 'switching to default' so I guess the parallelization was working. However, to do each step, it takes more than 10h... am I doing something incorrectly or is just the calculation very demanding? On the cluster that I'm using, I can't use more than 24CPU but it seems that using more threads doesn't speed up the calculation a lot, is it true?
Laura Caputo
Ph.D. Student
Université Catholique de Louvain
https://uclouvain.be/fr/repertoires/laura.caputo
Ph.D. Student
Université Catholique de Louvain
https://uclouvain.be/fr/repertoires/laura.caputo
- Daniele Varsano
- Posts: 3875
- Joined: Tue Mar 17, 2009 2:23 pm
- Contact:
Re: Band gap convergence
Dear Laura,
You can use threads, but I would not use more than 8/16, as you are using already 24 MPI I do not know how much would be performant the hyperthreading.
In order to improve memory distribution, you can probably put more CPU on conduction and reduce the valence ones. Note this will distribute the memory in an efficient wat but it is not
guaranteed that it is the most performant strategy.
Best,
Daniele
You can use threads, but I would not use more than 8/16, as you are using already 24 MPI I do not know how much would be performant the hyperthreading.
In order to improve memory distribution, you can probably put more CPU on conduction and reduce the valence ones. Note this will distribute the memory in an efficient wat but it is not
guaranteed that it is the most performant strategy.
Best,
Daniele
Dr. Daniele Varsano
S3-CNR Institute of Nanoscience and MaX Center, Italy
MaX - Materials design at the Exascale
http://www.nano.cnr.it
http://www.max-centre.eu/
S3-CNR Institute of Nanoscience and MaX Center, Italy
MaX - Materials design at the Exascale
http://www.nano.cnr.it
http://www.max-centre.eu/
-
- Posts: 88
- Joined: Sun Apr 11, 2021 3:02 pm
Re: Band gap convergence
Dear Daniele,
Thanks for the suggestion. So, if also in this case the calculation is slow, the only way for speeding it up is to use more MPI, right?
Thanks for the suggestion. So, if also in this case the calculation is slow, the only way for speeding it up is to use more MPI, right?
Laura Caputo
Ph.D. Student
Université Catholique de Louvain
https://uclouvain.be/fr/repertoires/laura.caputo
Ph.D. Student
Université Catholique de Louvain
https://uclouvain.be/fr/repertoires/laura.caputo
- Daniele Varsano
- Posts: 3875
- Joined: Tue Mar 17, 2009 2:23 pm
- Contact:
Re: Band gap convergence
Dear Laura,
Using more MPI of course it will improve the speed of the calculation. Another way is to see if there are parameters you can reduce without losing accuracy.
Finally, it also depends on which part of the code is the more time consuming (computation of X0, X, dipoles, GW summation etc...).
Best,
Daniele
Using more MPI of course it will improve the speed of the calculation. Another way is to see if there are parameters you can reduce without losing accuracy.
Finally, it also depends on which part of the code is the more time consuming (computation of X0, X, dipoles, GW summation etc...).
Best,
Daniele
Dr. Daniele Varsano
S3-CNR Institute of Nanoscience and MaX Center, Italy
MaX - Materials design at the Exascale
http://www.nano.cnr.it
http://www.max-centre.eu/
S3-CNR Institute of Nanoscience and MaX Center, Italy
MaX - Materials design at the Exascale
http://www.nano.cnr.it
http://www.max-centre.eu/
-
- Posts: 88
- Joined: Sun Apr 11, 2021 3:02 pm
Re: Band gap convergence
Dear Daniele,
Thank you very much for the clarification.
Have a nice day,
Laura
Thank you very much for the clarification.
Have a nice day,
Laura
Laura Caputo
Ph.D. Student
Université Catholique de Louvain
https://uclouvain.be/fr/repertoires/laura.caputo
Ph.D. Student
Université Catholique de Louvain
https://uclouvain.be/fr/repertoires/laura.caputo
-
- Posts: 88
- Joined: Sun Apr 11, 2021 3:02 pm
Re: Band gap convergence
Dear Daniele,
Reviewing my calculations, I've noticed that I have converged the band gap using Gamma point instead of the point where the direct band gap is. Could the convergence be strongly affected by this?
Reviewing my calculations, I've noticed that I have converged the band gap using Gamma point instead of the point where the direct band gap is. Could the convergence be strongly affected by this?
Laura Caputo
Ph.D. Student
Université Catholique de Louvain
https://uclouvain.be/fr/repertoires/laura.caputo
Ph.D. Student
Université Catholique de Louvain
https://uclouvain.be/fr/repertoires/laura.caputo