advice/help on scalability of yambo (Fermi-CINECA/PRACE)

[perezlist2015]

hi

I am looking into the application(s) procedure(s) for CPU time via the PRACE programme, in order to submit
one to the "project access" option for GW and BSE calculations with yambo (starting from qe)

Since the "project access" description mentions the need to submit scalability data, I thought the team of developers
could be so kind as to provide the most recent scalability data; it would be greatly appreciated

I have seen somewhere a scalability test done on CINECA/Fermi that shows 1024-core as keeping linearity
(sorry but I did not keep track of the website). Will you please let me know whether I am right by assuming that
Fermi/CINECA is probably the best supercomputing center to run yambo in parallel, in light of the existing scalability information?
(the 11th PRACE project access call is open now, and they have restricted it to either CINECA/Fermi or barcelona supercomputing center/mare nostrum)
Probably, you could recommend any other technical reason that makes my assumption right or wrong?
From my point of view it would be as difficult to get the application through in either place (there does not exist any national supercomputing center in Luxembourg)

Another technical question: do you happen to know whether everyone is supposed to do his/her own scalability test via a "preparatory access" (2 month)
application, to test the code, or that is not necessary? I mean would they accept the information I could get from the developers?

thanks a lot for your understanding

regards

Manuel

Dr Manuel Pérez Jigato, Chargé de Recherche
Luxembourg Institute of Science and Technology (LIST)
Materials Research and Technology (MRT)
41 rue du Brill
L-4422 BELVAUX
Grand-Duché de Luxembourg
Tel (+352) 47 02 61 - 584
Fax (+352) 47 02 64
e-mail manuel.perez@list.lu

PS I would be very happy to discuss the possibility of a collaboration, including the ETSF via

[perezlist2015]

hi

I found the website I was mentioning in my previous email message, with the scalability work done at CINECA

http://www.hpc.cineca.it/news/yambo-ena ... 1024-cores

this other details is also important for me: does yambo run on 8192 cores at Fermi?

regards

Manuel

Dr Manuel Pérez Jigato, Chargé de Recherche
Luxembourg Institute of Science and Technology (LIST)
Materials Research and Technology (MRT)
41 rue du Brill
L-4422 BELVAUX
Grand-Duché de Luxembourg
Tel (+352) 47 02 61 - 584
Fax (+352) 47 02 64
e-mail manuel.perez@list.lu

[andrea.ferretti]

Hi Manuel,

the pictures on the yambo website that you mention
http://www.hpc.cineca.it/news/yambo-ena ... 1024-cores
were obtained on a rather small system (a graphene nano ribbon with few tens of atoms) and were already showing some decent scalability up to one whole rack of Fermi BGQ (1024 nodes, 16cores/node).

More recently we have been checking the scalability further on larger systems and the situation seems even to improve, so I would say it safe to state that yambo can exploit 1024*16 cores.
This under the following conditions:
* the scalability above has been obtained by using the current development version of yambo (implementing MPI+OpenMP parallelism), which is going to be released at worst before April (in view of the yambo 2015 CECAM school)
* The OpenMP parallelism is actually very important to run on machines with little memory per core such as BGQ. At the moment we use
not more than 1 or 2 MPI tasks per node (against 16 cores/node on BGQ), plus 8-64 OpenMP threads per task (64 threads per node on BGQ is a reasonable maximum value)
* since yambo performs a lot of computation of independent qwuantities, the openmp scaling is rather good

regarding the prace application:

In general I think it is even recommended to try to use standard scaling data (in principle one should try to be somehow specific on the planned applications, but standard data for standard codes are usually well accepted)
Concerning whether to run on Fermi or MareNostrum, I think the main difference is the memory per core (which is also a very critical parameter for yambo)...

Fermi, as a BGQ, has 1 GB/core (16GB per node), while Marenostrum is based on a Intel architecture and has slim and fat nodes (2GB to 8GB/core)... I would then tentatively say that Fermi is more extreme for yambo since it surely requires the whole MPI+OpenMP support, while perhaps, if your memory needs were not too severe, MareNostrum would allow for a more classical parallelisation scheme...

hope it helps
Andrea

[Daniele Varsano]

Dear Manuel,
ETSF collaborations are still possible.
The call is open until 16th April, more information can be found here:
http://www.etsf.eu/news/spring_2015_cal ... e_16_april
http://www.etsf.eu/services/call_proposals

Please let us know if you intend to submit a proposal, by writing to some of us.
Best,
Daniele

[perezlist2015]

hi Andrea,

thanks a lot for the reply, really helpful information!

will you please send to me concrete scalability data that I can use in my application, I cannot find anywhere what you call standard scalability data for yambo

in the main paper at computer physics communications "yambo: an ab-initio tool for excited state calculations", 180 (2009) 1392, by A Marini et al, there is not a single scalability plot

The web site I mentioned in my previous email (at cineca) only mentions the 1024-core information, but there are no plots

I found on another website a limited dataset/plot for scability on MareNostrum, but they only go up to 256 core, and that is not within the minimum accepted number of cores for none of the machines of the prace call

[perezlist2015]

hi Danielle

thanks for your kind reply

I certainly intend to submit an application to ETSF, although at the moment, I have to concentrate in preparing my "preparatory access" application to PRACE, with deadline in a week. I wil discuss the details of the project with you
(and other interested developers) as soon as I have more time to draft something

regards

Manuel

[Daniele Varsano]

Dear Manule,
please fill your signature with your complete affiliation, it is a rule of the forum.
You can find a scalability plot in the news page of the yambo website, dated 11/12/2013. In the yambo paper there is not any plot as it is 6 year old and the scalability we are talking about is a recent feature, and the new parallelization scheme will come with the new release.
Andrea Ferretti can eventually give you more hint and information about the scalability tests.
Best,
Daniele