Test-suite
Web-Interface
The updated list of test-suite runs can be found on the old Yambo web-page.
Tests' list is splitted in branches. For each branch many robots are shown. Check the bug-fixes branch as an example.
This list is automatically updated by the test-suite driver.pl when the -report option is used.
Installation
The test suite is available on GitHub via git
$git clone https://github.com/yambo-code/yambo-tests.git yambo-tests
you may need to provide your GitHub username and password.
Then init and update the ROBOTS submodule
$cd yambo-tests $git submodule init $git submodule update $cd ROBOTS $git checkout master $cd ..
After the first thing to do is the configuration
$./configure --with-yambo=/home/marini/Yambo/yambo/master/
If the configuration works you should see something similar to the following message:
$./configure --with-yambo=/home/marini/Yambo/yambo/master/ checking build system type... x86_64-unknown-linux-gnu checking host system type... x86_64-unknown-linux-gnu checking the anomaly.mlib.ism.cnr.it ROBOT... created hecking for nccopy... no checking for ncftp... ncftp checking for ncftpls... ncftpls checking for ncftpput... ncftpput checking for awk... awk checking for txt2html... txt2html checking the Yambo source... /home/marini/Yambo/yambo/master/ configure: creating ./config.status config.status: creating config/MODULES.pl config.status: creating config/TOOLS.pl
Robot informations
By typing
$./driver.pl -i
you will see some basic info about your machine. This command is useful to see the available flows, configurations and so on. An example is
Running on host: anomaly.mlib.ism.cnr.it By user : marini Version : 5.0 Available CPU's: 8 Available confs: default.sh Available flows: failed_master.pl validate.pl default.pl failed_SLK.pl failed_devel-qp-dbs.pl minimal.pl
Database download
Now it is time to download all databases. This can be easily done using the -d option
$./driver.pl -d all
Compilation
By default the configure copies a basic compilation script in ROBOTS/<YOUR ROBOT>/CONFIGURATIONS/default.sh. This uses internally compiled libraries so it should work easily. In order to test it launch
$./driver.pl -compile
Internal Test
Use the autotest to verify your compiled source:
$./driver.pl -autotest
If everything works you should see as log the following message
==================================================================================
= Starting Yambo test-suite
==================================================================================
- Check requirements : ...ncftp...ncftpls...ncftpput...txt2html
- Test selection : from input
ncdump : /opt/gcc4/netcdf-4.0.1/bin/ncdump
- Executable checks : (yambo OK ) (ypp FAIL )(a2y FAIL )(p2y FAIL )
Running tests : hBN/GW-OPTICS 01_init 02_HF
Projects : nopj
Verbosity : low
Precision : 0.01
Hostname : anomaly.mlib.ism.cnr.it
revision : 14505
build : MPI+SLK
bin dir : bin
shortname :
source : master_-precompiled
Parallel Loop : SERIAL
=--------------------------------------------------------------------------------=
> [1 /1 ] hBN/GW-OPTICS[serial] 5 passes
=--------------------------------------------------------------------------------=
The 5 passes are important as they represent the number of passed tests.
For the impatiens: quick cron run and final reporting
If you do not want to bother with all details of the testing script and want just to install the suite read this section and stop.
In order to properly setup your machine for night test-suite runs you have to:
- Choose the branch(s) to test
For this purpose use
$./driver.pl -edit branches
- create a crontab entry
See here for detailled informations on cron. My personal crontab file looks like
# m h dom mon dow command 30 18 * * Thu ~/bin/scripts/yambo_validation.tcsh validate >> /home/marini/Yambo/sources/svn/yambo-tests/test-suite/LOG 2>&1
with
$cat bin/scripts/yambo_validation.tcsh #!/usr/bin/tcsh cd /home/marini/Yambo/sources/svn/yambo-tests/test-suite ./driver.pl -flow $argv[1] -report
That's all you need.
Available tests
The list of tests can be done using the -l option:
$./driver.pl -l
Note that you can add a TEST to -l to list the specific inputs of that test. For example
$./driver.pl -l GaAs001-c4x4/YAMBO Available input files for GaAs001-c4x4/YAMBO are: 01_init 02_RAS_full 03_RAS_full 04_RAS_front 05_RAS_dimer
Reports & Logs
The autotest and any other run of the test-suite produces three files. Each file is labelled with the date of the run.
In order to see the meaning and the in formations written in the files let's do a quick run of a Dummy test created in order to reproduce some of the possible error conditions.
Let's first clean the suite
$./driver.pl -c
and use the -force to run also tests tagged as BROKEN (like the Dummy one). First thing to do is to download the Dummy database adding the -force option.
$./driver.pl -d Dummy -force
Then we can run the Dummy test using
$./driver.pl -tests Dummy -force
The result will now be
=--------------------------------------------------------------------------------= > [1 /1 ] Dummy[serial] 11 passes 5 fails =--------------------------------------------------------------------------------=
The 5 fails cover most of the potential errors detected by the test-suite. Details can be now found in the report/log files
REPORT file
This file contains very condensed information about the run. There is the duration time plus some detail of the reasons behind the 5 fails.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Yambo test suite global REPORT %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% ================================================================================== Running tests :Dummy Projects :nopj Date - Time :Fri-19-May - 11-46 Build :master - MPI+SLK - rev.14505 Parallel Conf :[serial] - none =--------------------------------------------------------------------------------= FAIL: 5 & NO RUN: 0 & SKIPS: 0 % WHITELIST: 1 % OKs: 10 FAIL details: 1 wrong output % 1 no reference % 1 no output % 2 no DB =--------------------------------------------------------------------------------= % Section Duration : 0:0:3 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % TOTAL Duration : 0:0:3 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
In the dummy case indeed the suite should detect:
- 1 wrong output
- 1 output missing the REFERENCE
- 1 missing output (not generated by the run)
- 2 missing databases.
More details can be found in the LOG and and in the ERROR.
The REPORT file is the one used to send final logs to the mailing list.
ERROR file
This is a useful file with detailed information about the errors only.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Yambo test suite ERRORs log %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% ================================================================================== Running tests :Dummy Projects :nopj Date - Time :Fri-19-May - 11-46 Build :master - MPI+SLK - rev.14505 Parallel Conf :[serial] - none =--------------------------------------------------------------------------------= Dummy/02_hf-serial-master/o-02_hf.hf : VALUE(# 17)@COL# 6 is -5.4691 [REF] vs -4.4691 [OUT] corresponding to 6.1013 [% RELATIVE TO MAX] Dummy/02_hf-serial-master : NO REFERENCE/o-02_hf.ndb.em1s_fragment_1 in OUTPUT Dummy/04_em1s-serial-master : NO o-04_em1s.ndb.em1s_fragment_1 in REFERENCE % Section Duration : 0:0:3 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % TOTAL Duration : 0:0:3 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Running
Before entering in the details of the different running methods let's first describe the Parallel Options
Parallel Options
(parallel options)
-np <N> Fixed number of CPU used
-np_min <N> Minimum number of CPU used
-np_max <N> Maximum number of CPU used
-nt <T> # of OMP threads
-nl <L> # of CPU used for linear algebra
-def_par Use the default parallelization scheme
-rand_par Use randomly generated parallel structures
Examples:
$./driver.pl -tests Dummy -force -np 2
produces all possible combinations of 2 cpu's for the Dummy tests. Each run is labelled in such a way to be unique. The Dummy folder will look like
$ls tests/Dummy/ 01_init-Nmpi2-1-master/ 02_hf-Nmpi2-3-master/ 03_optics-Nmpi2-3-master/ 04_em1s-Nmpi2-3-master/ INPUTS/ yambo.in 02_hf 03_optics 04_em1s 04_em1s-Nmpi2-4-master/ LOG/ 02_hf-Nmpi2-1-master/ 03_optics-Nmpi2-1-master/ 04_em1s-Nmpi2-1-master/ BROKEN REFERENCE/ 02_hf-Nmpi2-2-master/ 03_optics-Nmpi2-2-master/ 04_em1s-Nmpi2-2-master/ Dummy.tar.gz SAVE/
Note that each run is labelled by the number of MPI cpus (Nmpi2) and by the increasing counter of potential combinations of the two cpu's (-1-master,-2-master,...).
Other potential combinations are
- MPI, default parallelization (no cycle)
$./driver.pl -tests Dummy -force -np 2 -def_par
- MPI, random parallelization (no cycle, good for testing)
$./driver.pl -tests Dummy -force -np 2 -rand_par
- MPI, using 2 and 4 cpu's, no cycle on combinations, using random parallelization
$./driver.pl -tests Dummy -force -np_min 2 -np_max 4 -rand_par
- OpenMP (if compiled with OpenMP)
$./driver.pl -tests Dummy -force -nt 2
- MPI+SLK
$./driver.pl -tests Dummy -force -np 4 -nl 4 -def_par
Tests selection
The test-suite can be run in three different ways
- Using a theme
- Passing individual sets of tests
- Using flows
Moreover tests can be selected using projects.
Flows are discussed in the Flows.
Theme
A theme is a just a series of tests. Themes are stored in the THEMES/ folder and are used simply using
%./driver.pl -theme test
test is a text file in the THEMES folder.
Individual Tests
This is the way we used to run the Dummy test. Just use
%./driver.pl -tests Dummy
to select a combination of specific inputs and tests you can use more complex tests options:
%./driver.pl -tests "Dummy 01_init 02_hf; Si_bulk/GW-OPTICS 00_init 02Cohsex"
Projects & Keys
Project are listed with the test lists and are accessed with the lowercase options: elph,sc,rt,nl...
So in order to run all tests belonging to the sc project just type
$./driver.pl -tests all -keys sc
Actually the -keys option is more powerful as, in the future, should also select all inputs with specific observables (like gw, optics and so on).
The list of project are accessed by simply the first column in the list obtained by using the -l option. In the following are in bold.
$./driver.pl -l
Available test materials/systems:
[YAMBO] Al111
C6H3Cl3
PA_chain[H]
SiH4
Al_bulk/GW-OPTICS
H2/OPTICS
H_chain/2.05
H_chain/2.5
hBN/GW-OPTICS
LiF/GW-OPTICS
Si_bulk/GW-OPTICS
Si_surface/OPTICS
Si_wire/OPTICS
MoS2/pwscf/OPTICS
MoS2/abinit/OPTICS/With-SOC
MoS2/abinit/OPTICS/Without-SOC-sp1
MoS2/abinit/OPTICS/Without-SOC-sp2
[QED] Si_bulk/QED
[NL] hBN/NL
[RT] AlAs[H]
H2/RT
hBN/RT
Si_bulk/RT
WSe2/RT[H]
MoS2/pwscf/RT
[SC] H2/SC
hBN/SC
Si_bulk/SC
[MAGNETIC] Si_bulk/MAGNETIC
[ELPH] Diamond/ELPH1[H]
Si_bulk/ELPH/BSE
Si_bulk/ELPH/ELPH_base
Si_bulk/ELPH/ELPH_for_BSE
Si_bulk/ELPH/QP_CTL
[KERR] Cobalt
Nickel
Iron/pwscf
Iron/abinit/With-SOC
Iron/abinit/Without-SOC
[BROKEN] Dummy
GaAs001-c4x4/YAMBO
hBN/PL[PL]
Si001-c4x2/YAMBO
WS2/PL[PL]
Note that to access any of these projects you must use the lowercase option. For example
- SC → -keys sc
- RT → -keys rt
- SC+RT → -keys 'sc rt'
HARD tests
Some of the tests are long. In the tests list they are labelled with a [H]. In order to run them the key hard must be used. For example
%./driver.pl -tests all -keys 'sc rt hard'
runs all SC+RT tests including the heavy ones.
FLOWS
All properties of a single launch of the test-suite and also a more complex test-suite launch looping on several parallel, tests, branches, combinations can be controlled via the FLOW feature.
A FLOW is a perl file (like validate.pl) containing combinations of descriptors.
All FLOW files are in the ROBOTS/<YOUR ROBOT>/FLOWS.
A FLOW is run using
%./driver.pl -flow validate
A general flow descriptor is
% {
% ACTIVE => "yes", # can be yes or no
% MPI_CPU => "NP",
% SLK_CPU => "NM",
% PAR_MODE => "TEXT", # (TEXT can be default, random, loop)
% THREADS => "NT",
% BRANCH => "/home/marini/Yambo/sources/git/yambo/branches/SLK", # (SAVED)
% THEME => "G_parallelization", # (SAVED)
% CONFIG => "gfortran_slk.sh", # (SAVED)
% TESTS => "hBN/GW-OPTICS", # (SAVED)
% KEYS => "nopj elph hard bse rpa", # (SAVED)
%}
The fields labelled with a (SAVE) are not changed if this descriptor is included in a more complex FLOW. In order to explain this feature let's just have a look at the default validate FLOW provided by default:
%#!/usr/bin/perl
%#
%@flow = (
%{
% ACTIVE => "yes",
% CONFIG => "default.sh",
% KEYS => "nopj elph sc hard",
%},
%{
% MPI_CPU => $SYSTEM_NP,
% PAR_MODE => "default",
%},
%{
% MPI_CPU => $SYSTEM_NP,
% PAR_MODE => "random",
%},
%{
% MPI_CPU => $SYSTEM_NP_half,
% PAR_MODE => "loop",
%},
%{
% THREADS => $SYSTEM_NP_half,
%},
%{
% MPI_CPU => $SYSTEM_NP,
% SLK_CPU => $SYSTEM_NP_half,
% PAR_MODE => "default",
%},
%{
% MPI_CPU => $SYSTEM_NP_half,
% THREADS => $SYSTEM_NP_half,
% SLK_CPU => $SYSTEM_SLK,
% PAR_MODE => "default",
%},
%);
When this flow is launched the driver.pl will run all tests of the "nopj elph sc hard" projects using: a serial run, all system CPU's with default parallelization, with random parallelization, cycling among all possible combinations and so so...
Note that also the -autotest is just a simple calculation flow.
Robots
In version 5.0 the concept of Robots was introduced. In practice each time you run the driver.pl test-suite in any form (compilation, flow, manual tests and so on) the perl script will create a ROBOT.
In practice this means that all operations will be labelled with a <ROBOT> string. Single tests, executables will be copied to a dedicated ROBOT folder and also the LOG and REPORT files will be labelled by the specific ROBOT they belong to.
The benefits of this ROBOT structure is that multiple ROBOTS can be run simultaneously on the same test-suite!
Try yourself by simply running two instances of driver.pl.
WARNING: when you run more than 1 robot at the same time be sure to give time to the driver.pl to open all log files.
In addition if you are compiling the same source leave the robot the time to compile it before running a second robot on the same source.
Robot specific operations
You can clean/kill only the robot ID by using
%./driver.pl -c -robot ID %./driver.pl -kill -robot ID
You can also launch the driver.pl with a specific robot ID
%./driver.pl -flow validate -robot ID
Robot branches
When several robots are run at the same time it is often necessary to specify for each robot a branch to do. This can be easily done via the branch file. This is an example of robot dependent branch file (obtained via the ./driver.pl -e branches) command
/home/marini/yambo/master master R1
/home/marini/yambo/master master R2
/home/marini/yambo/gpl master R3
/home/marini/yambo/gpl master R4
/home/marini/yambo/gpl master R5
/home/marini/yambo/gpl master R6
/home/marini/yambo/branches/devel-rt-stable devel-rt-stable R7
/home/marini/yambo/branches/devel-rt-stable devel-rt-stable R8
This file commands the driver.pl to run the master when the robot ID (-robot ID) is 1 or 2, the gpl for robots from 3 to 6 and so on.
Adding New Tests
Adding a new test is easy.
Let's use a simple example.
Imagine I want to add the test hBN/PIPPO for which I have the SAVE and a list of input files. I copy everything in TESTS/MAIN/hBN creating a new directory, PIPPO
hBN>ls GW-OPTICS hBN.tar.gz NL PIPPO PL RT SC
In PIPPO I have only the INPUTS and SAVE folders
PIPPO>ls -R .: INPUTS SAVE
./INPUTS:
01_init 02_HF
./SAVE: ns.db1 ns.kb_pp ns.kb_pp_fragment_1 ns.kb_pp_fragment_2 ns.kb_pp_fragment_3 ns.kb_pp_fragment_4 ns.wf ns.wf_fragments_1_1 ns.wf_fragments_2_1 ns.wf_fragments_3_1 ns.wf_fragments_4_1
At this point I simply run
./driver.pl -update hBN/PIPPO
The driver will run the inputs, create the REFERENCE folder, the .tar.gz of the new test which is uploaded on the ftp server.
Remember to add -compile if your source is not compiled.
At the end of the calculation the git status command should give
yambo-tests>git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes to be committed:
(use "git reset HEAD <file>..." to unstage)
new file: TESTS/MAIN/hBN/PIPPO/INPUTS/01_init
new file: TESTS/MAIN/hBN/PIPPO/INPUTS/02_HF
new file: TESTS/MAIN/hBN/PIPPO/REFERENCE/l-02_HF_HF_and_locXC
new file: TESTS/MAIN/hBN/PIPPO/REFERENCE/o-02_HF.hf
new file: TESTS/MAIN/hBN/PIPPO/REFERENCE/o-02_HF.ndb.HF_and_locXC
new file: TESTS/MAIN/hBN/PIPPO/REFERENCE/r-02_HF_HF_and_locXC
new file: TESTS/MAIN/hBN/PIPPO/SAVE/.empty
It's now enough that you push and the new test will be ready!