Our usage setting are based on the latest release of the Einstein toolkit and we manage our simulation using The Simulation Factory and we will keep to have all the information updated.
Setting the base directory
mkdir -p /home/kepler/${USER}/ET/
mkdir -p /home/kepler/${USER}/simulations
cd /home/kepler/${USER}/ET/
svn --username=depietri co https://einstein.pr.infn.it/svn//PRV/WEB//PRIVATE/UsefulFiles/ UsefulFiles
setting the directory for the various compilation tree
mkdir -p /home/kepler/${USER}/ET/whisky11
mkdir -p /home/kepler/${USER}/ET/dev
mkdir -p /home/kepler/${USER}/ET/ET_2013_11
mkdir -p /home/kepler/${USER}/ET/ET_2012_11
cd /home/kepler/${USER}/ET/dev
curl -kO https://raw.github.com/gridaphobe/CRL/master/GetComponents
chmod a+x GetComponents
./GetComponents --parallel https://svn.einsteintoolkit.org/manifest/trunk/einsteintoolkit.th
cd Cactus
Setting SIMFACTORY
cd /home/kepler/${USER}/ET/dev/Cactus
# ** SIMFACTORY ****
# -------------------
cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.cfg simfactory/mdb/optionlists/
cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.ini simfactory/mdb/machines/
cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.run simfactory/mdb/runscripts/
cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.sub simfactory/mdb/submitscripts/
# -------------------
sed "s/whisky11/dev/" ../../UsefulFiles/SimFactory/defs.local.ini > simfactory/etc/defs.local.ini
emacs -nw simfactory/etc/defs.local.ini
cp ../../UsefulFiles/Thornlists/ET_2013_11.th thornlists/
cp ../../UsefulFiles/Thornlists/EXE_MinimalBarFrank_dev.th thornlists/EXE_MinimalBarFrank.th
cd /home/kepler/${USER}/ET/dev/Cactus
make Test-config options=simfactory/mdb/optionlists/kepler.pr.infn.it.cfg
make -j 8 Test #One should answer: no in order to confirm the Thornlist#
The executable for BarMode with Frank
cd /home/kepler/${USER}/ET/dev/Cactus
simfactory/bin/sim build MinimalBarFrank --thornlist thornlists/EXE_MinimalBarFrank.th
The full ET distributions
# -------------------
cd /home/kepler/${USER}/ET/dev/Cactus
simfactory/bin/sim build ET_2013_11 --thornlist thornlists/ET_2013_11.th
simfactory/bin/sim create-run static_tov --configuration ET_2013_11 --parfile=par/static_tov.par --num-threads=4 --procs=8
For QUEEN BEE all the same after having sync and logged in the machine.
First one has to have the par file up-to-date. For example:
cd /home/kepler/${USER}/ET/dev/Cactus
#### Copy the par file generator
mkdir par/MyProject/
cp ~/SVN_parma/Examples/Par/rnsid_MAR.rpar par/MyProject/
cd par/MyProject/
./rnsid_MAR.rpar
cd /home/kepler/${USER}/ET/dev/Cactus
simfactory/bin/sim sync queenbee
To run executable for the BarMode work now login on the queen bee machine
# # Now the run command # simfactory/bin/sim create A100M10b200_MAR_r50 --parfile=par/MyProject/rnsid_MAR.rpar.dir/A100M10b200_MAR_r50.par --configuration MinimalBarFrank simfactory/bin/sim submit --procs=128 A100M10b200_MAR_r50 #got: PBS -l nodes=16:ppn=8 (that requires ~29 Gbyte of RAM) ## -we got INFO (Carpet): Carpet is running on 16 processes ## INFO (Carpet): This process contains 8 threads, this is thread 0 ## ## Memory statistics from CarpetLib: Current allocated memory: 2027.857 MB ## A total of 128 processors on 16 nodes allocated ## PBS -l nodes=16:ppn=8 ## *me_per_hour = ~49 meaning around 6ms simulated per-day
cd /home/kepler/${USER}/ET/dev/Cactus
# ** SIMFACTORY ****
# -------------------
cp ../../UsefulFiles/SimFactory/fermi.cfg simfactory/mdb/optionlists/
cp ../../UsefulFiles/SimFactory/fermi.ini simfactory/mdb/machines/
cp ../../UsefulFiles/SimFactory/fermi.run simfactory/mdb/runscripts/
cp ../../UsefulFiles/SimFactory/fermi.sub simfactory/mdb/submitscripts/
# -------------------
simfactory/bin/sim login fermi
Once you login on fermi with no problem (eventually creating the request directory)
simfactory/bin/sim sync fermi simfactory/bin/sim build ET_2013_11 --thornlist thornlists/ET_2013_11.th
simfactory/bin/sim list-simulations simfactory/bin/sim show-output A100M10b200_MAR_r50
mkdir -p /home/kepler/${USER}/ET/ET_2013_11
cd /home/kepler/${USER}/ET/ET_2013_11
# -------------------
curl -kO https://raw.github.com/gridaphobe/CRL/master/GetComponents
chmod a+x GetComponents
./GetComponents --parallel https://svn.einsteintoolkit.org/manifest/branches/ET_2013_11/einsteintoolkit.th
./GetComponents --noanonymous ../UsefulFiles/Thornlists/ParmaThorns.th
# -------------------
cd /home/kepler/${USER}/ET/ET_2013_11/Cactus
# -------------------
cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.cfg simfactory/mdb/optionlists/
cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.ini simfactory/mdb/machines/
cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.run simfactory/mdb/runscripts/
cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.sub simfactory/mdb/submitscripts/
# -------------------
sed "s/whisky11/ET_2013_11/" ../../UsefulFiles/SimFactory/defs.local.ini > simfactory/etc/defs.local.ini
simfactory/bin/sim build --thornlist manifest/einsteintoolkit.th
# now there is a running executable: "exe/cactus_sim"
The rest as usual (i.e., as it is for the "dev" tree)
mkdir -p /home/kepler/${USER}/ET/ET_2012_11
cd /home/kepler/${USER}/ET/ET_2012_11
# -------------------
curl -kO https://raw.github.com/gridaphobe/CRL/master/GetComponents
chmod a+x GetComponents
curl -kO https://svn.einsteintoolkit.org/manifest/branches/ET_2012_11/einsteintoolkit.th
./GetComponents --parallel einsteintoolkit_2012_11.th
./GetComponents --noanonymous ../UsefulFiles/Thornlists/ParmaThorns.th
# -------------------
cd /home/kepler/${USER}/ET/ET_2012_11/Cactus
svn checkout https://svn.cct.lsu.edu/repos/numrel/simfactory2/trunk simfactory
# -------------------
cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.cfg simfactory/mdb/optionlists/
cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.ini simfactory/mdb/machines/
cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.run simfactory/mdb/runscripts/
cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.sub simfactory/mdb/submitscripts/
# -------------------
sed "s/whisky11/ET_2012_11/" ../../UsefulFiles/SimFactory/defs.local.ini > simfactory/etc/defs.local.ini
cp ../../UsefulFiles/Thornlists/EXE_MinimalBarFrank_Orsted.th thornlists/EXE_MinimalBarFrank.th
# -------------------
cd /home/kepler/${USER}/ET/ET_2012_11/Cactus
#### Copy the par file generator
mkdir par/MyProject/
cp ~/SVN_parma/Examples/Par/rnsid_MAR_Orstead.rpar par/MyProject/rnsid_MAR.rpar
cd par/MyProject/
./rnsid_MAR.rpar
cd /home/kepler/${USER}/ET/ET_2012_11/Cactus
simfactory/bin/sim build MinimalBarFrank --thornlist thornlists/EXE_MinimalBarFrank.th
The rest as usual (i.e., as it is for the "dev" tree)
cd /home/kepler/${USER}/ET/whisky11
# --- copy our original thorns
tar xzvf /numrel/storageQ01/MagneticBar/CactusTree_WhiskyEXP_March11.tgz
find . -name .svn | awk '{print "rm -rf " $1}'
# --- GET our new thorns
curl -kO https://raw.github.com/gridaphobe/CRL/master/GetComponents
chmod a+x GetComponents
./GetComponents --noanonymous ../UsefulFiles/Thornlists/ParmaThorns.th
Setting SIMFACTORY
cd Cactus/ # ** # ** SIMFACTORY **** # ** # --- Get the new version of simfactory # ** mv simfactory/ old_simfactory svn checkout https://svn.cct.lsu.edu/repos/numrel/simfactory2/trunk simfactory # ** # ------------------- # ** cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.cfg simfactory/mdb/optionlists/ cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.ini simfactory/mdb/machines/ cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.run simfactory/mdb/runscripts/ cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.sub simfactory/mdb/submitscripts/ # ** # ------------------- # ** ########echo "HDF5=yes" >> simfactory/mdb/optionlists/kepler.pr.infn.it.cfg echo "MPI=CUSTOM" >> simfactory/mdb/optionlists/kepler.pr.infn.it.cfg echo "MPI_INC_DIRS=/opt/ExternalLibraryGNU/openmpi/include" >> simfactory/mdb/optionlists/kepler.pr.infn.it.cfg echo "MPI_LIB_DIRS=/opt/ExternalLibraryGNU/openmpi/lib" >> simfactory/mdb/optionlists/kepler.pr.infn.it.cfg echo "MPI_LIBS=mpi_cxx util numa mpi" >> simfactory/mdb/optionlists/kepler.pr.infn.it.cfg # ** # ** sed "s/-L 3/-L 1/" simfactory/mdb/runscripts/queenbee-mvapich2.run > TMP mv TMP simfactory/mdb/runscripts/mike-openmpi.run sed "s/-L 3/-L 1/" simfactory/mdb/runscripts/mike-openmpi.run> TMP mv TMP simfactory/mdb/runscripts/mike-openmpi.run # ** # correct queen bee to use old interface # ** echo "MPI=CUSTOM" >> simfactory/mdb/optionlists/queenbee-mvapich2.cfg echo "MPI=CUSTOM" >> simfactory/mdb/optionlists/mike-openmpi.cfg # ** # ------------------- # ** cp ../../UsefulFiles/SimFactory/defs.local.ini simfactory/etc emacs -nw simfactory/etc/defs.local.ini # **
cd /home/kepler/${USER}/ET/whisky11/Cactus
cp ../../UsefulFiles/Thornlists/EXE_BarModeMar11.th thornlists/
make whisky11-config options=simfactory/mdb/optionlists/kepler.pr.infn.it.cfg THORNLIST=../../UsefulFiles/Thornlists/EXE_BarModeMar11.th
make whisky11
cd /home/kepler/${USER}/ET/whisky11/Cactus
simfactory/bin/sim build whisky11 --thornlist thornlists/EXE_BarModeMar11.th
(to check MPI=OpenMPI OPENMPI_DIR=/opt/ExternalLibraryGNU/openmpi -lmpi_cxx -lutil -lnuma -lmpi)
simfactory/bin/sim create S01b1_14_r80 --parfile=par/MyProject/MagWhisky.rpar.dir/S01b1_14_r80.par --configuration whisky11 simfactory/bin/sim create S01b1_14_r40 --parfile=par/MyProject/MagWhisky.rpar.dir/S01b1_14_r40.par --configuration whisky11 simfactory/bin/sim create S01b1_14_r20 --parfile=par/MyProject/MagWhisky.rpar.dir/S01b1_14_r20.par --configuration whisky11 simfactory/bin/sim submit --procs=32 --num-threads=1 S01b1_14_r80 #got: PBS -l nodes=4:ppn=8 simfactory/bin/sim submit --procs=128 --num-threads=1 S01b1_14_r40 simfactory/bin/sim submit --procs=1024 --num-threads=1 S01b1_14_r20 simfactory/bin/sim show-output S01b1_14_r80
First one has to have the par file up-to-date. For example:
cd /home/kepler/${USER}/ET/whisky11/Cactus
#### Copy the par file generator
mkdir par/MyProject/
cp ~/SVN_parma/Examples/Par/MagWhisky.rpar par/MyProject/
cd par/MyProject/
./MagWhisky.rpar
cd /home/kepler/${USER}/ET/whisky11/Cactus
Now I can check it on kepler!
simfactory/bin/sim create S01b1_14_r80 --parfile=par/MyProject/MagWhisky.rpar.dir/S01b1_14_r80.par --configuration whisky11 simfactory/bin/sim run S01b1_14_r80 --num-threads=1 --procs=8
We got access to the following allocation on HPC machine:
ssh l rdepietr login.fermi.cineca.it
ssh -l depietri qb3.loni.org
ssh -l depietri mike1.hpc.lsu.edu
We do have a tree on our WEB server where we store useful configuration and file needed to better run our simulations. They can be get making a local SVN version
svn --username=depietri co https://einstein.pr.infn.it/svn//PRV/WEB//PRIVATE/UsefulFiles/ UsefulFiles
From here a common tree I usually create subtree for the various version of cactus that I am using. Typically, a very old version of Whisky, the current version of the Einstein Toolkit and the developer tree.
The first step is to download the main cactus components from the EinsteinTollkit repository. We also need the following ThornLists (AEIThorns.th, ParmaThorns.th, Whisky_DEV.th, Whisky_EXP.th, Whisky_PUB.th) that can be downloaded from our PRIVATE web pages ThornLists First you have to go the Directory where you would like to install CACTUS. And then you will have to execute the following commands: (they will create a working Cactus tree ready to be compiled)
curl -O https://raw.github.com/gridaphobe/CRL/master/GetComponents chmod a+x GetComponents ./GetComponents --parallel https://svn.einsteintoolkit.org/manifest/branches/ET_2012_11/einsteintoolkit.th ./GetComponents --noanonymous ./Whisky_DEV.th ./GetComponents --noanonymous ./Whisky_EXP.th ./GetComponents --noanonymous ./AEIThorns.th ./GetComponents --noanonymous ./ParmaThorns.th
curl -O https://raw.github.com/gridaphobe/CRL/master/GetComponents chmod a+x GetComponents ./GetComponents --parallel https://svn.einsteintoolkit.org/manifest/trunk/einsteintoolkit.th
Now Our local arrangements…
./GetComponents --noanonymous ../Orsted/Whisky_DEV.th ./GetComponents --noanonymous ../Orsted/Whisky_EXP.th ./GetComponents --noanonymous ../Orsted/AEIThorns.th ./GetComponents --noanonymous ../Orsted/ParmaThorns.th
tar xzvf /storageQ01/MagneticBar/CactusTree_WhiskyEXP_March11.tgz
find . -name .svn | awk '{print "rm -rf " $1}'
curl -O https://raw.github.com/gridaphobe/CRL/master/GetComponents
chmod a+x GetComponents
./GetComponents --noanonymous ../UsefulFiles/Thornlists/ParmaThorns.th
svn checkout https://svn.cct.lsu.edu/repos/numrel/simfactory2/trunk simfactory
I obtained (on Einstein) an executable suitable for our work on Magnetic Bar with the command "make BarModeMar11-config options=../../UsefulFiles/SimFactory/einstein.pr.infn.it.cfg THORNLIST=../../UsefulFiles/Thornlists/EXE_BarModeMar11.th MPI=OpenMPI OPENMPI_DIR=/usr/lib64/openmpi/1.4-gcc/"
Simfactory to work need that the local machine we use should be defined. This is done properly setting the file "simfactory/etc/defs.local.ini" and for any machine we would like to use to provide in:
[simfactory/mdb/machines/] the machine definitions (eg. SYSTEMNAME.ini) [simfactory/mdb/optionlist/] the machine compilation options (eg. SYSTEMNAME.cgf)
For example we may give the commands:
cp ../../UsefulFiles/SimFactory/*.ini simfactory/mdb/machines/ cp ../../UsefulFiles/SimFactory/*.cfg simfactory/mdb/optionlists/ mv simfactory/mdb/machines/defs.local.ini simfactory/etc/ emacs -nw simfactory/etc/defs.local.ini
Example of all these file can be downloaded from: SIMFACTORY It is also very useful to define the following alias that would simplify the use of simfactory "alias SIM=simfactory/bin/sim"
./simfactory/bin/sim create S01b1_14_r40 --parfile=par/S01b1_14_r40.par --configuration BarModeMar11 ./simfactory/bin/sim submit --procs=256 --num-threads=1 S01b1_14_r40 ./simfactory/bin/sim show-output S01b1_14_r40
To compile on fermi (once SIM factory is properly set):
SIM sync fermiBGQ
SIM login fermiBGQ SIM build PlainOerstead --thornlist thornlists/PlainOerstead.th
That should be enough to create a working executable.
CPP = cpp FPP = cpp CC = /opt/ibmcmp/vacpp/bg/12.1/bin/bgxlc_r CXX = /opt/ibmcmp/vacpp/bg/12.1/bin/bgxlC_r F77 = /opt/ibmcmp/xlf/bg/14.1/bin/bgxlf_r F90 = /opt/ibmcmp/xlf/bg/14.1/bin/bgxlf90_r C_DEPEND = $(CPP) -M $(CPPFLAGS) CXX_DEPEND = $(CPP) -M $(CPPFLAGS) #CPPFLAGS = -DMPICH_IGNORE_CXX_SEEK CPPFLAGS = FPPFLAGS = -traditional CFLAGS = -qarch=qp -qtune=qp -qsuppress=1506-1108:1506-1197 -qlanglvl=stdc99 CXXFLAGS = -qarch=qp -qtune=qp -qsuppress=1506-1108 -qrtti F77FLAGS = -qarch=qp -qtune=qp -qsuppress=1520-022:1501-510:cmpmsg -qextname F90FLAGS = -qarch=qp -qtune=qp -qsuppress=1520-022:1501-510:cmpmsg -qextname #### -qextname (This option preserve the trailing underscore on external #### -qextname ... fortran linkange .... needed to mix up fortran and C code) LDFLAGS = -Wl,-relax SYS_INC_DIRS = LIBDIRS = /opt/ibmcmp/xlf/bg/14.1/bglib64/ LIBS = xlf90_r xlfmath SPI SPI_cnk pthread rt stdc++ #######---------------------------------------------------------- DEBUG = no CPP_DEBUG_FLAGS = -g -DCARPET_DEBUG FPP_DEBUG_FLAGS = -g -DCARPET_DEBUG C_DEBUG_FLAGS = -g -O0 -qnoipa -qcheck=bounds CXX_DEBUG_FLAGS = -g -O0 -qnoipa -qcheck=bounds F77_DEBUG_FLAGS = -g -O0 -qnoipa -qcheck F90_DEBUG_FLAGS = -g -O0 -qnoipa -qcheck #######---------------------------------------------------------- # Try: -qhot=simd # -O3 for C takes >30min to compile ML_BSSN # -O3 for C++ and Fortran crashes with SIGFPE # -qipa leads to undefined functions while linking OPTIMISE = yes CPP_OPTIMISE_FLAGS = # -DCARPET_OPTIMISE -DNDEBUG FPP_OPTIMISE_FLAGS = # -DCARPET_OPTIMISE -DNDEBUG C_OPTIMISE_FLAGS = -O3 -qspill=100000 -qmaxmem=-1 -qhot -qlibmpi CXX_OPTIMISE_FLAGS = -O3 -qspill=100000 -qmaxmem=-1 -qhot -qlibmpi F77_OPTIMISE_FLAGS = -O3 -qspill=100000 -qmaxmem=-1 -qhot -qlibmpi F90_OPTIMISE_FLAGS = -O3 -qspill=100000 -qmaxmem=-1 -qhot -qlibmpi #######---------------------------------------------------------- PROFILE = no CPP_PROFILE_FLAGS = FPP_PROFILE_FLAGS = C_PROFILE_FLAGS = -pg CXX_PROFILE_FLAGS = -pg F77_PROFILE_FLAGS = -pg F90_PROFILE_FLAGS = -pg #######---------------------------------------------------------- OPENMP = yes CPP_OPENMP_FLAGS = FPP_OPENMP_FLAGS = -D_OPENMP C_OPENMP_FLAGS = -qsmp=omp CXX_OPENMP_FLAGS = -qsmp=omp F77_OPENMP_FLAGS = -qsmp=omp F90_OPENMP_FLAGS = -qsmp=omp #######---------------------------------------------------------- WARN = yes CPP_WARN_FLAGS = FPP_WARN_FLAGS = C_WARN_FLAGS = CXX_WARN_FLAGS = F77_WARN_FLAGS = F90_WARN_FLAGS = #######---------------------------------------------------------- ## /bgsys/drivers/ppcfloor/comm/xl.ndebug/bin/mpixlcxx_r -show ## mpixlcxx_r -show #######---------------------------------------------------------- MPI_DIR = /bgsys/drivers/V1R1M1/ppc64/comm/xl.ndebug/ MPI_INC_DIRS = /bgsys/drivers/ppcfloor/comm/default/include /bgsys/drivers/ppcfloor/comm/sys/include /bgsys/drivers/V1R1M1/ppc64/spi/include/kernel/cnk /bgsys/drivers/V1R1M1/ppc64/comm/xl.ndebug/include MPI_LIB_DIRS = /bgsys/drivers/V1R1M1/ppc64/comm/xl.ndebug/lib /bgsys/drivers/V1R1M1/ppc64/comm/sys-fast/lib /bgsys/drivers/V1R1M1/ppc64/spi/lib MPI_LIBS = cxxmpich mpich opa mpl pami SPI SPI_cnk #######---------------------------------------------------------- PTHREADS = yes #######---------------------------------------------------------- BLAS_DIR = /gpfs/scratch/userexternal/rdepietr/external/BLAS LAPACK_DIR = /gpfs/scratch/userexternal/rdepietr/external/LAPACK LORENE_DIR = /gpfs/scratch/userexternal/rdepietr/external/LORENE #######---------------------------------------------------------- GSL_DIR = /cineca/prod/libraries/gsl/1.15/bgq-xl--1.0 HDF5_DIR = /cineca/prod/libraries/hdf5/1.8.9_ser/bgq-xl--1.0 ZLIB_DIR = /cineca/prod/libraries/zlib/1.2.7/bgq-gnu--4.4.6 LIBSZ_DIR = /cineca/prod/libraries/szip/2.1/bgq-xl--1.0/lib LIBJPEG_DIR = /cineca/prod/libraries/libjpeg/8d/bgq-gnu--4.4.6 #######---------------------------------------------------------- FFTW3_DIR = /cineca/prod/libraries/fftw/3.3.2/bgq-xl--1.0 FFTW3_LIBS = fftw3f #######---------------------------------------------------------- ## May need a patch to FFTW3.sh in "ExternalLibraries/FFTW3" #######---------------------------------------------------------- #######----------------------------------------------------------
Compiling the old tree the linker is not able to find libz.a. I simple workaround is to manually add to the default linker search path the directory where the library is located
LIBDIRS=/opt/ibmcmp/xlf/bg/14.1/lib64/ /cineca/prod/libraries/zlib/1.2.7/bgq-gnu--4.4.6/lib /cineca/prod/libraries/szip/2.1/bgq-xl--1.0/lib LIBS= xlf90_r xlfmath SPI SPI_cnk pthread rt stdc++ z sz
instead of
LIBDIRS=/opt/ibmcmp/xlf/bg/14.1/lib64/ LIBS= xlf90_r xlfmath SPI SPI_cnk pthread rt stdc++
VECTORIZATION resulted on internal compiler error.
############################################ #### Vectorization #### #### This created in vectors.h (ToReal macro) #### from thorn LSUThorns/Vectors #### /bgxlC_r: 1501-230 (S) Internal compiler error; please contact your Service ############################################ ## VECTORISE = yes ## VECTORISE_ALIGNED_ARRAYS = no ## VECTORISE_ALWAYS_USE_ALIGNED_LOADS = yes ## VECTORISE_INLINE = no ############################################ VECTORISE = no VECTORISE_ALIGNED_ARRAYS = no VECTORISE_ALWAYS_USE_ALIGNED_LOADS = no VECTORISE_INLINE = no ############################################
Compiling on fermi present that between the March 2011 version till early 2012 there is a difference on the way is are set the MPI options. This result on different way of define the configuration variables
OLD VERSION:
MPI = CUSTOM
NEW VERSION
### MPI = CUSTOM
The other settings are unchanged
MPI_DIR = /bgsys/drivers/V1R1M1/ppc64/comm/xl.ndebug/ MPI_INC_DIRS = /bgsys/drivers/ppcfloor/comm/default/include /bgsys/drivers/ppcfloor/comm/sys/include /bgsys/drivers/V1R1M1/ppc64/spi/include/kernel/cnk /bgsys/drivers/V1R1M1/ppc64/comm/xl.ndebug/include MPI_LIB_DIRS = /bgsys/drivers/V1R1M1/ppc64/comm/xl.ndebug/lib /bgsys/drivers/V1R1M1/ppc64/comm/sys-fast/lib /bgsys/drivers/V1R1M1/ppc64/spi/lib MPI_LIBS = cxxmpich mpich opa mpl pami SPI SPI_cnk
The new version of Cactus has a main problem on "Fermi". The plain compilation will create an executable that result on the errors:
2012-12-14 13:44:52.417 (FATAL) [0x40000c99050] :18315:ibm.runjob.client.Job: could not start job: job failed to start 2012-12-14 13:44:52.417 (FATAL) [0x40000c99050] :18315:ibm.runjob.client.Job: Load failed on R31-ID-J04: Generating static TLB map for appl
Cineca support say that this error is due to: "lrunjob non e' in grado di allocare tutta la memoria richiesta dall'eseguibile." Any easy way to take out this error is to remove the support for the library "hwloc".
Compiling with March 2011 source tree results on the above error. The assertion is generated in "void CarpetLib_setmemlimit (CCTK_ARGUMENTS)" of file Carpet/CarpetLib/src/mem.cc (line 274). Solution: Take out lines:
// {
// check (not setrlimit (RLIMIT_AS, & aslimit));
// }
The setting of the IO ascii has a BUG specifying the info for "out_fileinfo". In particular if "all" is specified then the simulation end with "2013-02-06 08:21:24.990 (WARN ) [0x40001059050] LL13020608132090:111206:ibm.runjob.client.Job: terminated by signal 6" and the assert error "cactus_MinimalBarWhisky: /gpfs/scratch/userexternal/rdepietr/SRC/WhiskyMar11/Cactus/configs/MinimalBarWhisky/build/CarpetIOScalar/ioscalar.cc:388: CarpetIOScalar::OutputVarAs: Assertion `file.good()' failed."
The options "parameter filename" and "axis labels" do works.
KEYWORD out_fileinfo "Add some useful file information to output files ?" STEERABLE = RECOVER
{
"none" :: "no file information"
"creation date" :: "add creation date"
"parameter filename" :: "add parameter filename"
"axis labels" :: "add axis labels information to output files"
"all" :: "add all available file information"
} "all"
We did check on the motion of star on the grid with ML_BSSN. We checked the following possibility:
(o2) CXX_OPTIMISE_FLAGS = -O2 -qsmp=noauto:noostls
(NR) CXXFLAGS = ...... -qnokeyword=restrict -qalias=norestric
CXX_OPTIMISE_FLAGS = -O3 -qspill=100000 -qmaxmem=-1 -qhot -qlibmpi
(CC) CXXFLAGS = ...... -qnokeyword=restrict -qalias=norestric
CXX_OPTIMISE_FLAGS = -O3 -qspill=100000 -qmaxmem=-1 -qhot -qlibmpi
cctk_cv_cxx_restrict=no
We observed a different behavior of the the moving of the grid with respect to what happen on INTEL machine. Same compilation tree code.
The fermi system submission is based on the loadlever batch system and the main submition command is "llsubmit <JOB.cmd>
Here is an example of a command file to lunch pure MPI on 2048 cores:
#!/bin/bash
#
# @ job_name = JOBNAME.$(jobid)
# @ output = z.JOBNAME.$(jobid).out
# @ error = z.JOBNAME.$(jobid).err
# @ shell = /bin/bash
# @ job_type = bluegene
# @ wall_clock_limit = 2:30:00
# @ bg_size = 128
# @ class = parallel
# @ account_no = INFN_OG51
# @ queue
#### (name of par FILE) ==> PARFILE (without .par extension)
#### 4 ==> NT number of openMP threads
#### 512 ==> NP number of NPI processes
#### 128 ==> BLUGENE size of the run
export NT=1
export NP=2048
export PAROLEE=xCARPET_RL3_dx0.400it32
export EXE=/gpfs/scratch/userexternal/rdepietr/SRC/Orsted/Cactus/exe/cactus_PlainOerstead
export PAR=/gpfs/scratch/userexternal/rdepietr/SpeedTest/ParFiles/${PARFILE}.par
export WDIR=/gpfs/scratch/userexternal/rdepietr/SpeedTest/Runs/${PARFILE}/128_np${NP}_nt${NT}/
export RANKSperNODE=$((16/${NT}))
echo "================================="
echo "Executable is : ${EXE}"
echo "NP is : ${NP}"
echo "NT is : ${NT}"
echo "rank per node : ${RANKSperNODE}"
echo "Blue gene size is: xBGsize"
echo "Total Number of treads is: $((${NP}*${NT}))"
echo "Total Number of BLUE GENE: $((128*${RANKSoperNODE}*${NT}))"
echo "PAR file is : ${PAR}"
echo "Working Dir is: ${WDIR}"
echo "================================="
mkdir -p ${WDIR}
cd ${WDIR}
pwd
ls
echo "================================="
# -mode SMP
# -mode DUAL
# -mode VN
# -np `${nodes}*${taskspernode}`
echo runjob --np ${NP} --ranks-per-node $RANKSperNODE --envs OMP_NUM_THREADS=4 --exe $EXE --args ${PAR}
echo "**************************************************"
echo "**(TIME) START: " $(date)
echo "**************************************************"
export start=$(date +"%s" )
runjob --np ${NP} --ranks-per-node $RANKSperNODE --envs OMP_NUM_THREADS=4 --exe $EXE --args ${PAR} | tee OUTPUT
export end=$(date +"%s" )
export timeSEC=$(($end - $start))
echo "**************************************************"
echo "**(TIME) STOP: " $(date)
echo "**************************************************"
echo "**(WALL TIME) SEC:" $timeSEC
echo "**************************************************"
Just to start
> SIM setup
resulted in
Determining local machine name: fen08.fermi.cineca.it Creating machine fen08.fermi.cineca.it from generic: machine fen08.fermi.cineca.it [/fermi/home/userexternal/rdepietr/DEV/Cactus/simfactory/mdb/machines/fen08.fermi.cineca.it.ini] created successfully enter value for key user [rdepietr]: enter value for key email [rdepietr]: roberto.depietri@unipr.it enter value for key allocation []: INFN_OG51 enter value for key sourcebasedir (the parent directory containing the Cactus sourcetree) [/fermi/home/userexternal/rdepietr/DEV]: enter value for key basedir (the location of simfactory simulations) [/fermi/home/userexternal/rdepietr/simulations]: /gpfs/scratch/userexternal/rdepietr/simulations would you like to enter key/value pairs for a specific machine? [Y/N*]: --------------------SUMMARY---------------------[default] user = rdepietr email = roberto.depietri@unipr.it allocation = INFN_OG51 sourcebasedir = /fermi/home/userexternal/rdepietr/DEV basedir = /gpfs/scratch/userexternal/rdepietr/simulations ------------------END SUMMARY------------------: Save contents [Y*/N]: Contents successfully written to /fermi/home/userexternal/rdepietr/DEV/Cactus/simfactory/etc/defs.local.ini
To properly define the setting I decided to name the system as BlueGeneQfermi