UNIVERSITÀ DI PARMA
DIPARTIMENTO DI SCIENZE MATEMATICHE, FISICHE E INFORMATICHE

Strumenti Utente

Strumenti Sito

Traccia: • How to set up CACTUS
roberto.depietri:user:cactus

Indice

How to set up CACTUS

Our usage setting are based on the latest release of the Einstein toolkit and we manage our simulation using The Simulation Factory and we will keep to have all the information updated.

SETTING KEPLER

Setting the base directory 

mkdir -p /home/kepler/${USER}/ET/
mkdir -p /home/kepler/${USER}/simulations
cd /home/kepler/${USER}/ET/
svn --username=depietri co https://einstein.pr.infn.it/svn//PRV/WEB//PRIVATE/UsefulFiles/ UsefulFiles

setting the directory for the various compilation tree 

mkdir -p /home/kepler/${USER}/ET/whisky11
mkdir -p /home/kepler/${USER}/ET/dev
mkdir -p /home/kepler/${USER}/ET/ET_2013_11
mkdir -p /home/kepler/${USER}/ET/ET_2012_11

Get the dev tree working

cd  /home/kepler/${USER}/ET/dev
curl -kO https://raw.github.com/gridaphobe/CRL/master/GetComponents 
chmod a+x GetComponents
./GetComponents --parallel https://svn.einsteintoolkit.org/manifest/trunk/einsteintoolkit.th
cd Cactus

Setting SIMFACTORY 

cd  /home/kepler/${USER}/ET/dev/Cactus
# ** SIMFACTORY ****
# -------------------
cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.cfg simfactory/mdb/optionlists/
cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.ini simfactory/mdb/machines/
cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.run simfactory/mdb/runscripts/
cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.sub simfactory/mdb/submitscripts/
# -------------------
sed "s/whisky11/dev/" ../../UsefulFiles/SimFactory/defs.local.ini > simfactory/etc/defs.local.ini 
emacs -nw simfactory/etc/defs.local.ini 
cp ../../UsefulFiles/Thornlists/ET_2013_11.th thornlists/
cp  ../../UsefulFiles/Thornlists/EXE_MinimalBarFrank_dev.th  thornlists/EXE_MinimalBarFrank.th

Example of standard build

cd  /home/kepler/${USER}/ET/dev/Cactus
make Test-config options=simfactory/mdb/optionlists/kepler.pr.infn.it.cfg
make -j 8 Test   #One should answer: no in order to confirm the Thornlist#

Example of SIMFACTORY build

The executable for BarMode with Frank 

cd  /home/kepler/${USER}/ET/dev/Cactus
simfactory/bin/sim build MinimalBarFrank --thornlist thornlists/EXE_MinimalBarFrank.th

The full ET distributions 

# -------------------
cd  /home/kepler/${USER}/ET/dev/Cactus
simfactory/bin/sim build ET_2013_11 --thornlist thornlists/ET_2013_11.th
simfactory/bin/sim create-run static_tov --configuration ET_2013_11 --parfile=par/static_tov.par --num-threads=4 --procs=8

For QUEEN BEE all the same after having sync and logged in the machine.

First one has to have the par file up-to-date. For example: 

cd  /home/kepler/${USER}/ET/dev/Cactus
#### Copy the par file generator
mkdir par/MyProject/
cp  ~/SVN_parma/Examples/Par/rnsid_MAR.rpar par/MyProject/
cd par/MyProject/
./rnsid_MAR.rpar
cd  /home/kepler/${USER}/ET/dev/Cactus
simfactory/bin/sim sync queenbee

To run executable for the BarMode work now login on the queen bee machine 

#
#  Now the run command
#
simfactory/bin/sim create A100M10b200_MAR_r50 --parfile=par/MyProject/rnsid_MAR.rpar.dir/A100M10b200_MAR_r50.par --configuration MinimalBarFrank
simfactory/bin/sim submit --procs=128  A100M10b200_MAR_r50  #got: PBS -l nodes=16:ppn=8 (that requires ~29 Gbyte of RAM)
## -we got INFO (Carpet): Carpet is running on 16 processes
##         INFO (Carpet): This process contains 8 threads, this is thread 0
##         
##         Memory statistics from CarpetLib: Current allocated memory:  2027.857 MB
## A total of 128 processors on 16 nodes allocated
## PBS -l nodes=16:ppn=8
## *me_per_hour = ~49 meaning around 6ms simulated per-day

SIMFACTORY setting for the FERMI machine

cd  /home/kepler/${USER}/ET/dev/Cactus
# ** SIMFACTORY ****
# -------------------
cp ../../UsefulFiles/SimFactory/fermi.cfg simfactory/mdb/optionlists/
cp ../../UsefulFiles/SimFactory/fermi.ini simfactory/mdb/machines/
cp ../../UsefulFiles/SimFactory/fermi.run simfactory/mdb/runscripts/
cp ../../UsefulFiles/SimFactory/fermi.sub simfactory/mdb/submitscripts/
# -------------------
simfactory/bin/sim login fermi

Once you login on fermi with no problem (eventually creating the request directory) 

simfactory/bin/sim sync fermi
simfactory/bin/sim build ET_2013_11 --thornlist thornlists/ET_2013_11.th  
simfactory/bin/sim list-simulations
simfactory/bin/sim show-output   A100M10b200_MAR_r50

Get the ET_2013_11 tree working

 mkdir -p /home/kepler/${USER}/ET/ET_2013_11
 cd /home/kepler/${USER}/ET/ET_2013_11
 # -------------------
 curl -kO https://raw.github.com/gridaphobe/CRL/master/GetComponents 
 chmod a+x GetComponents
 ./GetComponents --parallel https://svn.einsteintoolkit.org/manifest/branches/ET_2013_11/einsteintoolkit.th
 ./GetComponents  --noanonymous  ../UsefulFiles/Thornlists/ParmaThorns.th
 # -------------------
 cd /home/kepler/${USER}/ET/ET_2013_11/Cactus
 # -------------------
 cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.cfg simfactory/mdb/optionlists/
 cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.ini simfactory/mdb/machines/
 cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.run simfactory/mdb/runscripts/
 cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.sub simfactory/mdb/submitscripts/
 # -------------------
 sed "s/whisky11/ET_2013_11/" ../../UsefulFiles/SimFactory/defs.local.ini > simfactory/etc/defs.local.ini 
 simfactory/bin/sim build --thornlist manifest/einsteintoolkit.th
 # now there is a running executable: "exe/cactus_sim"
 The rest as usual (i.e., as it is for the  "dev" tree)

Get the ET_2012_11 tree working

 mkdir -p /home/kepler/${USER}/ET/ET_2012_11
 cd /home/kepler/${USER}/ET/ET_2012_11
 # -------------------
 curl -kO https://raw.github.com/gridaphobe/CRL/master/GetComponents 
 chmod a+x GetComponents
 curl -kO https://svn.einsteintoolkit.org/manifest/branches/ET_2012_11/einsteintoolkit.th
 ./GetComponents --parallel einsteintoolkit_2012_11.th 
 ./GetComponents  --noanonymous  ../UsefulFiles/Thornlists/ParmaThorns.th
 # -------------------
 cd /home/kepler/${USER}/ET/ET_2012_11/Cactus
 svn checkout https://svn.cct.lsu.edu/repos/numrel/simfactory2/trunk simfactory
 # -------------------
 cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.cfg simfactory/mdb/optionlists/
 cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.ini simfactory/mdb/machines/
 cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.run simfactory/mdb/runscripts/
 cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.sub simfactory/mdb/submitscripts/
 # -------------------
 sed "s/whisky11/ET_2012_11/" ../../UsefulFiles/SimFactory/defs.local.ini > simfactory/etc/defs.local.ini
 cp  ../../UsefulFiles/Thornlists/EXE_MinimalBarFrank_Orsted.th  thornlists/EXE_MinimalBarFrank.th
 # -------------------
 cd  /home/kepler/${USER}/ET/ET_2012_11/Cactus
 #### Copy the par file generator
 mkdir par/MyProject/
 cp  ~/SVN_parma/Examples/Par/rnsid_MAR_Orstead.rpar par/MyProject/rnsid_MAR.rpar
 cd par/MyProject/
 ./rnsid_MAR.rpar
 cd  /home/kepler/${USER}/ET/ET_2012_11/Cactus
 simfactory/bin/sim build MinimalBarFrank --thornlist thornlists/EXE_MinimalBarFrank.th
 The rest as usual (i.e., as it is for the  "dev" tree)

Get the whisky11 tree working

cd  /home/kepler/${USER}/ET/whisky11
# --- copy our original thorns
tar xzvf /numrel/storageQ01/MagneticBar/CactusTree_WhiskyEXP_March11.tgz    
find . -name .svn | awk '{print "rm -rf " $1}'
# --- GET our new thorns
curl -kO https://raw.github.com/gridaphobe/CRL/master/GetComponents 
chmod a+x GetComponents
./GetComponents  --noanonymous  ../UsefulFiles/Thornlists/ParmaThorns.th

Setting SIMFACTORY 

cd Cactus/
# **
# ** SIMFACTORY ****
# **
# --- Get the new version of simfactory
# **
mv simfactory/ old_simfactory
svn checkout https://svn.cct.lsu.edu/repos/numrel/simfactory2/trunk simfactory
# **
# -------------------
# **
cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.cfg simfactory/mdb/optionlists/
cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.ini simfactory/mdb/machines/
cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.run simfactory/mdb/runscripts/
cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.sub simfactory/mdb/submitscripts/
# **
# -------------------
# **
########echo "HDF5=yes"                                     >>  simfactory/mdb/optionlists/kepler.pr.infn.it.cfg
echo "MPI=CUSTOM"                                           >>  simfactory/mdb/optionlists/kepler.pr.infn.it.cfg
echo "MPI_INC_DIRS=/opt/ExternalLibraryGNU/openmpi/include" >>  simfactory/mdb/optionlists/kepler.pr.infn.it.cfg
echo "MPI_LIB_DIRS=/opt/ExternalLibraryGNU/openmpi/lib"     >>  simfactory/mdb/optionlists/kepler.pr.infn.it.cfg
echo "MPI_LIBS=mpi_cxx  util  numa  mpi"                    >>  simfactory/mdb/optionlists/kepler.pr.infn.it.cfg 
# **
# **
sed "s/-L 3/-L 1/" simfactory/mdb/runscripts/queenbee-mvapich2.run > TMP
mv TMP simfactory/mdb/runscripts/mike-openmpi.run
sed "s/-L 3/-L 1/" simfactory/mdb/runscripts/mike-openmpi.run> TMP
mv TMP simfactory/mdb/runscripts/mike-openmpi.run
# **
# correct queen bee to use old interface
# **
echo "MPI=CUSTOM"        >>  simfactory/mdb/optionlists/queenbee-mvapich2.cfg 
echo "MPI=CUSTOM"        >>  simfactory/mdb/optionlists/mike-openmpi.cfg
# **
# -------------------
# **
cp ../../UsefulFiles/SimFactory/defs.local.ini simfactory/etc
emacs -nw simfactory/etc/defs.local.ini 
 # **

Example of standard build

cd  /home/kepler/${USER}/ET/whisky11/Cactus
cp ../../UsefulFiles/Thornlists/EXE_BarModeMar11.th thornlists/
make whisky11-config options=simfactory/mdb/optionlists/kepler.pr.infn.it.cfg THORNLIST=../../UsefulFiles/Thornlists/EXE_BarModeMar11.th 
make whisky11

Example of SIMFACTORY build

cd  /home/kepler/${USER}/ET/whisky11/Cactus
simfactory/bin/sim build whisky11 --thornlist thornlists/EXE_BarModeMar11.th

(to check MPI=OpenMPI OPENMPI_DIR=/opt/ExternalLibraryGNU/openmpi -lmpi_cxx -lutil -lnuma -lmpi)

Example of run on queen bee

simfactory/bin/sim create S01b1_14_r80 --parfile=par/MyProject/MagWhisky.rpar.dir/S01b1_14_r80.par --configuration whisky11
simfactory/bin/sim create S01b1_14_r40 --parfile=par/MyProject/MagWhisky.rpar.dir/S01b1_14_r40.par --configuration whisky11
simfactory/bin/sim create S01b1_14_r20 --parfile=par/MyProject/MagWhisky.rpar.dir/S01b1_14_r20.par --configuration whisky11
simfactory/bin/sim submit --procs=32   --num-threads=1  S01b1_14_r80  #got: PBS -l nodes=4:ppn=8
simfactory/bin/sim submit --procs=128  --num-threads=1  S01b1_14_r40
simfactory/bin/sim submit --procs=1024 --num-threads=1  S01b1_14_r20
simfactory/bin/sim show-output   S01b1_14_r80

First one has to have the par file up-to-date. For example: 

 cd  /home/kepler/${USER}/ET/whisky11/Cactus
 #### Copy the par file generator
 mkdir par/MyProject/
 cp  ~/SVN_parma/Examples/Par/MagWhisky.rpar  par/MyProject/
 cd par/MyProject/
 ./MagWhisky.rpar 
 cd  /home/kepler/${USER}/ET/whisky11/Cactus

Now I can check it on kepler! 

simfactory/bin/sim create S01b1_14_r80 --parfile=par/MyProject/MagWhisky.rpar.dir/S01b1_14_r80.par --configuration whisky11
simfactory/bin/sim run S01b1_14_r80 --num-threads=1 --procs=8

HPC MACHINE available to the group

We got access to the following allocation on HPC machine:

  • Fermi SYSTEM 
  ssh l rdepietr login.fermi.cineca.it
  • Loni queenbee at LSU (loni_numrel08)
   ssh -l depietri qb3.loni.org
  • HPC mike2 at LSU (hpc_hyrel13)
   ssh -l depietri mike1.hpc.lsu.edu
  • Xsede Kraken, Stampede (TG-MCA02N014)

OLD NOTES (before November the 15th, 2013)

We do have a tree on our WEB server where we store useful configuration and file needed to better run our simulations. They can be get making a local SVN version 

svn --username=depietri co https://einstein.pr.infn.it/svn//PRV/WEB//PRIVATE/UsefulFiles/ UsefulFiles

From here a common tree I usually create subtree for the various version of cactus that I am using. Typically, a very old version of Whisky, the current version of the Einstein Toolkit and the developer tree.

Download CACTUS (Orsted)

The first step is to download the main cactus components from the EinsteinTollkit repository. We also need the following ThornLists (AEIThorns.th, ParmaThorns.th, Whisky_DEV.th, Whisky_EXP.th, Whisky_PUB.th) that can be downloaded from our PRIVATE web pages ThornLists First you have to go the Directory where you would like to install CACTUS. And then you will have to execute the following commands: (they will create a working Cactus tree ready to be compiled) 

curl -O https://raw.github.com/gridaphobe/CRL/master/GetComponents 
chmod a+x GetComponents
./GetComponents --parallel https://svn.einsteintoolkit.org/manifest/branches/ET_2012_11/einsteintoolkit.th
./GetComponents  --noanonymous  ./Whisky_DEV.th
./GetComponents  --noanonymous  ./Whisky_EXP.th
./GetComponents  --noanonymous  ./AEIThorns.th
./GetComponents  --noanonymous  ./ParmaThorns.th

Download CACTUS (Developer TREE)

curl -O https://raw.github.com/gridaphobe/CRL/master/GetComponents 
chmod a+x GetComponents
./GetComponents --parallel https://svn.einsteintoolkit.org/manifest/trunk/einsteintoolkit.th

Now Our local arrangements… 

./GetComponents  --noanonymous  ../Orsted/Whisky_DEV.th
./GetComponents  --noanonymous  ../Orsted/Whisky_EXP.th
./GetComponents  --noanonymous  ../Orsted/AEIThorns.th
./GetComponents  --noanonymous  ../Orsted/ParmaThorns.th

Our old Version (Just not to get mixed up)

  tar xzvf /storageQ01/MagneticBar/CactusTree_WhiskyEXP_March11.tgz    
  find . -name .svn | awk '{print "rm -rf " $1}'
  curl -O https://raw.github.com/gridaphobe/CRL/master/GetComponents 
  chmod a+x GetComponents
  ./GetComponents  --noanonymous  ../UsefulFiles/Thornlists/ParmaThorns.th
  svn checkout https://svn.cct.lsu.edu/repos/numrel/simfactory2/trunk simfactory

I obtained (on Einstein) an executable suitable for our work on Magnetic Bar with the command "make BarModeMar11-config options=../../UsefulFiles/SimFactory/einstein.pr.infn.it.cfg THORNLIST=../../UsefulFiles/Thornlists/EXE_BarModeMar11.th MPI=OpenMPI OPENMPI_DIR=/usr/lib64/openmpi/1.4-gcc/"

Set up SIMFACTORY

Simfactory to work need that the local machine we use should be defined. This is done properly setting the file "simfactory/etc/defs.local.ini" and for any machine we would like to use to provide in: 

[simfactory/mdb/machines/] the machine definitions (eg. SYSTEMNAME.ini)
[simfactory/mdb/optionlist/] the machine compilation options (eg. SYSTEMNAME.cgf)

For example we may give the commands: 

 cp ../../UsefulFiles/SimFactory/*.ini simfactory/mdb/machines/
 cp ../../UsefulFiles/SimFactory/*.cfg simfactory/mdb/optionlists/
 mv simfactory/mdb/machines/defs.local.ini simfactory/etc/
 emacs -nw simfactory/etc/defs.local.ini

Example of all these file can be downloaded from: SIMFACTORY It is also very useful to define the following alias that would simplify the use of simfactory "alias SIM=simfactory/bin/sim" 

 ./simfactory/bin/sim create  S01b1_14_r40 --parfile=par/S01b1_14_r40.par --configuration BarModeMar11
 ./simfactory/bin/sim submit --procs=256  --num-threads=1  S01b1_14_r40
 ./simfactory/bin/sim show-output   S01b1_14_r40

COMPILING on FERMI

To compile on fermi (once SIM factory is properly set):

SIM sync fermiBGQ 

SIM login fermiBGQ
SIM build PlainOerstead  --thornlist thornlists/PlainOerstead.th

That should be enough to create a working executable.

General setting for cactus compilation

CPP = cpp
FPP = cpp
CC  = /opt/ibmcmp/vacpp/bg/12.1/bin/bgxlc_r
CXX = /opt/ibmcmp/vacpp/bg/12.1/bin/bgxlC_r
F77 = /opt/ibmcmp/xlf/bg/14.1/bin/bgxlf_r
F90 = /opt/ibmcmp/xlf/bg/14.1/bin/bgxlf90_r
C_DEPEND   = $(CPP) -M $(CPPFLAGS)
CXX_DEPEND = $(CPP) -M $(CPPFLAGS)
#CPPFLAGS = -DMPICH_IGNORE_CXX_SEEK
CPPFLAGS = 
FPPFLAGS = -traditional
CFLAGS   = -qarch=qp -qtune=qp -qsuppress=1506-1108:1506-1197 -qlanglvl=stdc99
CXXFLAGS = -qarch=qp -qtune=qp -qsuppress=1506-1108 -qrtti
F77FLAGS = -qarch=qp -qtune=qp -qsuppress=1520-022:1501-510:cmpmsg -qextname
F90FLAGS = -qarch=qp -qtune=qp -qsuppress=1520-022:1501-510:cmpmsg -qextname
####  -qextname (This option preserve the trailing underscore on external  
####  -qextname ... fortran linkange .... needed to mix up fortran and C code)
LDFLAGS = -Wl,-relax
SYS_INC_DIRS =
LIBDIRS      =  /opt/ibmcmp/xlf/bg/14.1/bglib64/ 
LIBS         = xlf90_r xlfmath SPI SPI_cnk pthread rt stdc++   
#######----------------------------------------------------------
DEBUG           = no
CPP_DEBUG_FLAGS = -g -DCARPET_DEBUG
FPP_DEBUG_FLAGS = -g -DCARPET_DEBUG
C_DEBUG_FLAGS   = -g -O0 -qnoipa -qcheck=bounds
CXX_DEBUG_FLAGS = -g -O0 -qnoipa -qcheck=bounds
F77_DEBUG_FLAGS = -g -O0 -qnoipa -qcheck
F90_DEBUG_FLAGS = -g -O0 -qnoipa -qcheck
#######----------------------------------------------------------
# Try: -qhot=simd
# -O3 for C takes >30min to compile ML_BSSN
# -O3 for C++ and Fortran crashes with SIGFPE
# -qipa leads to undefined functions while linking
OPTIMISE           = yes
CPP_OPTIMISE_FLAGS = # -DCARPET_OPTIMISE -DNDEBUG
FPP_OPTIMISE_FLAGS = # -DCARPET_OPTIMISE -DNDEBUG
C_OPTIMISE_FLAGS   = -O3 -qspill=100000 -qmaxmem=-1 -qhot -qlibmpi
CXX_OPTIMISE_FLAGS = -O3 -qspill=100000 -qmaxmem=-1 -qhot -qlibmpi
F77_OPTIMISE_FLAGS = -O3 -qspill=100000 -qmaxmem=-1 -qhot -qlibmpi
F90_OPTIMISE_FLAGS = -O3 -qspill=100000 -qmaxmem=-1 -qhot -qlibmpi
#######----------------------------------------------------------
PROFILE           = no
CPP_PROFILE_FLAGS =
FPP_PROFILE_FLAGS =
C_PROFILE_FLAGS   = -pg
CXX_PROFILE_FLAGS = -pg
F77_PROFILE_FLAGS = -pg
F90_PROFILE_FLAGS = -pg
#######----------------------------------------------------------
OPENMP           = yes
CPP_OPENMP_FLAGS = 
FPP_OPENMP_FLAGS = -D_OPENMP
C_OPENMP_FLAGS   = -qsmp=omp
CXX_OPENMP_FLAGS = -qsmp=omp
F77_OPENMP_FLAGS = -qsmp=omp
F90_OPENMP_FLAGS = -qsmp=omp
#######----------------------------------------------------------
WARN           = yes
CPP_WARN_FLAGS =
FPP_WARN_FLAGS =
C_WARN_FLAGS   =
CXX_WARN_FLAGS =
F77_WARN_FLAGS =
F90_WARN_FLAGS = 
#######----------------------------------------------------------
##  /bgsys/drivers/ppcfloor/comm/xl.ndebug/bin/mpixlcxx_r  -show 
## mpixlcxx_r -show 
#######----------------------------------------------------------
MPI_DIR      = /bgsys/drivers/V1R1M1/ppc64/comm/xl.ndebug/
MPI_INC_DIRS = /bgsys/drivers/ppcfloor/comm/default/include /bgsys/drivers/ppcfloor/comm/sys/include /bgsys/drivers/V1R1M1/ppc64/spi/include/kernel/cnk /bgsys/drivers/V1R1M1/ppc64/comm/xl.ndebug/include
MPI_LIB_DIRS = /bgsys/drivers/V1R1M1/ppc64/comm/xl.ndebug/lib /bgsys/drivers/V1R1M1/ppc64/comm/sys-fast/lib /bgsys/drivers/V1R1M1/ppc64/spi/lib
MPI_LIBS     = cxxmpich mpich opa mpl    pami   SPI SPI_cnk
#######----------------------------------------------------------
PTHREADS = yes
#######----------------------------------------------------------
BLAS_DIR    = /gpfs/scratch/userexternal/rdepietr/external/BLAS
LAPACK_DIR  = /gpfs/scratch/userexternal/rdepietr/external/LAPACK
LORENE_DIR  =  /gpfs/scratch/userexternal/rdepietr/external/LORENE
#######----------------------------------------------------------
GSL_DIR = /cineca/prod/libraries/gsl/1.15/bgq-xl--1.0
HDF5_DIR = /cineca/prod/libraries/hdf5/1.8.9_ser/bgq-xl--1.0
ZLIB_DIR = /cineca/prod/libraries/zlib/1.2.7/bgq-gnu--4.4.6
LIBSZ_DIR = /cineca/prod/libraries/szip/2.1/bgq-xl--1.0/lib
LIBJPEG_DIR = /cineca/prod/libraries/libjpeg/8d/bgq-gnu--4.4.6
#######----------------------------------------------------------
FFTW3_DIR   = /cineca/prod/libraries/fftw/3.3.2/bgq-xl--1.0
FFTW3_LIBS  = fftw3f
#######----------------------------------------------------------
##  May need a patch to FFTW3.sh in "ExternalLibraries/FFTW3" 
#######----------------------------------------------------------
#######----------------------------------------------------------

Know problems

ZliB not found

Compiling the old tree the linker is not able to find libz.a. I simple workaround is to manually add to the default linker search path the directory where the library is located 

LIBDIRS=/opt/ibmcmp/xlf/bg/14.1/lib64/ /cineca/prod/libraries/zlib/1.2.7/bgq-gnu--4.4.6/lib  /cineca/prod/libraries/szip/2.1/bgq-xl--1.0/lib 
LIBS= xlf90_r xlfmath SPI SPI_cnk pthread rt stdc++ z sz

instead of 

LIBDIRS=/opt/ibmcmp/xlf/bg/14.1/lib64/
LIBS= xlf90_r xlfmath SPI SPI_cnk pthread rt stdc++
Vectorization

VECTORIZATION resulted on internal compiler error. 

############################################
####  Vectorization 
#### 
#### This created in vectors.h (ToReal macro)
#### from thorn LSUThorns/Vectors 
#### /bgxlC_r: 1501-230 (S) Internal compiler error; please contact your Service 
############################################
## VECTORISE                          = yes
## VECTORISE_ALIGNED_ARRAYS           = no
## VECTORISE_ALWAYS_USE_ALIGNED_LOADS = yes
## VECTORISE_INLINE                   = no
############################################
VECTORISE                          = no
VECTORISE_ALIGNED_ARRAYS           = no
VECTORISE_ALWAYS_USE_ALIGNED_LOADS = no
VECTORISE_INLINE                   = no
############################################
MPI

Compiling on fermi present that between the March 2011 version till early 2012 there is a difference on the way is are set the MPI options. This result on different way of define the configuration variables

OLD VERSION: 

MPI          = CUSTOM

NEW VERSION 

### MPI          = CUSTOM

The other settings are unchanged 

MPI_DIR      = /bgsys/drivers/V1R1M1/ppc64/comm/xl.ndebug/
MPI_INC_DIRS = /bgsys/drivers/ppcfloor/comm/default/include /bgsys/drivers/ppcfloor/comm/sys/include /bgsys/drivers/V1R1M1/ppc64/spi/include/kernel/cnk /bgsys/drivers/V1R1M1/ppc64/comm/xl.ndebug/include
MPI_LIB_DIRS = /bgsys/drivers/V1R1M1/ppc64/comm/xl.ndebug/lib /bgsys/drivers/V1R1M1/ppc64/comm/sys-fast/lib /bgsys/drivers/V1R1M1/ppc64/spi/lib
MPI_LIBS     = cxxmpich mpich opa mpl    pami   SPI SPI_cnk
hwloc

The new version of Cactus has a main problem on "Fermi". The plain compilation will create an executable that result on the errors: 

2012-12-14 13:44:52.417 (FATAL) [0x40000c99050] :18315:ibm.runjob.client.Job: could not start job: job failed to start
2012-12-14 13:44:52.417 (FATAL) [0x40000c99050] :18315:ibm.runjob.client.Job: Load failed on R31-ID-J04: Generating static TLB map for appl

Cineca support say that this error is due to: "lrunjob non e' in grado di allocare tutta la memoria richiesta dall'eseguibile." Any easy way to take out this error is to remove the support for the library "hwloc".

CarpetLib_setmemlimit: Assertion `_val' failed.

Compiling with March 2011 source tree results on the above error. The assertion is generated in "void CarpetLib_setmemlimit (CCTK_ARGUMENTS)" of file Carpet/CarpetLib/src/mem.cc (line 274). Solution: Take out lines: 

// {                                                                                                                                             
//   check (not setrlimit (RLIMIT_AS, & aslimit));                                                                                               
// }
The March 2011 version problem with CARPET IOASCII (ON FERMI)

The setting of the IO ascii has a BUG specifying the info for "out_fileinfo". In particular if "all" is specified then the simulation end with "2013-02-06 08:21:24.990 (WARN ) [0x40001059050] LL13020608132090:111206:ibm.runjob.client.Job: terminated by signal 6" and the assert error "cactus_MinimalBarWhisky: /gpfs/scratch/userexternal/rdepietr/SRC/WhiskyMar11/Cactus/configs/MinimalBarWhisky/build/CarpetIOScalar/ioscalar.cc:388: CarpetIOScalar::OutputVarAs: Assertion `file.good()' failed."

The options "parameter filename" and "axis labels" do works. 

KEYWORD out_fileinfo "Add some useful file information to output files ?" STEERABLE = RECOVER
{
  "none"               :: "no file information"
  "creation date"      :: "add creation date"
  "parameter filename" :: "add parameter filename"
  "axis labels"        :: "add axis labels information to output files"
  "all"                :: "add all available file information"
} "all"
Orsted restrict problem ML_BSSN

We did check on the motion of star on the grid with ML_BSSN. We checked the following possibility: 

(o2)  CXX_OPTIMISE_FLAGS = -O2 -qsmp=noauto:noostls   
(NR)  CXXFLAGS = ...... -qnokeyword=restrict -qalias=norestric
      CXX_OPTIMISE_FLAGS = -O3 -qspill=100000 -qmaxmem=-1 -qhot -qlibmpi
(CC)  CXXFLAGS = ...... -qnokeyword=restrict -qalias=norestric
      CXX_OPTIMISE_FLAGS = -O3 -qspill=100000 -qmaxmem=-1 -qhot -qlibmpi
      cctk_cv_cxx_restrict=no

We observed a different behavior of the the moving of the grid with respect to what happen on INTEL machine. Same compilation tree code.

EXECUTING on FERMI

The fermi system submission is based on the loadlever batch system and the main submition command is "llsubmit <JOB.cmd>

Here is an example of a command file to lunch pure MPI on 2048 cores: 

#!/bin/bash
#
# @ job_name = JOBNAME.$(jobid)
# @ output = z.JOBNAME.$(jobid).out
# @ error  = z.JOBNAME.$(jobid).err
# @ shell = /bin/bash
# @ job_type = bluegene
# @ wall_clock_limit = 2:30:00
# @ bg_size = 128
# @ class = parallel
# @ account_no = INFN_OG51
# @ queue

#### (name of par FILE) ==> PARFILE (without .par extension) 
#### 4      ==> NT number of openMP threads
#### 512  ==> NP number of NPI processes
#### 128  ==> BLUGENE size of the run
export NT=1
export NP=2048
export PAROLEE=xCARPET_RL3_dx0.400it32
export EXE=/gpfs/scratch/userexternal/rdepietr/SRC/Orsted/Cactus/exe/cactus_PlainOerstead
export PAR=/gpfs/scratch/userexternal/rdepietr/SpeedTest/ParFiles/${PARFILE}.par
export WDIR=/gpfs/scratch/userexternal/rdepietr/SpeedTest/Runs/${PARFILE}/128_np${NP}_nt${NT}/
export RANKSperNODE=$((16/${NT}))
echo "================================="
echo "Executable is : ${EXE}" 
echo "NP is         : ${NP}" 
echo "NT is         : ${NT}"
echo "rank per node : ${RANKSperNODE}"
echo "Blue gene size is: xBGsize"
echo "Total Number of treads is: $((${NP}*${NT}))"
echo "Total Number of BLUE GENE: $((128*${RANKSoperNODE}*${NT}))"
echo "PAR file is   : ${PAR}" 
echo "Working Dir is: ${WDIR}" 
echo "================================="
mkdir -p  ${WDIR}
cd ${WDIR}
pwd
ls 
echo "================================="
# -mode SMP
# -mode DUAL
# -mode VN
# -np `${nodes}*${taskspernode}`
echo runjob --np ${NP} --ranks-per-node $RANKSperNODE --envs OMP_NUM_THREADS=4 --exe $EXE --args ${PAR}
echo "**************************************************" 
echo "**(TIME)  START: " $(date)
echo "**************************************************" 
export start=$(date +"%s" )
runjob --np ${NP} --ranks-per-node  $RANKSperNODE --envs OMP_NUM_THREADS=4 --exe $EXE --args ${PAR} | tee OUTPUT
export end=$(date +"%s" )
export timeSEC=$(($end - $start))
echo "**************************************************" 
echo "**(TIME)   STOP: " $(date)
echo "**************************************************" 
echo "**(WALL TIME) SEC:" $timeSEC
echo "**************************************************"

SIMFACTORY on FERMI

Just to start 

> SIM setup

resulted in 

Determining local machine name: fen08.fermi.cineca.it
Creating machine fen08.fermi.cineca.it from generic: machine fen08.fermi.cineca.it
[/fermi/home/userexternal/rdepietr/DEV/Cactus/simfactory/mdb/machines/fen08.fermi.cineca.it.ini] created successfully
enter value for key user [rdepietr]:  
enter value for key email [rdepietr]: roberto.depietri@unipr.it
enter value for key allocation []: INFN_OG51
enter value for key sourcebasedir (the parent directory containing the Cactus sourcetree) [/fermi/home/userexternal/rdepietr/DEV]: 
enter value for key basedir (the location of simfactory simulations) [/fermi/home/userexternal/rdepietr/simulations]: /gpfs/scratch/userexternal/rdepietr/simulations
would you like to enter key/value pairs for a specific machine? [Y/N*]: 
--------------------SUMMARY---------------------[default]
user            = rdepietr
email           = roberto.depietri@unipr.it
allocation      = INFN_OG51
sourcebasedir   = /fermi/home/userexternal/rdepietr/DEV
basedir         = /gpfs/scratch/userexternal/rdepietr/simulations
------------------END SUMMARY------------------:
Save contents [Y*/N]: 
Contents successfully written to /fermi/home/userexternal/rdepietr/DEV/Cactus/simfactory/etc/defs.local.ini

To properly define the setting I decided to name the system as BlueGeneQfermi

roberto.depietri/user/cactus.txt · Ultima modifica: 05/12/2013 04:52 da roberto.depietri

Strumenti Pagina

Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki