UNIVERSITÀ DI PARMA
DIPARTIMENTO DI SCIENZE MATEMATICHE, FISICHE E INFORMATICHE

Questa pagina è in sola lettura. Puoi visualizzare il sorgente, ma non puoi modificarlo. Contatta l'amministratore se pensi che ci sia un errore.
====== How to set up CACTUS ======

Our usage setting are based on the latest release of the [[http://einsteintoolkit.org|Einstein toolkit]] and we manage our simulation
using [[http://simfactory.org|The Simulation Factory]] and we will keep to have all the information updated.




===== SETTING KEPLER =====

Setting the base directory 

  mkdir -p /home/kepler/${USER}/ET/
  mkdir -p /home/kepler/${USER}/simulations

  cd /home/kepler/${USER}/ET/
  svn --username=depietri co https://einstein.pr.infn.it/svn//PRV/WEB//PRIVATE/UsefulFiles/ UsefulFiles

setting the directory for the various compilation tree

  mkdir -p /home/kepler/${USER}/ET/whisky11
  mkdir -p /home/kepler/${USER}/ET/dev
  mkdir -p /home/kepler/${USER}/ET/ET_2013_11
  mkdir -p /home/kepler/${USER}/ET/ET_2012_11

==== Get the dev tree working ====
  
  cd  /home/kepler/${USER}/ET/dev
  curl -kO https://raw.github.com/gridaphobe/CRL/master/GetComponents 
  chmod a+x GetComponents
  ./GetComponents --parallel https://svn.einsteintoolkit.org/manifest/trunk/einsteintoolkit.th
  cd Cactus

Setting SIMFACTORY

  cd  /home/kepler/${USER}/ET/dev/Cactus
  # ** SIMFACTORY ****
  # -------------------
  cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.cfg simfactory/mdb/optionlists/
  cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.ini simfactory/mdb/machines/
  cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.run simfactory/mdb/runscripts/
  cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.sub simfactory/mdb/submitscripts/
  # -------------------
  sed "s/whisky11/dev/" ../../UsefulFiles/SimFactory/defs.local.ini > simfactory/etc/defs.local.ini 
  emacs -nw simfactory/etc/defs.local.ini 
  cp ../../UsefulFiles/Thornlists/ET_2013_11.th thornlists/
  cp  ../../UsefulFiles/Thornlists/EXE_MinimalBarFrank_dev.th  thornlists/EXE_MinimalBarFrank.th

=== Example of standard build ===

  cd  /home/kepler/${USER}/ET/dev/Cactus
  make Test-config options=simfactory/mdb/optionlists/kepler.pr.infn.it.cfg
  make -j 8 Test   #One should answer: no in order to confirm the Thornlist#


=== Example of SIMFACTORY build ===

The executable for BarMode with Frank

  cd  /home/kepler/${USER}/ET/dev/Cactus
  simfactory/bin/sim build MinimalBarFrank --thornlist thornlists/EXE_MinimalBarFrank.th 

The full ET distributions

  # -------------------
  cd  /home/kepler/${USER}/ET/dev/Cactus
  simfactory/bin/sim build ET_2013_11 --thornlist thornlists/ET_2013_11.th
  simfactory/bin/sim create-run static_tov --configuration ET_2013_11 --parfile=par/static_tov.par --num-threads=4 --procs=8

For QUEEN BEE all the same after having sync and logged in the machine. 

First one has to have the par file up-to-date. For example:

  cd  /home/kepler/${USER}/ET/dev/Cactus
  #### Copy the par file generator
  mkdir par/MyProject/
  cp  ~/SVN_parma/Examples/Par/rnsid_MAR.rpar par/MyProject/
  cd par/MyProject/
  ./rnsid_MAR.rpar
  cd  /home/kepler/${USER}/ET/dev/Cactus
  simfactory/bin/sim sync queenbee

To run executable for the BarMode work now login on the queen bee machine

  #
  #  Now the run command
  #
  simfactory/bin/sim create A100M10b200_MAR_r50 --parfile=par/MyProject/rnsid_MAR.rpar.dir/A100M10b200_MAR_r50.par --configuration MinimalBarFrank
  simfactory/bin/sim submit --procs=128  A100M10b200_MAR_r50  #got: PBS -l nodes=16:ppn=8 (that requires ~29 Gbyte of RAM)
  ## -we got INFO (Carpet): Carpet is running on 16 processes
  ##         INFO (Carpet): This process contains 8 threads, this is thread 0
  ##         
  ##         Memory statistics from CarpetLib: Current allocated memory:  2027.857 MB
  ## A total of 128 processors on 16 nodes allocated
  ## PBS -l nodes=16:ppn=8
  ## *me_per_hour = ~49 meaning around 6ms simulated per-day

=== SIMFACTORY setting for the FERMI machine ===

  cd  /home/kepler/${USER}/ET/dev/Cactus
  # ** SIMFACTORY ****
  # -------------------
  cp ../../UsefulFiles/SimFactory/fermi.cfg simfactory/mdb/optionlists/
  cp ../../UsefulFiles/SimFactory/fermi.ini simfactory/mdb/machines/
  cp ../../UsefulFiles/SimFactory/fermi.run simfactory/mdb/runscripts/
  cp ../../UsefulFiles/SimFactory/fermi.sub simfactory/mdb/submitscripts/
  # -------------------
  simfactory/bin/sim login fermi

Once you login on fermi with no problem (eventually creating the request directory)

  simfactory/bin/sim sync fermi
  simfactory/bin/sim build ET_2013_11 --thornlist thornlists/ET_2013_11.th  

  simfactory/bin/sim list-simulations
  simfactory/bin/sim show-output   A100M10b200_MAR_r50



==== Get the ET_2013_11 tree working ====

   mkdir -p /home/kepler/${USER}/ET/ET_2013_11
   cd /home/kepler/${USER}/ET/ET_2013_11
   # -------------------
   curl -kO https://raw.github.com/gridaphobe/CRL/master/GetComponents 
   chmod a+x GetComponents
   ./GetComponents --parallel https://svn.einsteintoolkit.org/manifest/branches/ET_2013_11/einsteintoolkit.th
   ./GetComponents  --noanonymous  ../UsefulFiles/Thornlists/ParmaThorns.th
   # -------------------
   cd /home/kepler/${USER}/ET/ET_2013_11/Cactus
   # -------------------
   cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.cfg simfactory/mdb/optionlists/
   cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.ini simfactory/mdb/machines/
   cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.run simfactory/mdb/runscripts/
   cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.sub simfactory/mdb/submitscripts/
   # -------------------
   sed "s/whisky11/ET_2013_11/" ../../UsefulFiles/SimFactory/defs.local.ini > simfactory/etc/defs.local.ini 
   simfactory/bin/sim build --thornlist manifest/einsteintoolkit.th
   # now there is a running executable: "exe/cactus_sim"

   The rest as usual (i.e., as it is for the  "dev" tree)

==== Get the ET_2012_11 tree working ====

   mkdir -p /home/kepler/${USER}/ET/ET_2012_11
   cd /home/kepler/${USER}/ET/ET_2012_11
   # -------------------
   curl -kO https://raw.github.com/gridaphobe/CRL/master/GetComponents 
   chmod a+x GetComponents
   curl -kO https://svn.einsteintoolkit.org/manifest/branches/ET_2012_11/einsteintoolkit.th
   ./GetComponents --parallel einsteintoolkit_2012_11.th 
   ./GetComponents  --noanonymous  ../UsefulFiles/Thornlists/ParmaThorns.th
   # -------------------
   cd /home/kepler/${USER}/ET/ET_2012_11/Cactus
   svn checkout https://svn.cct.lsu.edu/repos/numrel/simfactory2/trunk simfactory
   # -------------------
   cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.cfg simfactory/mdb/optionlists/
   cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.ini simfactory/mdb/machines/
   cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.run simfactory/mdb/runscripts/
   cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.sub simfactory/mdb/submitscripts/
   # -------------------
   sed "s/whisky11/ET_2012_11/" ../../UsefulFiles/SimFactory/defs.local.ini > simfactory/etc/defs.local.ini
   cp  ../../UsefulFiles/Thornlists/EXE_MinimalBarFrank_Orsted.th  thornlists/EXE_MinimalBarFrank.th
   # -------------------
   cd  /home/kepler/${USER}/ET/ET_2012_11/Cactus
   #### Copy the par file generator
   mkdir par/MyProject/
   cp  ~/SVN_parma/Examples/Par/rnsid_MAR_Orstead.rpar par/MyProject/rnsid_MAR.rpar
   cd par/MyProject/
   ./rnsid_MAR.rpar
   cd  /home/kepler/${USER}/ET/ET_2012_11/Cactus

   simfactory/bin/sim build MinimalBarFrank --thornlist thornlists/EXE_MinimalBarFrank.th

   The rest as usual (i.e., as it is for the  "dev" tree)


==== Get the whisky11 tree working ====

  cd  /home/kepler/${USER}/ET/whisky11
  # --- copy our original thorns
  tar xzvf /numrel/storageQ01/MagneticBar/CactusTree_WhiskyEXP_March11.tgz    
  find . -name .svn | awk '{print "rm -rf " $1}'
  # --- GET our new thorns
  curl -kO https://raw.github.com/gridaphobe/CRL/master/GetComponents 
  chmod a+x GetComponents
  ./GetComponents  --noanonymous  ../UsefulFiles/Thornlists/ParmaThorns.th

Setting SIMFACTORY

  cd Cactus/
  # **
  # ** SIMFACTORY ****
  # **
  # --- Get the new version of simfactory
  # **
  mv simfactory/ old_simfactory
  svn checkout https://svn.cct.lsu.edu/repos/numrel/simfactory2/trunk simfactory
  # **
  # -------------------
  # **
  cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.cfg simfactory/mdb/optionlists/
  cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.ini simfactory/mdb/machines/
  cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.run simfactory/mdb/runscripts/
  cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.sub simfactory/mdb/submitscripts/
  # **
  # -------------------
  # **
  ########echo "HDF5=yes"                                     >>  simfactory/mdb/optionlists/kepler.pr.infn.it.cfg
  echo "MPI=CUSTOM"                                           >>  simfactory/mdb/optionlists/kepler.pr.infn.it.cfg
  echo "MPI_INC_DIRS=/opt/ExternalLibraryGNU/openmpi/include" >>  simfactory/mdb/optionlists/kepler.pr.infn.it.cfg
  echo "MPI_LIB_DIRS=/opt/ExternalLibraryGNU/openmpi/lib"     >>  simfactory/mdb/optionlists/kepler.pr.infn.it.cfg
  echo "MPI_LIBS=mpi_cxx  util  numa  mpi"                    >>  simfactory/mdb/optionlists/kepler.pr.infn.it.cfg 
  # **
  # **
  sed "s/-L 3/-L 1/" simfactory/mdb/runscripts/queenbee-mvapich2.run > TMP
  mv TMP simfactory/mdb/runscripts/mike-openmpi.run
  sed "s/-L 3/-L 1/" simfactory/mdb/runscripts/mike-openmpi.run> TMP
  mv TMP simfactory/mdb/runscripts/mike-openmpi.run
  # **
  # correct queen bee to use old interface
  # **
  echo "MPI=CUSTOM"        >>  simfactory/mdb/optionlists/queenbee-mvapich2.cfg 
  echo "MPI=CUSTOM"        >>  simfactory/mdb/optionlists/mike-openmpi.cfg
  # **
  # -------------------
  # **
  cp ../../UsefulFiles/SimFactory/defs.local.ini simfactory/etc
  emacs -nw simfactory/etc/defs.local.ini 
   # **

=== Example of standard build ===

  cd  /home/kepler/${USER}/ET/whisky11/Cactus
  cp ../../UsefulFiles/Thornlists/EXE_BarModeMar11.th thornlists/
  make whisky11-config options=simfactory/mdb/optionlists/kepler.pr.infn.it.cfg THORNLIST=../../UsefulFiles/Thornlists/EXE_BarModeMar11.th 
  make whisky11

=== Example of SIMFACTORY build ===

  cd  /home/kepler/${USER}/ET/whisky11/Cactus
  simfactory/bin/sim build whisky11 --thornlist thornlists/EXE_BarModeMar11.th

(to check MPI=OpenMPI OPENMPI_DIR=/opt/ExternalLibraryGNU/openmpi -lmpi_cxx  -lutil  -lnuma  -lmpi)

=== Example of run on queen bee ===

  simfactory/bin/sim create S01b1_14_r80 --parfile=par/MyProject/MagWhisky.rpar.dir/S01b1_14_r80.par --configuration whisky11
  simfactory/bin/sim create S01b1_14_r40 --parfile=par/MyProject/MagWhisky.rpar.dir/S01b1_14_r40.par --configuration whisky11
  simfactory/bin/sim create S01b1_14_r20 --parfile=par/MyProject/MagWhisky.rpar.dir/S01b1_14_r20.par --configuration whisky11
  simfactory/bin/sim submit --procs=32   --num-threads=1  S01b1_14_r80  #got: PBS -l nodes=4:ppn=8
  simfactory/bin/sim submit --procs=128  --num-threads=1  S01b1_14_r40
  simfactory/bin/sim submit --procs=1024 --num-threads=1  S01b1_14_r20
  simfactory/bin/sim show-output   S01b1_14_r80

First one has to have the par file up-to-date. For example:

   cd  /home/kepler/${USER}/ET/whisky11/Cactus
   #### Copy the par file generator
   mkdir par/MyProject/
   cp  ~/SVN_parma/Examples/Par/MagWhisky.rpar  par/MyProject/
   cd par/MyProject/
   ./MagWhisky.rpar 
   cd  /home/kepler/${USER}/ET/whisky11/Cactus

Now I can check it on kepler!

  simfactory/bin/sim create S01b1_14_r80 --parfile=par/MyProject/MagWhisky.rpar.dir/S01b1_14_r80.par --configuration whisky11
  simfactory/bin/sim run S01b1_14_r80 --num-threads=1 --procs=8

===== HPC MACHINE available to the group =====

We got access to the following allocation on HPC machine:

  * Fermi SYSTEM 

    ssh l rdepietr login.fermi.cineca.it

  * [[https://allocations.loni.org/balances.php|Loni]] queenbee at LSU (loni_numrel08)

     ssh -l depietri qb3.loni.org

  * [[https://accounts.hpc.lsu.edu/profile.php|HPC]] mike2 at LSU (hpc_hyrel13)

     ssh -l depietri mike1.hpc.lsu.edu

  * [[https://portal.xsede.org/my-xsede|Xsede]] Kraken, Stampede (TG-MCA02N014)

====== OLD NOTES (before November the 15th, 2013) ======

We do have a tree on our WEB server where we store useful configuration and file
needed to better run our simulations. They can be get making a local SVN version

  svn --username=depietri co https://einstein.pr.infn.it/svn//PRV/WEB//PRIVATE/UsefulFiles/ UsefulFiles

From here a common tree I usually create subtree for the various version of cactus that I am using.
Typically, a very old version of Whisky, the current version of the Einstein Toolkit and the 
developer tree.   


===== Download CACTUS (Orsted) =====

The first step is to download the main cactus components from the [[http://einsteintoolkit.org|EinsteinTollkit]]
repository. We also need the following ThornLists (AEIThorns.th, ParmaThorns.th, Whisky_DEV.th, Whisky_EXP.th, Whisky_PUB.th) that can be downloaded from our PRIVATE web pages [[https://einstein.pr.infn.it/PRIVATE/UsefulFiles/Thornlists/|ThornLists]]
First you have to go the Directory where you would like to install CACTUS. And then you will have to
execute the following commands: (they will create a working Cactus tree ready to be compiled)

  curl -O https://raw.github.com/gridaphobe/CRL/master/GetComponents 
  chmod a+x GetComponents
  ./GetComponents --parallel https://svn.einsteintoolkit.org/manifest/branches/ET_2012_11/einsteintoolkit.th
  ./GetComponents  --noanonymous  ./Whisky_DEV.th
  ./GetComponents  --noanonymous  ./Whisky_EXP.th
  ./GetComponents  --noanonymous  ./AEIThorns.th
  ./GetComponents  --noanonymous  ./ParmaThorns.th
 

===== Download CACTUS (Developer TREE) =====

  curl -O https://raw.github.com/gridaphobe/CRL/master/GetComponents 
  chmod a+x GetComponents
  ./GetComponents --parallel https://svn.einsteintoolkit.org/manifest/trunk/einsteintoolkit.th

Now Our local arrangements...

  ./GetComponents  --noanonymous  ../Orsted/Whisky_DEV.th
  ./GetComponents  --noanonymous  ../Orsted/Whisky_EXP.th
  ./GetComponents  --noanonymous  ../Orsted/AEIThorns.th
  ./GetComponents  --noanonymous  ../Orsted/ParmaThorns.th




===== Our old Version (Just not to get mixed up) =====

    tar xzvf /storageQ01/MagneticBar/CactusTree_WhiskyEXP_March11.tgz    
    find . -name .svn | awk '{print "rm -rf " $1}'
    curl -O https://raw.github.com/gridaphobe/CRL/master/GetComponents 
    chmod a+x GetComponents
    ./GetComponents  --noanonymous  ../UsefulFiles/Thornlists/ParmaThorns.th
    svn checkout https://svn.cct.lsu.edu/repos/numrel/simfactory2/trunk simfactory


I obtained (on Einstein) an executable suitable for our work on Magnetic Bar with the command
"make BarModeMar11-config options=../../UsefulFiles/SimFactory/einstein.pr.infn.it.cfg 
THORNLIST=../../UsefulFiles/Thornlists/EXE_BarModeMar11.th MPI=OpenMPI OPENMPI_DIR=/usr/lib64/openmpi/1.4-gcc/"
 



===== Set up SIMFACTORY =====

Simfactory to work need that the local machine we use should be defined. This is done
properly setting the file "simfactory/etc/defs.local.ini" and for any machine we 
would like to use to provide in:

  [simfactory/mdb/machines/] the machine definitions (eg. SYSTEMNAME.ini)
  [simfactory/mdb/optionlist/] the machine compilation options (eg. SYSTEMNAME.cgf)


For example we may give the commands:

   cp ../../UsefulFiles/SimFactory/*.ini simfactory/mdb/machines/
   cp ../../UsefulFiles/SimFactory/*.cfg simfactory/mdb/optionlists/
   mv simfactory/mdb/machines/defs.local.ini simfactory/etc/
   emacs -nw simfactory/etc/defs.local.ini

Example of all these file can be downloaded from: [[https://einstein.pr.infn.it/PRIVATE/UsefulFiles/SimFactory/|SIMFACTORY]]
It is also very useful to define the following alias that would simplify the use of simfactory //"alias SIM=simfactory/bin/sim"//


   ./simfactory/bin/sim create  S01b1_14_r40 --parfile=par/S01b1_14_r40.par --configuration BarModeMar11
   ./simfactory/bin/sim submit --procs=256  --num-threads=1  S01b1_14_r40
   ./simfactory/bin/sim show-output   S01b1_14_r40


===== COMPILING on FERMI =====

To compile on fermi (once SIM factory is properly set):
 
  SIM sync  fermiBGQ
  SIM login fermiBGQ
  SIM build PlainOerstead  --thornlist thornlists/PlainOerstead.th 

That should be enough to create a working executable.

=== General setting for cactus compilation ===

  CPP = cpp
  FPP = cpp
  CC  = /opt/ibmcmp/vacpp/bg/12.1/bin/bgxlc_r
  CXX = /opt/ibmcmp/vacpp/bg/12.1/bin/bgxlC_r
  F77 = /opt/ibmcmp/xlf/bg/14.1/bin/bgxlf_r
  F90 = /opt/ibmcmp/xlf/bg/14.1/bin/bgxlf90_r
  C_DEPEND   = $(CPP) -M $(CPPFLAGS)
  CXX_DEPEND = $(CPP) -M $(CPPFLAGS)
  #CPPFLAGS = -DMPICH_IGNORE_CXX_SEEK
  CPPFLAGS = 
  FPPFLAGS = -traditional
  CFLAGS   = -qarch=qp -qtune=qp -qsuppress=1506-1108:1506-1197 -qlanglvl=stdc99
  CXXFLAGS = -qarch=qp -qtune=qp -qsuppress=1506-1108 -qrtti
  F77FLAGS = -qarch=qp -qtune=qp -qsuppress=1520-022:1501-510:cmpmsg -qextname
  F90FLAGS = -qarch=qp -qtune=qp -qsuppress=1520-022:1501-510:cmpmsg -qextname
  ####  -qextname (This option preserve the trailing underscore on external  
  ####  -qextname ... fortran linkange .... needed to mix up fortran and C code)
  LDFLAGS = -Wl,-relax
  SYS_INC_DIRS =
  LIBDIRS      =  /opt/ibmcmp/xlf/bg/14.1/bglib64/ 
  LIBS         = xlf90_r xlfmath SPI SPI_cnk pthread rt stdc++   
  #######----------------------------------------------------------
  DEBUG           = no
  CPP_DEBUG_FLAGS = -g -DCARPET_DEBUG
  FPP_DEBUG_FLAGS = -g -DCARPET_DEBUG
  C_DEBUG_FLAGS   = -g -O0 -qnoipa -qcheck=bounds
  CXX_DEBUG_FLAGS = -g -O0 -qnoipa -qcheck=bounds
  F77_DEBUG_FLAGS = -g -O0 -qnoipa -qcheck
  F90_DEBUG_FLAGS = -g -O0 -qnoipa -qcheck
  #######----------------------------------------------------------
  # Try: -qhot=simd
  # -O3 for C takes >30min to compile ML_BSSN
  # -O3 for C++ and Fortran crashes with SIGFPE
  # -qipa leads to undefined functions while linking
  OPTIMISE           = yes
  CPP_OPTIMISE_FLAGS = # -DCARPET_OPTIMISE -DNDEBUG
  FPP_OPTIMISE_FLAGS = # -DCARPET_OPTIMISE -DNDEBUG
  C_OPTIMISE_FLAGS   = -O3 -qspill=100000 -qmaxmem=-1 -qhot -qlibmpi
  CXX_OPTIMISE_FLAGS = -O3 -qspill=100000 -qmaxmem=-1 -qhot -qlibmpi
  F77_OPTIMISE_FLAGS = -O3 -qspill=100000 -qmaxmem=-1 -qhot -qlibmpi
  F90_OPTIMISE_FLAGS = -O3 -qspill=100000 -qmaxmem=-1 -qhot -qlibmpi
  #######----------------------------------------------------------
  PROFILE           = no
  CPP_PROFILE_FLAGS =
  FPP_PROFILE_FLAGS =
  C_PROFILE_FLAGS   = -pg
  CXX_PROFILE_FLAGS = -pg
  F77_PROFILE_FLAGS = -pg
  F90_PROFILE_FLAGS = -pg
  #######----------------------------------------------------------
  OPENMP           = yes
  CPP_OPENMP_FLAGS = 
  FPP_OPENMP_FLAGS = -D_OPENMP
  C_OPENMP_FLAGS   = -qsmp=omp
  CXX_OPENMP_FLAGS = -qsmp=omp
  F77_OPENMP_FLAGS = -qsmp=omp
  F90_OPENMP_FLAGS = -qsmp=omp
  #######----------------------------------------------------------
  WARN           = yes
  CPP_WARN_FLAGS =
  FPP_WARN_FLAGS =
  C_WARN_FLAGS   =
  CXX_WARN_FLAGS =
  F77_WARN_FLAGS =
  F90_WARN_FLAGS = 
  #######----------------------------------------------------------
  ##  /bgsys/drivers/ppcfloor/comm/xl.ndebug/bin/mpixlcxx_r  -show 
  ## mpixlcxx_r -show 
  #######----------------------------------------------------------
  MPI_DIR      = /bgsys/drivers/V1R1M1/ppc64/comm/xl.ndebug/
  MPI_INC_DIRS = /bgsys/drivers/ppcfloor/comm/default/include /bgsys/drivers/ppcfloor/comm/sys/include /bgsys/drivers/V1R1M1/ppc64/spi/include/kernel/cnk /bgsys/drivers/V1R1M1/ppc64/comm/xl.ndebug/include
  MPI_LIB_DIRS = /bgsys/drivers/V1R1M1/ppc64/comm/xl.ndebug/lib /bgsys/drivers/V1R1M1/ppc64/comm/sys-fast/lib /bgsys/drivers/V1R1M1/ppc64/spi/lib
  MPI_LIBS     = cxxmpich mpich opa mpl    pami   SPI SPI_cnk
  #######----------------------------------------------------------
  PTHREADS = yes
  #######----------------------------------------------------------
  BLAS_DIR    = /gpfs/scratch/userexternal/rdepietr/external/BLAS
  LAPACK_DIR  = /gpfs/scratch/userexternal/rdepietr/external/LAPACK
  LORENE_DIR  =  /gpfs/scratch/userexternal/rdepietr/external/LORENE
  #######----------------------------------------------------------
  GSL_DIR = /cineca/prod/libraries/gsl/1.15/bgq-xl--1.0
  HDF5_DIR = /cineca/prod/libraries/hdf5/1.8.9_ser/bgq-xl--1.0
  ZLIB_DIR = /cineca/prod/libraries/zlib/1.2.7/bgq-gnu--4.4.6
  LIBSZ_DIR = /cineca/prod/libraries/szip/2.1/bgq-xl--1.0/lib
  LIBJPEG_DIR = /cineca/prod/libraries/libjpeg/8d/bgq-gnu--4.4.6
  #######----------------------------------------------------------
  FFTW3_DIR   = /cineca/prod/libraries/fftw/3.3.2/bgq-xl--1.0
  FFTW3_LIBS  = fftw3f
  #######----------------------------------------------------------
  ##  May need a patch to FFTW3.sh in "ExternalLibraries/FFTW3" 
  #######----------------------------------------------------------
  #######----------------------------------------------------------




=== Know problems ===

== ZliB not found ==

Compiling the old tree the linker is not able to find libz.a. I simple workaround is to manually add to the default linker search path the directory where the library is located

  LIBDIRS=/opt/ibmcmp/xlf/bg/14.1/lib64/ /cineca/prod/libraries/zlib/1.2.7/bgq-gnu--4.4.6/lib  /cineca/prod/libraries/szip/2.1/bgq-xl--1.0/lib 
  LIBS= xlf90_r xlfmath SPI SPI_cnk pthread rt stdc++ z sz

instead of 

  LIBDIRS=/opt/ibmcmp/xlf/bg/14.1/lib64/
  LIBS= xlf90_r xlfmath SPI SPI_cnk pthread rt stdc++  

== Vectorization ==

VECTORIZATION resulted on internal compiler error. 

  ############################################
  ####  Vectorization 
  #### 
  #### This created in vectors.h (ToReal macro)
  #### from thorn LSUThorns/Vectors 
  #### /bgxlC_r: 1501-230 (S) Internal compiler error; please contact your Service 
  ############################################
  ## VECTORISE                          = yes
  ## VECTORISE_ALIGNED_ARRAYS           = no
  ## VECTORISE_ALWAYS_USE_ALIGNED_LOADS = yes
  ## VECTORISE_INLINE                   = no
  ############################################
  VECTORISE                          = no
  VECTORISE_ALIGNED_ARRAYS           = no
  VECTORISE_ALWAYS_USE_ALIGNED_LOADS = no
  VECTORISE_INLINE                   = no
  ############################################


== MPI ==

Compiling on fermi present that between the March 2011 version till early 2012 there is a difference on the way is are set the MPI options. This result on different way of define the configuration variables

OLD VERSION:
  MPI          = CUSTOM
NEW VERSION
  ### MPI          = CUSTOM

The other settings are unchanged

  MPI_DIR      = /bgsys/drivers/V1R1M1/ppc64/comm/xl.ndebug/
  MPI_INC_DIRS = /bgsys/drivers/ppcfloor/comm/default/include /bgsys/drivers/ppcfloor/comm/sys/include /bgsys/drivers/V1R1M1/ppc64/spi/include/kernel/cnk /bgsys/drivers/V1R1M1/ppc64/comm/xl.ndebug/include
  MPI_LIB_DIRS = /bgsys/drivers/V1R1M1/ppc64/comm/xl.ndebug/lib /bgsys/drivers/V1R1M1/ppc64/comm/sys-fast/lib /bgsys/drivers/V1R1M1/ppc64/spi/lib
  MPI_LIBS     = cxxmpich mpich opa mpl    pami   SPI SPI_cnk

== hwloc == 
  
The new version of Cactus has a  main problem on "Fermi". The plain compilation will create an executable that result on the errors:

  2012-12-14 13:44:52.417 (FATAL) [0x40000c99050] :18315:ibm.runjob.client.Job: could not start job: job failed to start
  2012-12-14 13:44:52.417 (FATAL) [0x40000c99050] :18315:ibm.runjob.client.Job: Load failed on R31-ID-J04: Generating static TLB map for appl 

Cineca support say that this error is due to: "lrunjob non e' in grado di allocare tutta la memoria richiesta dall'eseguibile." Any easy way to take out this error is to remove the support for the library "hwloc".

== CarpetLib_setmemlimit: Assertion `_val' failed. ==

Compiling with March 2011 source tree results on the above error.  The assertion is generated in "void CarpetLib_setmemlimit (CCTK_ARGUMENTS)" of file Carpet/CarpetLib/src/mem.cc (line 274). Solution: Take out lines:

  // {                                                                                                                                             
  //   check (not setrlimit (RLIMIT_AS, & aslimit));                                                                                               
  // }                                                                                                                                             


== The March 2011 version problem with CARPET IOASCII  (ON FERMI) ==

The setting of the IO ascii has a BUG specifying the info for //"out_fileinfo"//. In particular 
if **"all"** is specified then the simulation end with 
"2013-02-06 08:21:24.990 (WARN ) [0x40001059050] LL13020608132090:111206:ibm.runjob.client.Job: terminated by signal 6"
and the assert error "cactus_MinimalBarWhisky: /gpfs/scratch/userexternal/rdepietr/SRC/WhiskyMar11/Cactus/configs/MinimalBarWhisky/build/CarpetIOScalar/ioscalar.cc:388: CarpetIOScalar::OutputVarAs: Assertion `file.good()' failed."

The options **"parameter filename"** and **"axis labels"** do works.

  KEYWORD out_fileinfo "Add some useful file information to output files ?" STEERABLE = RECOVER
  {
    "none"               :: "no file information"
    "creation date"      :: "add creation date"
    "parameter filename" :: "add parameter filename"
    "axis labels"        :: "add axis labels information to output files"
    "all"                :: "add all available file information"
  } "all"

== Orsted restrict problem ML_BSSN ==

We did check on the motion of star on the grid with ML_BSSN. We checked the following possibility:


  (o2)  CXX_OPTIMISE_FLAGS = -O2 -qsmp=noauto:noostls   
  (NR)  CXXFLAGS = ...... -qnokeyword=restrict -qalias=norestric
        CXX_OPTIMISE_FLAGS = -O3 -qspill=100000 -qmaxmem=-1 -qhot -qlibmpi
  (CC)  CXXFLAGS = ...... -qnokeyword=restrict -qalias=norestric
        CXX_OPTIMISE_FLAGS = -O3 -qspill=100000 -qmaxmem=-1 -qhot -qlibmpi
        cctk_cv_cxx_restrict=no

We observed a different behavior of the the moving of the grid with respect to what happen on INTEL machine. Same compilation tree code.

===== EXECUTING on FERMI =====


The fermi system submission is based on the loadlever batch system and the main submition command is "llsubmit <JOB.cmd> 

Here is an example of a command file to lunch pure MPI on 2048 cores:

  #!/bin/bash
  #
  # @ job_name = JOBNAME.$(jobid)
  # @ output = z.JOBNAME.$(jobid).out
  # @ error  = z.JOBNAME.$(jobid).err
  # @ shell = /bin/bash
  # @ job_type = bluegene
  # @ wall_clock_limit = 2:30:00
  # @ bg_size = 128
  # @ class = parallel
  # @ account_no = INFN_OG51
  # @ queue
  
  #### (name of par FILE) ==> PARFILE (without .par extension) 
  #### 4      ==> NT number of openMP threads
  #### 512  ==> NP number of NPI processes
  #### 128  ==> BLUGENE size of the run
  export NT=1
  export NP=2048
  export PAROLEE=xCARPET_RL3_dx0.400it32
  export EXE=/gpfs/scratch/userexternal/rdepietr/SRC/Orsted/Cactus/exe/cactus_PlainOerstead
  export PAR=/gpfs/scratch/userexternal/rdepietr/SpeedTest/ParFiles/${PARFILE}.par
  export WDIR=/gpfs/scratch/userexternal/rdepietr/SpeedTest/Runs/${PARFILE}/128_np${NP}_nt${NT}/
  export RANKSperNODE=$((16/${NT}))
  echo "================================="
  echo "Executable is : ${EXE}" 
  echo "NP is         : ${NP}" 
  echo "NT is         : ${NT}"
  echo "rank per node : ${RANKSperNODE}"
  echo "Blue gene size is: xBGsize"
  echo "Total Number of treads is: $((${NP}*${NT}))"
  echo "Total Number of BLUE GENE: $((128*${RANKSoperNODE}*${NT}))"
  echo "PAR file is   : ${PAR}" 
  echo "Working Dir is: ${WDIR}" 
  echo "================================="
  mkdir -p  ${WDIR}
  cd ${WDIR}
  pwd
  ls 
  echo "================================="
  # -mode SMP
  # -mode DUAL
  # -mode VN
  # -np `${nodes}*${taskspernode}`
  echo runjob --np ${NP} --ranks-per-node $RANKSperNODE --envs OMP_NUM_THREADS=4 --exe $EXE --args ${PAR}
  echo "**************************************************" 
  echo "**(TIME)  START: " $(date)
  echo "**************************************************" 
  export start=$(date +"%s" )
  runjob --np ${NP} --ranks-per-node  $RANKSperNODE --envs OMP_NUM_THREADS=4 --exe $EXE --args ${PAR} | tee OUTPUT
  export end=$(date +"%s" )
  export timeSEC=$(($end - $start))
  echo "**************************************************" 
  echo "**(TIME)   STOP: " $(date)
  echo "**************************************************" 
  echo "**(WALL TIME) SEC:" $timeSEC
  echo "**************************************************" 



====== SIMFACTORY on FERMI ======

Just to start 

  > SIM setup

resulted in 

  Determining local machine name: fen08.fermi.cineca.it
  Creating machine fen08.fermi.cineca.it from generic: machine fen08.fermi.cineca.it
  [/fermi/home/userexternal/rdepietr/DEV/Cactus/simfactory/mdb/machines/fen08.fermi.cineca.it.ini] created successfully
  enter value for key user [rdepietr]:  
  enter value for key email [rdepietr]: roberto.depietri@unipr.it
  enter value for key allocation []: INFN_OG51
  enter value for key sourcebasedir (the parent directory containing the Cactus sourcetree) [/fermi/home/userexternal/rdepietr/DEV]: 
  enter value for key basedir (the location of simfactory simulations) [/fermi/home/userexternal/rdepietr/simulations]: /gpfs/scratch/userexternal/rdepietr/simulations
  would you like to enter key/value pairs for a specific machine? [Y/N*]: 
  --------------------SUMMARY---------------------[default]
  user            = rdepietr
  email           = roberto.depietri@unipr.it
  allocation      = INFN_OG51
  sourcebasedir   = /fermi/home/userexternal/rdepietr/DEV
  basedir         = /gpfs/scratch/userexternal/rdepietr/simulations
  ------------------END SUMMARY------------------:
  Save contents [Y*/N]: 
  Contents successfully written to /fermi/home/userexternal/rdepietr/DEV/Cactus/simfactory/etc/defs.local.ini

To properly define the setting I decided to name the system as  BlueGeneQfermi
UNIVERSITÀ DI PARMADIPARTIMENTO DI SCIENZE MATEMATICHE, FISICHE E INFORMATICHE

Strumenti Utente

Strumenti Sito

Strumenti Pagina

UNIVERSITÀ DI PARMA
DIPARTIMENTO DI SCIENZE MATEMATICHE, FISICHE E INFORMATICHE
