Strumenti Utente

Strumenti Sito


roberto.depietri:user:cactus

Differenze

Queste sono le differenze tra la revisione selezionata e la versione attuale della pagina.

Link a questa pagina di confronto

Entrambe le parti precedenti la revisioneRevisione precedente
Prossima revisione
Revisione precedente
roberto.depietri:user:cactus [22/11/2013 10:43] roberto.depietriroberto.depietri:user:cactus [05/12/2013 04:52] (versione attuale) roberto.depietri
Linea 1: Linea 1:
 +====== How to set up CACTUS ======
  
 +Our usage setting are based on the latest release of the [[http://einsteintoolkit.org|Einstein toolkit]] and we manage our simulation
 +using [[http://simfactory.org|The Simulation Factory]] and we will keep to have all the information updated.
 +
 +
 +
 +
 +===== SETTING KEPLER =====
 +
 +Setting the base directory 
 +
 +  mkdir -p /home/kepler/${USER}/ET/
 +  mkdir -p /home/kepler/${USER}/simulations
 +
 +  cd /home/kepler/${USER}/ET/
 +  svn --username=depietri co https://einstein.pr.infn.it/svn//PRV/WEB//PRIVATE/UsefulFiles/ UsefulFiles
 +
 +setting the directory for the various compilation tree
 +
 +  mkdir -p /home/kepler/${USER}/ET/whisky11
 +  mkdir -p /home/kepler/${USER}/ET/dev
 +  mkdir -p /home/kepler/${USER}/ET/ET_2013_11
 +  mkdir -p /home/kepler/${USER}/ET/ET_2012_11
 +
 +==== Get the dev tree working ====
 +  
 +  cd  /home/kepler/${USER}/ET/dev
 +  curl -kO https://raw.github.com/gridaphobe/CRL/master/GetComponents 
 +  chmod a+x GetComponents
 +  ./GetComponents --parallel https://svn.einsteintoolkit.org/manifest/trunk/einsteintoolkit.th
 +  cd Cactus
 +
 +Setting SIMFACTORY
 +
 +  cd  /home/kepler/${USER}/ET/dev/Cactus
 +  # ** SIMFACTORY ****
 +  # -------------------
 +  cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.cfg simfactory/mdb/optionlists/
 +  cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.ini simfactory/mdb/machines/
 +  cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.run simfactory/mdb/runscripts/
 +  cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.sub simfactory/mdb/submitscripts/
 +  # -------------------
 +  sed "s/whisky11/dev/" ../../UsefulFiles/SimFactory/defs.local.ini > simfactory/etc/defs.local.ini 
 +  emacs -nw simfactory/etc/defs.local.ini 
 +  cp ../../UsefulFiles/Thornlists/ET_2013_11.th thornlists/
 +  cp  ../../UsefulFiles/Thornlists/EXE_MinimalBarFrank_dev.th  thornlists/EXE_MinimalBarFrank.th
 +
 +=== Example of standard build ===
 +
 +  cd  /home/kepler/${USER}/ET/dev/Cactus
 +  make Test-config options=simfactory/mdb/optionlists/kepler.pr.infn.it.cfg
 +  make -j 8 Test   #One should answer: no in order to confirm the Thornlist#
 +
 +
 +=== Example of SIMFACTORY build ===
 +
 +The executable for BarMode with Frank
 +
 +  cd  /home/kepler/${USER}/ET/dev/Cactus
 +  simfactory/bin/sim build MinimalBarFrank --thornlist thornlists/EXE_MinimalBarFrank.th 
 +
 +The full ET distributions
 +
 +  # -------------------
 +  cd  /home/kepler/${USER}/ET/dev/Cactus
 +  simfactory/bin/sim build ET_2013_11 --thornlist thornlists/ET_2013_11.th
 +  simfactory/bin/sim create-run static_tov --configuration ET_2013_11 --parfile=par/static_tov.par --num-threads=4 --procs=8
 +
 +For QUEEN BEE all the same after having sync and logged in the machine. 
 +
 +First one has to have the par file up-to-date. For example:
 +
 +  cd  /home/kepler/${USER}/ET/dev/Cactus
 +  #### Copy the par file generator
 +  mkdir par/MyProject/
 +  cp  ~/SVN_parma/Examples/Par/rnsid_MAR.rpar par/MyProject/
 +  cd par/MyProject/
 +  ./rnsid_MAR.rpar
 +  cd  /home/kepler/${USER}/ET/dev/Cactus
 +  simfactory/bin/sim sync queenbee
 +
 +To run executable for the BarMode work now login on the queen bee machine
 +
 +  #
 +  #  Now the run command
 +  #
 +  simfactory/bin/sim create A100M10b200_MAR_r50 --parfile=par/MyProject/rnsid_MAR.rpar.dir/A100M10b200_MAR_r50.par --configuration MinimalBarFrank
 +  simfactory/bin/sim submit --procs=128  A100M10b200_MAR_r50  #got: PBS -l nodes=16:ppn=8 (that requires ~29 Gbyte of RAM)
 +  ## -we got INFO (Carpet): Carpet is running on 16 processes
 +  ##         INFO (Carpet): This process contains 8 threads, this is thread 0
 +  ##         
 +  ##         Memory statistics from CarpetLib: Current allocated memory:  2027.857 MB
 +  ## A total of 128 processors on 16 nodes allocated
 +  ## PBS -l nodes=16:ppn=8
 +  ## *me_per_hour = ~49 meaning around 6ms simulated per-day
 +
 +=== SIMFACTORY setting for the FERMI machine ===
 +
 +  cd  /home/kepler/${USER}/ET/dev/Cactus
 +  # ** SIMFACTORY ****
 +  # -------------------
 +  cp ../../UsefulFiles/SimFactory/fermi.cfg simfactory/mdb/optionlists/
 +  cp ../../UsefulFiles/SimFactory/fermi.ini simfactory/mdb/machines/
 +  cp ../../UsefulFiles/SimFactory/fermi.run simfactory/mdb/runscripts/
 +  cp ../../UsefulFiles/SimFactory/fermi.sub simfactory/mdb/submitscripts/
 +  # -------------------
 +  simfactory/bin/sim login fermi
 +
 +Once you login on fermi with no problem (eventually creating the request directory)
 +
 +  simfactory/bin/sim sync fermi
 +  simfactory/bin/sim build ET_2013_11 --thornlist thornlists/ET_2013_11.th  
 +
 +  simfactory/bin/sim list-simulations
 +  simfactory/bin/sim show-output   A100M10b200_MAR_r50
 +
 +
 +
 +==== Get the ET_2013_11 tree working ====
 +
 +   mkdir -p /home/kepler/${USER}/ET/ET_2013_11
 +   cd /home/kepler/${USER}/ET/ET_2013_11
 +   # -------------------
 +   curl -kO https://raw.github.com/gridaphobe/CRL/master/GetComponents 
 +   chmod a+x GetComponents
 +   ./GetComponents --parallel https://svn.einsteintoolkit.org/manifest/branches/ET_2013_11/einsteintoolkit.th
 +   ./GetComponents  --noanonymous  ../UsefulFiles/Thornlists/ParmaThorns.th
 +   # -------------------
 +   cd /home/kepler/${USER}/ET/ET_2013_11/Cactus
 +   # -------------------
 +   cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.cfg simfactory/mdb/optionlists/
 +   cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.ini simfactory/mdb/machines/
 +   cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.run simfactory/mdb/runscripts/
 +   cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.sub simfactory/mdb/submitscripts/
 +   # -------------------
 +   sed "s/whisky11/ET_2013_11/" ../../UsefulFiles/SimFactory/defs.local.ini > simfactory/etc/defs.local.ini 
 +   simfactory/bin/sim build --thornlist manifest/einsteintoolkit.th
 +   # now there is a running executable: "exe/cactus_sim"
 +
 +   The rest as usual (i.e., as it is for the  "dev" tree)
 +
 +==== Get the ET_2012_11 tree working ====
 +
 +   mkdir -p /home/kepler/${USER}/ET/ET_2012_11
 +   cd /home/kepler/${USER}/ET/ET_2012_11
 +   # -------------------
 +   curl -kO https://raw.github.com/gridaphobe/CRL/master/GetComponents 
 +   chmod a+x GetComponents
 +   curl -kO https://svn.einsteintoolkit.org/manifest/branches/ET_2012_11/einsteintoolkit.th
 +   ./GetComponents --parallel einsteintoolkit_2012_11.th 
 +   ./GetComponents  --noanonymous  ../UsefulFiles/Thornlists/ParmaThorns.th
 +   # -------------------
 +   cd /home/kepler/${USER}/ET/ET_2012_11/Cactus
 +   svn checkout https://svn.cct.lsu.edu/repos/numrel/simfactory2/trunk simfactory
 +   # -------------------
 +   cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.cfg simfactory/mdb/optionlists/
 +   cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.ini simfactory/mdb/machines/
 +   cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.run simfactory/mdb/runscripts/
 +   cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.sub simfactory/mdb/submitscripts/
 +   # -------------------
 +   sed "s/whisky11/ET_2012_11/" ../../UsefulFiles/SimFactory/defs.local.ini > simfactory/etc/defs.local.ini
 +   cp  ../../UsefulFiles/Thornlists/EXE_MinimalBarFrank_Orsted.th  thornlists/EXE_MinimalBarFrank.th
 +   # -------------------
 +   cd  /home/kepler/${USER}/ET/ET_2012_11/Cactus
 +   #### Copy the par file generator
 +   mkdir par/MyProject/
 +   cp  ~/SVN_parma/Examples/Par/rnsid_MAR_Orstead.rpar par/MyProject/rnsid_MAR.rpar
 +   cd par/MyProject/
 +   ./rnsid_MAR.rpar
 +   cd  /home/kepler/${USER}/ET/ET_2012_11/Cactus
 +
 +   simfactory/bin/sim build MinimalBarFrank --thornlist thornlists/EXE_MinimalBarFrank.th
 +
 +   The rest as usual (i.e., as it is for the  "dev" tree)
 +
 +
 +==== Get the whisky11 tree working ====
 +
 +  cd  /home/kepler/${USER}/ET/whisky11
 +  # --- copy our original thorns
 +  tar xzvf /numrel/storageQ01/MagneticBar/CactusTree_WhiskyEXP_March11.tgz    
 +  find . -name .svn | awk '{print "rm -rf " $1}'
 +  # --- GET our new thorns
 +  curl -kO https://raw.github.com/gridaphobe/CRL/master/GetComponents 
 +  chmod a+x GetComponents
 +  ./GetComponents  --noanonymous  ../UsefulFiles/Thornlists/ParmaThorns.th
 +
 +Setting SIMFACTORY
 +
 +  cd Cactus/
 +  # **
 +  # ** SIMFACTORY ****
 +  # **
 +  # --- Get the new version of simfactory
 +  # **
 +  mv simfactory/ old_simfactory
 +  svn checkout https://svn.cct.lsu.edu/repos/numrel/simfactory2/trunk simfactory
 +  # **
 +  # -------------------
 +  # **
 +  cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.cfg simfactory/mdb/optionlists/
 +  cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.ini simfactory/mdb/machines/
 +  cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.run simfactory/mdb/runscripts/
 +  cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.sub simfactory/mdb/submitscripts/
 +  # **
 +  # -------------------
 +  # **
 +  ########echo "HDF5=yes"                                     >>  simfactory/mdb/optionlists/kepler.pr.infn.it.cfg
 +  echo "MPI=CUSTOM"                                           >>  simfactory/mdb/optionlists/kepler.pr.infn.it.cfg
 +  echo "MPI_INC_DIRS=/opt/ExternalLibraryGNU/openmpi/include" >>  simfactory/mdb/optionlists/kepler.pr.infn.it.cfg
 +  echo "MPI_LIB_DIRS=/opt/ExternalLibraryGNU/openmpi/lib"     >>  simfactory/mdb/optionlists/kepler.pr.infn.it.cfg
 +  echo "MPI_LIBS=mpi_cxx  util  numa  mpi"                    >>  simfactory/mdb/optionlists/kepler.pr.infn.it.cfg 
 +  # **
 +  # **
 +  sed "s/-L 3/-L 1/" simfactory/mdb/runscripts/queenbee-mvapich2.run > TMP
 +  mv TMP simfactory/mdb/runscripts/mike-openmpi.run
 +  sed "s/-L 3/-L 1/" simfactory/mdb/runscripts/mike-openmpi.run> TMP
 +  mv TMP simfactory/mdb/runscripts/mike-openmpi.run
 +  # **
 +  # correct queen bee to use old interface
 +  # **
 +  echo "MPI=CUSTOM"        >>  simfactory/mdb/optionlists/queenbee-mvapich2.cfg 
 +  echo "MPI=CUSTOM"        >>  simfactory/mdb/optionlists/mike-openmpi.cfg
 +  # **
 +  # -------------------
 +  # **
 +  cp ../../UsefulFiles/SimFactory/defs.local.ini simfactory/etc
 +  emacs -nw simfactory/etc/defs.local.ini 
 +   # **
 +
 +=== Example of standard build ===
 +
 +  cd  /home/kepler/${USER}/ET/whisky11/Cactus
 +  cp ../../UsefulFiles/Thornlists/EXE_BarModeMar11.th thornlists/
 +  make whisky11-config options=simfactory/mdb/optionlists/kepler.pr.infn.it.cfg THORNLIST=../../UsefulFiles/Thornlists/EXE_BarModeMar11.th 
 +  make whisky11
 +
 +=== Example of SIMFACTORY build ===
 +
 +  cd  /home/kepler/${USER}/ET/whisky11/Cactus
 +  simfactory/bin/sim build whisky11 --thornlist thornlists/EXE_BarModeMar11.th
 +
 +(to check MPI=OpenMPI OPENMPI_DIR=/opt/ExternalLibraryGNU/openmpi -lmpi_cxx  -lutil  -lnuma  -lmpi)
 +
 +=== Example of run on queen bee ===
 +
 +  simfactory/bin/sim create S01b1_14_r80 --parfile=par/MyProject/MagWhisky.rpar.dir/S01b1_14_r80.par --configuration whisky11
 +  simfactory/bin/sim create S01b1_14_r40 --parfile=par/MyProject/MagWhisky.rpar.dir/S01b1_14_r40.par --configuration whisky11
 +  simfactory/bin/sim create S01b1_14_r20 --parfile=par/MyProject/MagWhisky.rpar.dir/S01b1_14_r20.par --configuration whisky11
 +  simfactory/bin/sim submit --procs=32   --num-threads=1  S01b1_14_r80  #got: PBS -l nodes=4:ppn=8
 +  simfactory/bin/sim submit --procs=128  --num-threads=1  S01b1_14_r40
 +  simfactory/bin/sim submit --procs=1024 --num-threads=1  S01b1_14_r20
 +  simfactory/bin/sim show-output   S01b1_14_r80
 +
 +First one has to have the par file up-to-date. For example:
 +
 +   cd  /home/kepler/${USER}/ET/whisky11/Cactus
 +   #### Copy the par file generator
 +   mkdir par/MyProject/
 +   cp  ~/SVN_parma/Examples/Par/MagWhisky.rpar  par/MyProject/
 +   cd par/MyProject/
 +   ./MagWhisky.rpar 
 +   cd  /home/kepler/${USER}/ET/whisky11/Cactus
 +
 +Now I can check it on kepler!
 +
 +  simfactory/bin/sim create S01b1_14_r80 --parfile=par/MyProject/MagWhisky.rpar.dir/S01b1_14_r80.par --configuration whisky11
 +  simfactory/bin/sim run S01b1_14_r80 --num-threads=1 --procs=8
 +
 +===== HPC MACHINE available to the group =====
 +
 +We got access to the following allocation on HPC machine:
 +
 +  * Fermi SYSTEM 
 +
 +    ssh l rdepietr login.fermi.cineca.it
 +
 +  * [[https://allocations.loni.org/balances.php|Loni]] queenbee at LSU (loni_numrel08)
 +
 +     ssh -l depietri qb3.loni.org
 +
 +  * [[https://accounts.hpc.lsu.edu/profile.php|HPC]] mike2 at LSU (hpc_hyrel13)
 +
 +     ssh -l depietri mike1.hpc.lsu.edu
 +
 +  * [[https://portal.xsede.org/my-xsede|Xsede]] Kraken, Stampede (TG-MCA02N014)
 +
 +====== OLD NOTES (before November the 15th, 2013) ======
 +
 +We do have a tree on our WEB server where we store useful configuration and file
 +needed to better run our simulations. They can be get making a local SVN version
 +
 +  svn --username=depietri co https://einstein.pr.infn.it/svn//PRV/WEB//PRIVATE/UsefulFiles/ UsefulFiles
 +
 +From here a common tree I usually create subtree for the various version of cactus that I am using.
 +Typically, a very old version of Whisky, the current version of the Einstein Toolkit and the 
 +developer tree.   
 +
 +
 +===== Download CACTUS (Orsted) =====
 +
 +The first step is to download the main cactus components from the [[http://einsteintoolkit.org|EinsteinTollkit]]
 +repository. We also need the following ThornLists (AEIThorns.th, ParmaThorns.th, Whisky_DEV.th, Whisky_EXP.th, Whisky_PUB.th) that can be downloaded from our PRIVATE web pages [[https://einstein.pr.infn.it/PRIVATE/UsefulFiles/Thornlists/|ThornLists]]
 +First you have to go the Directory where you would like to install CACTUS. And then you will have to
 +execute the following commands: (they will create a working Cactus tree ready to be compiled)
 +
 +  curl -O https://raw.github.com/gridaphobe/CRL/master/GetComponents 
 +  chmod a+x GetComponents
 +  ./GetComponents --parallel https://svn.einsteintoolkit.org/manifest/branches/ET_2012_11/einsteintoolkit.th
 +  ./GetComponents  --noanonymous  ./Whisky_DEV.th
 +  ./GetComponents  --noanonymous  ./Whisky_EXP.th
 +  ./GetComponents  --noanonymous  ./AEIThorns.th
 +  ./GetComponents  --noanonymous  ./ParmaThorns.th
 + 
 +
 +===== Download CACTUS (Developer TREE) =====
 +
 +  curl -O https://raw.github.com/gridaphobe/CRL/master/GetComponents 
 +  chmod a+x GetComponents
 +  ./GetComponents --parallel https://svn.einsteintoolkit.org/manifest/trunk/einsteintoolkit.th
 +
 +Now Our local arrangements...
 +
 +  ./GetComponents  --noanonymous  ../Orsted/Whisky_DEV.th
 +  ./GetComponents  --noanonymous  ../Orsted/Whisky_EXP.th
 +  ./GetComponents  --noanonymous  ../Orsted/AEIThorns.th
 +  ./GetComponents  --noanonymous  ../Orsted/ParmaThorns.th
 +
 +
 +
 +
 +===== Our old Version (Just not to get mixed up) =====
 +
 +    tar xzvf /storageQ01/MagneticBar/CactusTree_WhiskyEXP_March11.tgz    
 +    find . -name .svn | awk '{print "rm -rf " $1}'
 +    curl -O https://raw.github.com/gridaphobe/CRL/master/GetComponents 
 +    chmod a+x GetComponents
 +    ./GetComponents  --noanonymous  ../UsefulFiles/Thornlists/ParmaThorns.th
 +    svn checkout https://svn.cct.lsu.edu/repos/numrel/simfactory2/trunk simfactory
 +
 +
 +I obtained (on Einstein) an executable suitable for our work on Magnetic Bar with the command
 +"make BarModeMar11-config options=../../UsefulFiles/SimFactory/einstein.pr.infn.it.cfg 
 +THORNLIST=../../UsefulFiles/Thornlists/EXE_BarModeMar11.th MPI=OpenMPI OPENMPI_DIR=/usr/lib64/openmpi/1.4-gcc/"
 + 
 +
 +
 +
 +===== Set up SIMFACTORY =====
 +
 +Simfactory to work need that the local machine we use should be defined. This is done
 +properly setting the file "simfactory/etc/defs.local.ini" and for any machine we 
 +would like to use to provide in:
 +
 +  [simfactory/mdb/machines/] the machine definitions (eg. SYSTEMNAME.ini)
 +  [simfactory/mdb/optionlist/] the machine compilation options (eg. SYSTEMNAME.cgf)
 +
 +
 +For example we may give the commands:
 +
 +   cp ../../UsefulFiles/SimFactory/*.ini simfactory/mdb/machines/
 +   cp ../../UsefulFiles/SimFactory/*.cfg simfactory/mdb/optionlists/
 +   mv simfactory/mdb/machines/defs.local.ini simfactory/etc/
 +   emacs -nw simfactory/etc/defs.local.ini
 +
 +Example of all these file can be downloaded from: [[https://einstein.pr.infn.it/PRIVATE/UsefulFiles/SimFactory/|SIMFACTORY]]
 +It is also very useful to define the following alias that would simplify the use of simfactory //"alias SIM=simfactory/bin/sim"//
 +
 +
 +   ./simfactory/bin/sim create  S01b1_14_r40 --parfile=par/S01b1_14_r40.par --configuration BarModeMar11
 +   ./simfactory/bin/sim submit --procs=256  --num-threads=1  S01b1_14_r40
 +   ./simfactory/bin/sim show-output   S01b1_14_r40
 +
 +
 +===== COMPILING on FERMI =====
 +
 +To compile on fermi (once SIM factory is properly set):
 + 
 +  SIM sync  fermiBGQ
 +  SIM login fermiBGQ
 +  SIM build PlainOerstead  --thornlist thornlists/PlainOerstead.th 
 +
 +That should be enough to create a working executable.
 +
 +=== General setting for cactus compilation ===
 +
 +  CPP = cpp
 +  FPP = cpp
 +  CC  = /opt/ibmcmp/vacpp/bg/12.1/bin/bgxlc_r
 +  CXX = /opt/ibmcmp/vacpp/bg/12.1/bin/bgxlC_r
 +  F77 = /opt/ibmcmp/xlf/bg/14.1/bin/bgxlf_r
 +  F90 = /opt/ibmcmp/xlf/bg/14.1/bin/bgxlf90_r
 +  C_DEPEND   = $(CPP) -M $(CPPFLAGS)
 +  CXX_DEPEND = $(CPP) -M $(CPPFLAGS)
 +  #CPPFLAGS = -DMPICH_IGNORE_CXX_SEEK
 +  CPPFLAGS = 
 +  FPPFLAGS = -traditional
 +  CFLAGS   = -qarch=qp -qtune=qp -qsuppress=1506-1108:1506-1197 -qlanglvl=stdc99
 +  CXXFLAGS = -qarch=qp -qtune=qp -qsuppress=1506-1108 -qrtti
 +  F77FLAGS = -qarch=qp -qtune=qp -qsuppress=1520-022:1501-510:cmpmsg -qextname
 +  F90FLAGS = -qarch=qp -qtune=qp -qsuppress=1520-022:1501-510:cmpmsg -qextname
 +  ####  -qextname (This option preserve the trailing underscore on external  
 +  ####  -qextname ... fortran linkange .... needed to mix up fortran and C code)
 +  LDFLAGS = -Wl,-relax
 +  SYS_INC_DIRS =
 +  LIBDIRS      =  /opt/ibmcmp/xlf/bg/14.1/bglib64/ 
 +  LIBS         = xlf90_r xlfmath SPI SPI_cnk pthread rt stdc++   
 +  #######----------------------------------------------------------
 +  DEBUG           = no
 +  CPP_DEBUG_FLAGS = -g -DCARPET_DEBUG
 +  FPP_DEBUG_FLAGS = -g -DCARPET_DEBUG
 +  C_DEBUG_FLAGS   = -g -O0 -qnoipa -qcheck=bounds
 +  CXX_DEBUG_FLAGS = -g -O0 -qnoipa -qcheck=bounds
 +  F77_DEBUG_FLAGS = -g -O0 -qnoipa -qcheck
 +  F90_DEBUG_FLAGS = -g -O0 -qnoipa -qcheck
 +  #######----------------------------------------------------------
 +  # Try: -qhot=simd
 +  # -O3 for C takes >30min to compile ML_BSSN
 +  # -O3 for C++ and Fortran crashes with SIGFPE
 +  # -qipa leads to undefined functions while linking
 +  OPTIMISE           = yes
 +  CPP_OPTIMISE_FLAGS = # -DCARPET_OPTIMISE -DNDEBUG
 +  FPP_OPTIMISE_FLAGS = # -DCARPET_OPTIMISE -DNDEBUG
 +  C_OPTIMISE_FLAGS   = -O3 -qspill=100000 -qmaxmem=-1 -qhot -qlibmpi
 +  CXX_OPTIMISE_FLAGS = -O3 -qspill=100000 -qmaxmem=-1 -qhot -qlibmpi
 +  F77_OPTIMISE_FLAGS = -O3 -qspill=100000 -qmaxmem=-1 -qhot -qlibmpi
 +  F90_OPTIMISE_FLAGS = -O3 -qspill=100000 -qmaxmem=-1 -qhot -qlibmpi
 +  #######----------------------------------------------------------
 +  PROFILE           = no
 +  CPP_PROFILE_FLAGS =
 +  FPP_PROFILE_FLAGS =
 +  C_PROFILE_FLAGS   = -pg
 +  CXX_PROFILE_FLAGS = -pg
 +  F77_PROFILE_FLAGS = -pg
 +  F90_PROFILE_FLAGS = -pg
 +  #######----------------------------------------------------------
 +  OPENMP           = yes
 +  CPP_OPENMP_FLAGS = 
 +  FPP_OPENMP_FLAGS = -D_OPENMP
 +  C_OPENMP_FLAGS   = -qsmp=omp
 +  CXX_OPENMP_FLAGS = -qsmp=omp
 +  F77_OPENMP_FLAGS = -qsmp=omp
 +  F90_OPENMP_FLAGS = -qsmp=omp
 +  #######----------------------------------------------------------
 +  WARN           = yes
 +  CPP_WARN_FLAGS =
 +  FPP_WARN_FLAGS =
 +  C_WARN_FLAGS   =
 +  CXX_WARN_FLAGS =
 +  F77_WARN_FLAGS =
 +  F90_WARN_FLAGS = 
 +  #######----------------------------------------------------------
 +  ##  /bgsys/drivers/ppcfloor/comm/xl.ndebug/bin/mpixlcxx_r  -show 
 +  ## mpixlcxx_r -show 
 +  #######----------------------------------------------------------
 +  MPI_DIR      = /bgsys/drivers/V1R1M1/ppc64/comm/xl.ndebug/
 +  MPI_INC_DIRS = /bgsys/drivers/ppcfloor/comm/default/include /bgsys/drivers/ppcfloor/comm/sys/include /bgsys/drivers/V1R1M1/ppc64/spi/include/kernel/cnk /bgsys/drivers/V1R1M1/ppc64/comm/xl.ndebug/include
 +  MPI_LIB_DIRS = /bgsys/drivers/V1R1M1/ppc64/comm/xl.ndebug/lib /bgsys/drivers/V1R1M1/ppc64/comm/sys-fast/lib /bgsys/drivers/V1R1M1/ppc64/spi/lib
 +  MPI_LIBS     = cxxmpich mpich opa mpl    pami   SPI SPI_cnk
 +  #######----------------------------------------------------------
 +  PTHREADS = yes
 +  #######----------------------------------------------------------
 +  BLAS_DIR    = /gpfs/scratch/userexternal/rdepietr/external/BLAS
 +  LAPACK_DIR  = /gpfs/scratch/userexternal/rdepietr/external/LAPACK
 +  LORENE_DIR  =  /gpfs/scratch/userexternal/rdepietr/external/LORENE
 +  #######----------------------------------------------------------
 +  GSL_DIR = /cineca/prod/libraries/gsl/1.15/bgq-xl--1.0
 +  HDF5_DIR = /cineca/prod/libraries/hdf5/1.8.9_ser/bgq-xl--1.0
 +  ZLIB_DIR = /cineca/prod/libraries/zlib/1.2.7/bgq-gnu--4.4.6
 +  LIBSZ_DIR = /cineca/prod/libraries/szip/2.1/bgq-xl--1.0/lib
 +  LIBJPEG_DIR = /cineca/prod/libraries/libjpeg/8d/bgq-gnu--4.4.6
 +  #######----------------------------------------------------------
 +  FFTW3_DIR   = /cineca/prod/libraries/fftw/3.3.2/bgq-xl--1.0
 +  FFTW3_LIBS  = fftw3f
 +  #######----------------------------------------------------------
 +  ##  May need a patch to FFTW3.sh in "ExternalLibraries/FFTW3" 
 +  #######----------------------------------------------------------
 +  #######----------------------------------------------------------
 +
 +
 +
 +
 +=== Know problems ===
 +
 +== ZliB not found ==
 +
 +Compiling the old tree the linker is not able to find libz.a. I simple workaround is to manually add to the default linker search path the directory where the library is located
 +
 +  LIBDIRS=/opt/ibmcmp/xlf/bg/14.1/lib64/ /cineca/prod/libraries/zlib/1.2.7/bgq-gnu--4.4.6/lib  /cineca/prod/libraries/szip/2.1/bgq-xl--1.0/lib 
 +  LIBS= xlf90_r xlfmath SPI SPI_cnk pthread rt stdc++ z sz
 +
 +instead of 
 +
 +  LIBDIRS=/opt/ibmcmp/xlf/bg/14.1/lib64/
 +  LIBS= xlf90_r xlfmath SPI SPI_cnk pthread rt stdc++  
 +
 +== Vectorization ==
 +
 +VECTORIZATION resulted on internal compiler error. 
 +
 +  ############################################
 +  ####  Vectorization 
 +  #### 
 +  #### This created in vectors.h (ToReal macro)
 +  #### from thorn LSUThorns/Vectors 
 +  #### /bgxlC_r: 1501-230 (S) Internal compiler error; please contact your Service 
 +  ############################################
 +  ## VECTORISE                          = yes
 +  ## VECTORISE_ALIGNED_ARRAYS           = no
 +  ## VECTORISE_ALWAYS_USE_ALIGNED_LOADS = yes
 +  ## VECTORISE_INLINE                   = no
 +  ############################################
 +  VECTORISE                          = no
 +  VECTORISE_ALIGNED_ARRAYS           = no
 +  VECTORISE_ALWAYS_USE_ALIGNED_LOADS = no
 +  VECTORISE_INLINE                   = no
 +  ############################################
 +
 +
 +== MPI ==
 +
 +Compiling on fermi present that between the March 2011 version till early 2012 there is a difference on the way is are set the MPI options. This result on different way of define the configuration variables
 +
 +OLD VERSION:
 +  MPI          = CUSTOM
 +NEW VERSION
 +  ### MPI          = CUSTOM
 +
 +The other settings are unchanged
 +
 +  MPI_DIR      = /bgsys/drivers/V1R1M1/ppc64/comm/xl.ndebug/
 +  MPI_INC_DIRS = /bgsys/drivers/ppcfloor/comm/default/include /bgsys/drivers/ppcfloor/comm/sys/include /bgsys/drivers/V1R1M1/ppc64/spi/include/kernel/cnk /bgsys/drivers/V1R1M1/ppc64/comm/xl.ndebug/include
 +  MPI_LIB_DIRS = /bgsys/drivers/V1R1M1/ppc64/comm/xl.ndebug/lib /bgsys/drivers/V1R1M1/ppc64/comm/sys-fast/lib /bgsys/drivers/V1R1M1/ppc64/spi/lib
 +  MPI_LIBS     = cxxmpich mpich opa mpl    pami   SPI SPI_cnk
 +
 +== hwloc == 
 +  
 +The new version of Cactus has a  main problem on "Fermi". The plain compilation will create an executable that result on the errors:
 +
 +  2012-12-14 13:44:52.417 (FATAL) [0x40000c99050] :18315:ibm.runjob.client.Job: could not start job: job failed to start
 +  2012-12-14 13:44:52.417 (FATAL) [0x40000c99050] :18315:ibm.runjob.client.Job: Load failed on R31-ID-J04: Generating static TLB map for appl 
 +
 +Cineca support say that this error is due to: "lrunjob non e' in grado di allocare tutta la memoria richiesta dall'eseguibile." Any easy way to take out this error is to remove the support for the library "hwloc".
 +
 +== CarpetLib_setmemlimit: Assertion `_val' failed. ==
 +
 +Compiling with March 2011 source tree results on the above error.  The assertion is generated in "void CarpetLib_setmemlimit (CCTK_ARGUMENTS)" of file Carpet/CarpetLib/src/mem.cc (line 274). Solution: Take out lines:
 +
 +  // {                                                                                                                                             
 +  //   check (not setrlimit (RLIMIT_AS, & aslimit));                                                                                               
 +  // }                                                                                                                                             
 +
 +
 +== The March 2011 version problem with CARPET IOASCII  (ON FERMI) ==
 +
 +The setting of the IO ascii has a BUG specifying the info for //"out_fileinfo"//. In particular 
 +if **"all"** is specified then the simulation end with 
 +"2013-02-06 08:21:24.990 (WARN ) [0x40001059050] LL13020608132090:111206:ibm.runjob.client.Job: terminated by signal 6"
 +and the assert error "cactus_MinimalBarWhisky: /gpfs/scratch/userexternal/rdepietr/SRC/WhiskyMar11/Cactus/configs/MinimalBarWhisky/build/CarpetIOScalar/ioscalar.cc:388: CarpetIOScalar::OutputVarAs: Assertion `file.good()' failed."
 +
 +The options **"parameter filename"** and **"axis labels"** do works.
 +
 +  KEYWORD out_fileinfo "Add some useful file information to output files ?" STEERABLE = RECOVER
 +  {
 +    "none"               :: "no file information"
 +    "creation date"      :: "add creation date"
 +    "parameter filename" :: "add parameter filename"
 +    "axis labels"        :: "add axis labels information to output files"
 +    "all"                :: "add all available file information"
 +  } "all"
 +
 +== Orsted restrict problem ML_BSSN ==
 +
 +We did check on the motion of star on the grid with ML_BSSN. We checked the following possibility:
 +
 +
 +  (o2)  CXX_OPTIMISE_FLAGS = -O2 -qsmp=noauto:noostls   
 +  (NR)  CXXFLAGS = ...... -qnokeyword=restrict -qalias=norestric
 +        CXX_OPTIMISE_FLAGS = -O3 -qspill=100000 -qmaxmem=-1 -qhot -qlibmpi
 +  (CC)  CXXFLAGS = ...... -qnokeyword=restrict -qalias=norestric
 +        CXX_OPTIMISE_FLAGS = -O3 -qspill=100000 -qmaxmem=-1 -qhot -qlibmpi
 +        cctk_cv_cxx_restrict=no
 +
 +We observed a different behavior of the the moving of the grid with respect to what happen on INTEL machine. Same compilation tree code.
 +
 +===== EXECUTING on FERMI =====
 +
 +
 +The fermi system submission is based on the loadlever batch system and the main submition command is "llsubmit <JOB.cmd> 
 +
 +Here is an example of a command file to lunch pure MPI on 2048 cores:
 +
 +  #!/bin/bash
 +  #
 +  # @ job_name = JOBNAME.$(jobid)
 +  # @ output = z.JOBNAME.$(jobid).out
 +  # @ error  = z.JOBNAME.$(jobid).err
 +  # @ shell = /bin/bash
 +  # @ job_type = bluegene
 +  # @ wall_clock_limit = 2:30:00
 +  # @ bg_size = 128
 +  # @ class = parallel
 +  # @ account_no = INFN_OG51
 +  # @ queue
 +  
 +  #### (name of par FILE) ==> PARFILE (without .par extension) 
 +  #### 4      ==> NT number of openMP threads
 +  #### 512  ==> NP number of NPI processes
 +  #### 128  ==> BLUGENE size of the run
 +  export NT=1
 +  export NP=2048
 +  export PAROLEE=xCARPET_RL3_dx0.400it32
 +  export EXE=/gpfs/scratch/userexternal/rdepietr/SRC/Orsted/Cactus/exe/cactus_PlainOerstead
 +  export PAR=/gpfs/scratch/userexternal/rdepietr/SpeedTest/ParFiles/${PARFILE}.par
 +  export WDIR=/gpfs/scratch/userexternal/rdepietr/SpeedTest/Runs/${PARFILE}/128_np${NP}_nt${NT}/
 +  export RANKSperNODE=$((16/${NT}))
 +  echo "================================="
 +  echo "Executable is : ${EXE}" 
 +  echo "NP is         : ${NP}" 
 +  echo "NT is         : ${NT}"
 +  echo "rank per node : ${RANKSperNODE}"
 +  echo "Blue gene size is: xBGsize"
 +  echo "Total Number of treads is: $((${NP}*${NT}))"
 +  echo "Total Number of BLUE GENE: $((128*${RANKSoperNODE}*${NT}))"
 +  echo "PAR file is   : ${PAR}" 
 +  echo "Working Dir is: ${WDIR}" 
 +  echo "================================="
 +  mkdir -p  ${WDIR}
 +  cd ${WDIR}
 +  pwd
 +  ls 
 +  echo "================================="
 +  # -mode SMP
 +  # -mode DUAL
 +  # -mode VN
 +  # -np `${nodes}*${taskspernode}`
 +  echo runjob --np ${NP} --ranks-per-node $RANKSperNODE --envs OMP_NUM_THREADS=4 --exe $EXE --args ${PAR}
 +  echo "**************************************************" 
 +  echo "**(TIME)  START: " $(date)
 +  echo "**************************************************" 
 +  export start=$(date +"%s" )
 +  runjob --np ${NP} --ranks-per-node  $RANKSperNODE --envs OMP_NUM_THREADS=4 --exe $EXE --args ${PAR} | tee OUTPUT
 +  export end=$(date +"%s" )
 +  export timeSEC=$(($end - $start))
 +  echo "**************************************************" 
 +  echo "**(TIME)   STOP: " $(date)
 +  echo "**************************************************" 
 +  echo "**(WALL TIME) SEC:" $timeSEC
 +  echo "**************************************************" 
 +
 +
 +
 +====== SIMFACTORY on FERMI ======
 +
 +Just to start 
 +
 +  > SIM setup
 +
 +resulted in 
 +
 +  Determining local machine name: fen08.fermi.cineca.it
 +  Creating machine fen08.fermi.cineca.it from generic: machine fen08.fermi.cineca.it
 +  [/fermi/home/userexternal/rdepietr/DEV/Cactus/simfactory/mdb/machines/fen08.fermi.cineca.it.ini] created successfully
 +  enter value for key user [rdepietr]:  
 +  enter value for key email [rdepietr]: roberto.depietri@unipr.it
 +  enter value for key allocation []: INFN_OG51
 +  enter value for key sourcebasedir (the parent directory containing the Cactus sourcetree) [/fermi/home/userexternal/rdepietr/DEV]: 
 +  enter value for key basedir (the location of simfactory simulations) [/fermi/home/userexternal/rdepietr/simulations]: /gpfs/scratch/userexternal/rdepietr/simulations
 +  would you like to enter key/value pairs for a specific machine? [Y/N*]: 
 +  --------------------SUMMARY---------------------[default]
 +  user            = rdepietr
 +  email           = roberto.depietri@unipr.it
 +  allocation      = INFN_OG51
 +  sourcebasedir   = /fermi/home/userexternal/rdepietr/DEV
 +  basedir         = /gpfs/scratch/userexternal/rdepietr/simulations
 +  ------------------END SUMMARY------------------:
 +  Save contents [Y*/N]: 
 +  Contents successfully written to /fermi/home/userexternal/rdepietr/DEV/Cactus/simfactory/etc/defs.local.ini
 +
 +To properly define the setting I decided to name the system as  BlueGeneQfermi
 +
 + 

Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki