====== How to set up CACTUS ====== Our usage setting are based on the latest release of the [[http://einsteintoolkit.org|Einstein toolkit]] and we manage our simulation using [[http://simfactory.org|The Simulation Factory]] and we will keep to have all the information updated. ===== SETTING KEPLER ===== Setting the base directory mkdir -p /home/kepler/${USER}/ET/ mkdir -p /home/kepler/${USER}/simulations cd /home/kepler/${USER}/ET/ svn --username=depietri co https://einstein.pr.infn.it/svn//PRV/WEB//PRIVATE/UsefulFiles/ UsefulFiles setting the directory for the various compilation tree mkdir -p /home/kepler/${USER}/ET/whisky11 mkdir -p /home/kepler/${USER}/ET/dev mkdir -p /home/kepler/${USER}/ET/ET_2013_11 mkdir -p /home/kepler/${USER}/ET/ET_2012_11 ==== Get the dev tree working ==== cd /home/kepler/${USER}/ET/dev curl -kO https://raw.github.com/gridaphobe/CRL/master/GetComponents chmod a+x GetComponents ./GetComponents --parallel https://svn.einsteintoolkit.org/manifest/trunk/einsteintoolkit.th cd Cactus Setting SIMFACTORY cd /home/kepler/${USER}/ET/dev/Cactus # ** SIMFACTORY **** # ------------------- cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.cfg simfactory/mdb/optionlists/ cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.ini simfactory/mdb/machines/ cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.run simfactory/mdb/runscripts/ cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.sub simfactory/mdb/submitscripts/ # ------------------- sed "s/whisky11/dev/" ../../UsefulFiles/SimFactory/defs.local.ini > simfactory/etc/defs.local.ini emacs -nw simfactory/etc/defs.local.ini cp ../../UsefulFiles/Thornlists/ET_2013_11.th thornlists/ cp ../../UsefulFiles/Thornlists/EXE_MinimalBarFrank_dev.th thornlists/EXE_MinimalBarFrank.th === Example of standard build === cd /home/kepler/${USER}/ET/dev/Cactus make Test-config options=simfactory/mdb/optionlists/kepler.pr.infn.it.cfg make -j 8 Test #One should answer: no in order to confirm the Thornlist# === Example of SIMFACTORY build === The executable for BarMode with Frank cd /home/kepler/${USER}/ET/dev/Cactus simfactory/bin/sim build MinimalBarFrank --thornlist thornlists/EXE_MinimalBarFrank.th The full ET distributions # ------------------- cd /home/kepler/${USER}/ET/dev/Cactus simfactory/bin/sim build ET_2013_11 --thornlist thornlists/ET_2013_11.th simfactory/bin/sim create-run static_tov --configuration ET_2013_11 --parfile=par/static_tov.par --num-threads=4 --procs=8 For QUEEN BEE all the same after having sync and logged in the machine. First one has to have the par file up-to-date. For example: cd /home/kepler/${USER}/ET/dev/Cactus #### Copy the par file generator mkdir par/MyProject/ cp ~/SVN_parma/Examples/Par/rnsid_MAR.rpar par/MyProject/ cd par/MyProject/ ./rnsid_MAR.rpar cd /home/kepler/${USER}/ET/dev/Cactus simfactory/bin/sim sync queenbee To run executable for the BarMode work now login on the queen bee machine # # Now the run command # simfactory/bin/sim create A100M10b200_MAR_r50 --parfile=par/MyProject/rnsid_MAR.rpar.dir/A100M10b200_MAR_r50.par --configuration MinimalBarFrank simfactory/bin/sim submit --procs=128 A100M10b200_MAR_r50 #got: PBS -l nodes=16:ppn=8 (that requires ~29 Gbyte of RAM) ## -we got INFO (Carpet): Carpet is running on 16 processes ## INFO (Carpet): This process contains 8 threads, this is thread 0 ## ## Memory statistics from CarpetLib: Current allocated memory: 2027.857 MB ## A total of 128 processors on 16 nodes allocated ## PBS -l nodes=16:ppn=8 ## *me_per_hour = ~49 meaning around 6ms simulated per-day === SIMFACTORY setting for the FERMI machine === cd /home/kepler/${USER}/ET/dev/Cactus # ** SIMFACTORY **** # ------------------- cp ../../UsefulFiles/SimFactory/fermi.cfg simfactory/mdb/optionlists/ cp ../../UsefulFiles/SimFactory/fermi.ini simfactory/mdb/machines/ cp ../../UsefulFiles/SimFactory/fermi.run simfactory/mdb/runscripts/ cp ../../UsefulFiles/SimFactory/fermi.sub simfactory/mdb/submitscripts/ # ------------------- simfactory/bin/sim login fermi Once you login on fermi with no problem (eventually creating the request directory) simfactory/bin/sim sync fermi simfactory/bin/sim build ET_2013_11 --thornlist thornlists/ET_2013_11.th simfactory/bin/sim list-simulations simfactory/bin/sim show-output A100M10b200_MAR_r50 ==== Get the ET_2013_11 tree working ==== mkdir -p /home/kepler/${USER}/ET/ET_2013_11 cd /home/kepler/${USER}/ET/ET_2013_11 # ------------------- curl -kO https://raw.github.com/gridaphobe/CRL/master/GetComponents chmod a+x GetComponents ./GetComponents --parallel https://svn.einsteintoolkit.org/manifest/branches/ET_2013_11/einsteintoolkit.th ./GetComponents --noanonymous ../UsefulFiles/Thornlists/ParmaThorns.th # ------------------- cd /home/kepler/${USER}/ET/ET_2013_11/Cactus # ------------------- cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.cfg simfactory/mdb/optionlists/ cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.ini simfactory/mdb/machines/ cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.run simfactory/mdb/runscripts/ cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.sub simfactory/mdb/submitscripts/ # ------------------- sed "s/whisky11/ET_2013_11/" ../../UsefulFiles/SimFactory/defs.local.ini > simfactory/etc/defs.local.ini simfactory/bin/sim build --thornlist manifest/einsteintoolkit.th # now there is a running executable: "exe/cactus_sim" The rest as usual (i.e., as it is for the "dev" tree) ==== Get the ET_2012_11 tree working ==== mkdir -p /home/kepler/${USER}/ET/ET_2012_11 cd /home/kepler/${USER}/ET/ET_2012_11 # ------------------- curl -kO https://raw.github.com/gridaphobe/CRL/master/GetComponents chmod a+x GetComponents curl -kO https://svn.einsteintoolkit.org/manifest/branches/ET_2012_11/einsteintoolkit.th ./GetComponents --parallel einsteintoolkit_2012_11.th ./GetComponents --noanonymous ../UsefulFiles/Thornlists/ParmaThorns.th # ------------------- cd /home/kepler/${USER}/ET/ET_2012_11/Cactus svn checkout https://svn.cct.lsu.edu/repos/numrel/simfactory2/trunk simfactory # ------------------- cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.cfg simfactory/mdb/optionlists/ cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.ini simfactory/mdb/machines/ cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.run simfactory/mdb/runscripts/ cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.sub simfactory/mdb/submitscripts/ # ------------------- sed "s/whisky11/ET_2012_11/" ../../UsefulFiles/SimFactory/defs.local.ini > simfactory/etc/defs.local.ini cp ../../UsefulFiles/Thornlists/EXE_MinimalBarFrank_Orsted.th thornlists/EXE_MinimalBarFrank.th # ------------------- cd /home/kepler/${USER}/ET/ET_2012_11/Cactus #### Copy the par file generator mkdir par/MyProject/ cp ~/SVN_parma/Examples/Par/rnsid_MAR_Orstead.rpar par/MyProject/rnsid_MAR.rpar cd par/MyProject/ ./rnsid_MAR.rpar cd /home/kepler/${USER}/ET/ET_2012_11/Cactus simfactory/bin/sim build MinimalBarFrank --thornlist thornlists/EXE_MinimalBarFrank.th The rest as usual (i.e., as it is for the "dev" tree) ==== Get the whisky11 tree working ==== cd /home/kepler/${USER}/ET/whisky11 # --- copy our original thorns tar xzvf /numrel/storageQ01/MagneticBar/CactusTree_WhiskyEXP_March11.tgz find . -name .svn | awk '{print "rm -rf " $1}' # --- GET our new thorns curl -kO https://raw.github.com/gridaphobe/CRL/master/GetComponents chmod a+x GetComponents ./GetComponents --noanonymous ../UsefulFiles/Thornlists/ParmaThorns.th Setting SIMFACTORY cd Cactus/ # ** # ** SIMFACTORY **** # ** # --- Get the new version of simfactory # ** mv simfactory/ old_simfactory svn checkout https://svn.cct.lsu.edu/repos/numrel/simfactory2/trunk simfactory # ** # ------------------- # ** cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.cfg simfactory/mdb/optionlists/ cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.ini simfactory/mdb/machines/ cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.run simfactory/mdb/runscripts/ cp ../../UsefulFiles/SimFactory/kepler.pr.infn.it.sub simfactory/mdb/submitscripts/ # ** # ------------------- # ** ########echo "HDF5=yes" >> simfactory/mdb/optionlists/kepler.pr.infn.it.cfg echo "MPI=CUSTOM" >> simfactory/mdb/optionlists/kepler.pr.infn.it.cfg echo "MPI_INC_DIRS=/opt/ExternalLibraryGNU/openmpi/include" >> simfactory/mdb/optionlists/kepler.pr.infn.it.cfg echo "MPI_LIB_DIRS=/opt/ExternalLibraryGNU/openmpi/lib" >> simfactory/mdb/optionlists/kepler.pr.infn.it.cfg echo "MPI_LIBS=mpi_cxx util numa mpi" >> simfactory/mdb/optionlists/kepler.pr.infn.it.cfg # ** # ** sed "s/-L 3/-L 1/" simfactory/mdb/runscripts/queenbee-mvapich2.run > TMP mv TMP simfactory/mdb/runscripts/mike-openmpi.run sed "s/-L 3/-L 1/" simfactory/mdb/runscripts/mike-openmpi.run> TMP mv TMP simfactory/mdb/runscripts/mike-openmpi.run # ** # correct queen bee to use old interface # ** echo "MPI=CUSTOM" >> simfactory/mdb/optionlists/queenbee-mvapich2.cfg echo "MPI=CUSTOM" >> simfactory/mdb/optionlists/mike-openmpi.cfg # ** # ------------------- # ** cp ../../UsefulFiles/SimFactory/defs.local.ini simfactory/etc emacs -nw simfactory/etc/defs.local.ini # ** === Example of standard build === cd /home/kepler/${USER}/ET/whisky11/Cactus cp ../../UsefulFiles/Thornlists/EXE_BarModeMar11.th thornlists/ make whisky11-config options=simfactory/mdb/optionlists/kepler.pr.infn.it.cfg THORNLIST=../../UsefulFiles/Thornlists/EXE_BarModeMar11.th make whisky11 === Example of SIMFACTORY build === cd /home/kepler/${USER}/ET/whisky11/Cactus simfactory/bin/sim build whisky11 --thornlist thornlists/EXE_BarModeMar11.th (to check MPI=OpenMPI OPENMPI_DIR=/opt/ExternalLibraryGNU/openmpi -lmpi_cxx -lutil -lnuma -lmpi) === Example of run on queen bee === simfactory/bin/sim create S01b1_14_r80 --parfile=par/MyProject/MagWhisky.rpar.dir/S01b1_14_r80.par --configuration whisky11 simfactory/bin/sim create S01b1_14_r40 --parfile=par/MyProject/MagWhisky.rpar.dir/S01b1_14_r40.par --configuration whisky11 simfactory/bin/sim create S01b1_14_r20 --parfile=par/MyProject/MagWhisky.rpar.dir/S01b1_14_r20.par --configuration whisky11 simfactory/bin/sim submit --procs=32 --num-threads=1 S01b1_14_r80 #got: PBS -l nodes=4:ppn=8 simfactory/bin/sim submit --procs=128 --num-threads=1 S01b1_14_r40 simfactory/bin/sim submit --procs=1024 --num-threads=1 S01b1_14_r20 simfactory/bin/sim show-output S01b1_14_r80 First one has to have the par file up-to-date. For example: cd /home/kepler/${USER}/ET/whisky11/Cactus #### Copy the par file generator mkdir par/MyProject/ cp ~/SVN_parma/Examples/Par/MagWhisky.rpar par/MyProject/ cd par/MyProject/ ./MagWhisky.rpar cd /home/kepler/${USER}/ET/whisky11/Cactus Now I can check it on kepler! simfactory/bin/sim create S01b1_14_r80 --parfile=par/MyProject/MagWhisky.rpar.dir/S01b1_14_r80.par --configuration whisky11 simfactory/bin/sim run S01b1_14_r80 --num-threads=1 --procs=8 ===== HPC MACHINE available to the group ===== We got access to the following allocation on HPC machine: * Fermi SYSTEM ssh l rdepietr login.fermi.cineca.it * [[https://allocations.loni.org/balances.php|Loni]] queenbee at LSU (loni_numrel08) ssh -l depietri qb3.loni.org * [[https://accounts.hpc.lsu.edu/profile.php|HPC]] mike2 at LSU (hpc_hyrel13) ssh -l depietri mike1.hpc.lsu.edu * [[https://portal.xsede.org/my-xsede|Xsede]] Kraken, Stampede (TG-MCA02N014) ====== OLD NOTES (before November the 15th, 2013) ====== We do have a tree on our WEB server where we store useful configuration and file needed to better run our simulations. They can be get making a local SVN version svn --username=depietri co https://einstein.pr.infn.it/svn//PRV/WEB//PRIVATE/UsefulFiles/ UsefulFiles From here a common tree I usually create subtree for the various version of cactus that I am using. Typically, a very old version of Whisky, the current version of the Einstein Toolkit and the developer tree. ===== Download CACTUS (Orsted) ===== The first step is to download the main cactus components from the [[http://einsteintoolkit.org|EinsteinTollkit]] repository. We also need the following ThornLists (AEIThorns.th, ParmaThorns.th, Whisky_DEV.th, Whisky_EXP.th, Whisky_PUB.th) that can be downloaded from our PRIVATE web pages [[https://einstein.pr.infn.it/PRIVATE/UsefulFiles/Thornlists/|ThornLists]] First you have to go the Directory where you would like to install CACTUS. And then you will have to execute the following commands: (they will create a working Cactus tree ready to be compiled) curl -O https://raw.github.com/gridaphobe/CRL/master/GetComponents chmod a+x GetComponents ./GetComponents --parallel https://svn.einsteintoolkit.org/manifest/branches/ET_2012_11/einsteintoolkit.th ./GetComponents --noanonymous ./Whisky_DEV.th ./GetComponents --noanonymous ./Whisky_EXP.th ./GetComponents --noanonymous ./AEIThorns.th ./GetComponents --noanonymous ./ParmaThorns.th ===== Download CACTUS (Developer TREE) ===== curl -O https://raw.github.com/gridaphobe/CRL/master/GetComponents chmod a+x GetComponents ./GetComponents --parallel https://svn.einsteintoolkit.org/manifest/trunk/einsteintoolkit.th Now Our local arrangements... ./GetComponents --noanonymous ../Orsted/Whisky_DEV.th ./GetComponents --noanonymous ../Orsted/Whisky_EXP.th ./GetComponents --noanonymous ../Orsted/AEIThorns.th ./GetComponents --noanonymous ../Orsted/ParmaThorns.th ===== Our old Version (Just not to get mixed up) ===== tar xzvf /storageQ01/MagneticBar/CactusTree_WhiskyEXP_March11.tgz find . -name .svn | awk '{print "rm -rf " $1}' curl -O https://raw.github.com/gridaphobe/CRL/master/GetComponents chmod a+x GetComponents ./GetComponents --noanonymous ../UsefulFiles/Thornlists/ParmaThorns.th svn checkout https://svn.cct.lsu.edu/repos/numrel/simfactory2/trunk simfactory I obtained (on Einstein) an executable suitable for our work on Magnetic Bar with the command "make BarModeMar11-config options=../../UsefulFiles/SimFactory/einstein.pr.infn.it.cfg THORNLIST=../../UsefulFiles/Thornlists/EXE_BarModeMar11.th MPI=OpenMPI OPENMPI_DIR=/usr/lib64/openmpi/1.4-gcc/" ===== Set up SIMFACTORY ===== Simfactory to work need that the local machine we use should be defined. This is done properly setting the file "simfactory/etc/defs.local.ini" and for any machine we would like to use to provide in: [simfactory/mdb/machines/] the machine definitions (eg. SYSTEMNAME.ini) [simfactory/mdb/optionlist/] the machine compilation options (eg. SYSTEMNAME.cgf) For example we may give the commands: cp ../../UsefulFiles/SimFactory/*.ini simfactory/mdb/machines/ cp ../../UsefulFiles/SimFactory/*.cfg simfactory/mdb/optionlists/ mv simfactory/mdb/machines/defs.local.ini simfactory/etc/ emacs -nw simfactory/etc/defs.local.ini Example of all these file can be downloaded from: [[https://einstein.pr.infn.it/PRIVATE/UsefulFiles/SimFactory/|SIMFACTORY]] It is also very useful to define the following alias that would simplify the use of simfactory //"alias SIM=simfactory/bin/sim"// ./simfactory/bin/sim create S01b1_14_r40 --parfile=par/S01b1_14_r40.par --configuration BarModeMar11 ./simfactory/bin/sim submit --procs=256 --num-threads=1 S01b1_14_r40 ./simfactory/bin/sim show-output S01b1_14_r40 ===== COMPILING on FERMI ===== To compile on fermi (once SIM factory is properly set): SIM sync fermiBGQ SIM login fermiBGQ SIM build PlainOerstead --thornlist thornlists/PlainOerstead.th That should be enough to create a working executable. === General setting for cactus compilation === CPP = cpp FPP = cpp CC = /opt/ibmcmp/vacpp/bg/12.1/bin/bgxlc_r CXX = /opt/ibmcmp/vacpp/bg/12.1/bin/bgxlC_r F77 = /opt/ibmcmp/xlf/bg/14.1/bin/bgxlf_r F90 = /opt/ibmcmp/xlf/bg/14.1/bin/bgxlf90_r C_DEPEND = $(CPP) -M $(CPPFLAGS) CXX_DEPEND = $(CPP) -M $(CPPFLAGS) #CPPFLAGS = -DMPICH_IGNORE_CXX_SEEK CPPFLAGS = FPPFLAGS = -traditional CFLAGS = -qarch=qp -qtune=qp -qsuppress=1506-1108:1506-1197 -qlanglvl=stdc99 CXXFLAGS = -qarch=qp -qtune=qp -qsuppress=1506-1108 -qrtti F77FLAGS = -qarch=qp -qtune=qp -qsuppress=1520-022:1501-510:cmpmsg -qextname F90FLAGS = -qarch=qp -qtune=qp -qsuppress=1520-022:1501-510:cmpmsg -qextname #### -qextname (This option preserve the trailing underscore on external #### -qextname ... fortran linkange .... needed to mix up fortran and C code) LDFLAGS = -Wl,-relax SYS_INC_DIRS = LIBDIRS = /opt/ibmcmp/xlf/bg/14.1/bglib64/ LIBS = xlf90_r xlfmath SPI SPI_cnk pthread rt stdc++ #######---------------------------------------------------------- DEBUG = no CPP_DEBUG_FLAGS = -g -DCARPET_DEBUG FPP_DEBUG_FLAGS = -g -DCARPET_DEBUG C_DEBUG_FLAGS = -g -O0 -qnoipa -qcheck=bounds CXX_DEBUG_FLAGS = -g -O0 -qnoipa -qcheck=bounds F77_DEBUG_FLAGS = -g -O0 -qnoipa -qcheck F90_DEBUG_FLAGS = -g -O0 -qnoipa -qcheck #######---------------------------------------------------------- # Try: -qhot=simd # -O3 for C takes >30min to compile ML_BSSN # -O3 for C++ and Fortran crashes with SIGFPE # -qipa leads to undefined functions while linking OPTIMISE = yes CPP_OPTIMISE_FLAGS = # -DCARPET_OPTIMISE -DNDEBUG FPP_OPTIMISE_FLAGS = # -DCARPET_OPTIMISE -DNDEBUG C_OPTIMISE_FLAGS = -O3 -qspill=100000 -qmaxmem=-1 -qhot -qlibmpi CXX_OPTIMISE_FLAGS = -O3 -qspill=100000 -qmaxmem=-1 -qhot -qlibmpi F77_OPTIMISE_FLAGS = -O3 -qspill=100000 -qmaxmem=-1 -qhot -qlibmpi F90_OPTIMISE_FLAGS = -O3 -qspill=100000 -qmaxmem=-1 -qhot -qlibmpi #######---------------------------------------------------------- PROFILE = no CPP_PROFILE_FLAGS = FPP_PROFILE_FLAGS = C_PROFILE_FLAGS = -pg CXX_PROFILE_FLAGS = -pg F77_PROFILE_FLAGS = -pg F90_PROFILE_FLAGS = -pg #######---------------------------------------------------------- OPENMP = yes CPP_OPENMP_FLAGS = FPP_OPENMP_FLAGS = -D_OPENMP C_OPENMP_FLAGS = -qsmp=omp CXX_OPENMP_FLAGS = -qsmp=omp F77_OPENMP_FLAGS = -qsmp=omp F90_OPENMP_FLAGS = -qsmp=omp #######---------------------------------------------------------- WARN = yes CPP_WARN_FLAGS = FPP_WARN_FLAGS = C_WARN_FLAGS = CXX_WARN_FLAGS = F77_WARN_FLAGS = F90_WARN_FLAGS = #######---------------------------------------------------------- ## /bgsys/drivers/ppcfloor/comm/xl.ndebug/bin/mpixlcxx_r -show ## mpixlcxx_r -show #######---------------------------------------------------------- MPI_DIR = /bgsys/drivers/V1R1M1/ppc64/comm/xl.ndebug/ MPI_INC_DIRS = /bgsys/drivers/ppcfloor/comm/default/include /bgsys/drivers/ppcfloor/comm/sys/include /bgsys/drivers/V1R1M1/ppc64/spi/include/kernel/cnk /bgsys/drivers/V1R1M1/ppc64/comm/xl.ndebug/include MPI_LIB_DIRS = /bgsys/drivers/V1R1M1/ppc64/comm/xl.ndebug/lib /bgsys/drivers/V1R1M1/ppc64/comm/sys-fast/lib /bgsys/drivers/V1R1M1/ppc64/spi/lib MPI_LIBS = cxxmpich mpich opa mpl pami SPI SPI_cnk #######---------------------------------------------------------- PTHREADS = yes #######---------------------------------------------------------- BLAS_DIR = /gpfs/scratch/userexternal/rdepietr/external/BLAS LAPACK_DIR = /gpfs/scratch/userexternal/rdepietr/external/LAPACK LORENE_DIR = /gpfs/scratch/userexternal/rdepietr/external/LORENE #######---------------------------------------------------------- GSL_DIR = /cineca/prod/libraries/gsl/1.15/bgq-xl--1.0 HDF5_DIR = /cineca/prod/libraries/hdf5/1.8.9_ser/bgq-xl--1.0 ZLIB_DIR = /cineca/prod/libraries/zlib/1.2.7/bgq-gnu--4.4.6 LIBSZ_DIR = /cineca/prod/libraries/szip/2.1/bgq-xl--1.0/lib LIBJPEG_DIR = /cineca/prod/libraries/libjpeg/8d/bgq-gnu--4.4.6 #######---------------------------------------------------------- FFTW3_DIR = /cineca/prod/libraries/fftw/3.3.2/bgq-xl--1.0 FFTW3_LIBS = fftw3f #######---------------------------------------------------------- ## May need a patch to FFTW3.sh in "ExternalLibraries/FFTW3" #######---------------------------------------------------------- #######---------------------------------------------------------- === Know problems === == ZliB not found == Compiling the old tree the linker is not able to find libz.a. I simple workaround is to manually add to the default linker search path the directory where the library is located LIBDIRS=/opt/ibmcmp/xlf/bg/14.1/lib64/ /cineca/prod/libraries/zlib/1.2.7/bgq-gnu--4.4.6/lib /cineca/prod/libraries/szip/2.1/bgq-xl--1.0/lib LIBS= xlf90_r xlfmath SPI SPI_cnk pthread rt stdc++ z sz instead of LIBDIRS=/opt/ibmcmp/xlf/bg/14.1/lib64/ LIBS= xlf90_r xlfmath SPI SPI_cnk pthread rt stdc++ == Vectorization == VECTORIZATION resulted on internal compiler error. ############################################ #### Vectorization #### #### This created in vectors.h (ToReal macro) #### from thorn LSUThorns/Vectors #### /bgxlC_r: 1501-230 (S) Internal compiler error; please contact your Service ############################################ ## VECTORISE = yes ## VECTORISE_ALIGNED_ARRAYS = no ## VECTORISE_ALWAYS_USE_ALIGNED_LOADS = yes ## VECTORISE_INLINE = no ############################################ VECTORISE = no VECTORISE_ALIGNED_ARRAYS = no VECTORISE_ALWAYS_USE_ALIGNED_LOADS = no VECTORISE_INLINE = no ############################################ == MPI == Compiling on fermi present that between the March 2011 version till early 2012 there is a difference on the way is are set the MPI options. This result on different way of define the configuration variables OLD VERSION: MPI = CUSTOM NEW VERSION ### MPI = CUSTOM The other settings are unchanged MPI_DIR = /bgsys/drivers/V1R1M1/ppc64/comm/xl.ndebug/ MPI_INC_DIRS = /bgsys/drivers/ppcfloor/comm/default/include /bgsys/drivers/ppcfloor/comm/sys/include /bgsys/drivers/V1R1M1/ppc64/spi/include/kernel/cnk /bgsys/drivers/V1R1M1/ppc64/comm/xl.ndebug/include MPI_LIB_DIRS = /bgsys/drivers/V1R1M1/ppc64/comm/xl.ndebug/lib /bgsys/drivers/V1R1M1/ppc64/comm/sys-fast/lib /bgsys/drivers/V1R1M1/ppc64/spi/lib MPI_LIBS = cxxmpich mpich opa mpl pami SPI SPI_cnk == hwloc == The new version of Cactus has a main problem on "Fermi". The plain compilation will create an executable that result on the errors: 2012-12-14 13:44:52.417 (FATAL) [0x40000c99050] :18315:ibm.runjob.client.Job: could not start job: job failed to start 2012-12-14 13:44:52.417 (FATAL) [0x40000c99050] :18315:ibm.runjob.client.Job: Load failed on R31-ID-J04: Generating static TLB map for appl Cineca support say that this error is due to: "lrunjob non e' in grado di allocare tutta la memoria richiesta dall'eseguibile." Any easy way to take out this error is to remove the support for the library "hwloc". == CarpetLib_setmemlimit: Assertion `_val' failed. == Compiling with March 2011 source tree results on the above error. The assertion is generated in "void CarpetLib_setmemlimit (CCTK_ARGUMENTS)" of file Carpet/CarpetLib/src/mem.cc (line 274). Solution: Take out lines: // { // check (not setrlimit (RLIMIT_AS, & aslimit)); // } == The March 2011 version problem with CARPET IOASCII (ON FERMI) == The setting of the IO ascii has a BUG specifying the info for //"out_fileinfo"//. In particular if **"all"** is specified then the simulation end with "2013-02-06 08:21:24.990 (WARN ) [0x40001059050] LL13020608132090:111206:ibm.runjob.client.Job: terminated by signal 6" and the assert error "cactus_MinimalBarWhisky: /gpfs/scratch/userexternal/rdepietr/SRC/WhiskyMar11/Cactus/configs/MinimalBarWhisky/build/CarpetIOScalar/ioscalar.cc:388: CarpetIOScalar::OutputVarAs: Assertion `file.good()' failed." The options **"parameter filename"** and **"axis labels"** do works. KEYWORD out_fileinfo "Add some useful file information to output files ?" STEERABLE = RECOVER { "none" :: "no file information" "creation date" :: "add creation date" "parameter filename" :: "add parameter filename" "axis labels" :: "add axis labels information to output files" "all" :: "add all available file information" } "all" == Orsted restrict problem ML_BSSN == We did check on the motion of star on the grid with ML_BSSN. We checked the following possibility: (o2) CXX_OPTIMISE_FLAGS = -O2 -qsmp=noauto:noostls (NR) CXXFLAGS = ...... -qnokeyword=restrict -qalias=norestric CXX_OPTIMISE_FLAGS = -O3 -qspill=100000 -qmaxmem=-1 -qhot -qlibmpi (CC) CXXFLAGS = ...... -qnokeyword=restrict -qalias=norestric CXX_OPTIMISE_FLAGS = -O3 -qspill=100000 -qmaxmem=-1 -qhot -qlibmpi cctk_cv_cxx_restrict=no We observed a different behavior of the the moving of the grid with respect to what happen on INTEL machine. Same compilation tree code. ===== EXECUTING on FERMI ===== The fermi system submission is based on the loadlever batch system and the main submition command is "llsubmit Here is an example of a command file to lunch pure MPI on 2048 cores: #!/bin/bash # # @ job_name = JOBNAME.$(jobid) # @ output = z.JOBNAME.$(jobid).out # @ error = z.JOBNAME.$(jobid).err # @ shell = /bin/bash # @ job_type = bluegene # @ wall_clock_limit = 2:30:00 # @ bg_size = 128 # @ class = parallel # @ account_no = INFN_OG51 # @ queue #### (name of par FILE) ==> PARFILE (without .par extension) #### 4 ==> NT number of openMP threads #### 512 ==> NP number of NPI processes #### 128 ==> BLUGENE size of the run export NT=1 export NP=2048 export PAROLEE=xCARPET_RL3_dx0.400it32 export EXE=/gpfs/scratch/userexternal/rdepietr/SRC/Orsted/Cactus/exe/cactus_PlainOerstead export PAR=/gpfs/scratch/userexternal/rdepietr/SpeedTest/ParFiles/${PARFILE}.par export WDIR=/gpfs/scratch/userexternal/rdepietr/SpeedTest/Runs/${PARFILE}/128_np${NP}_nt${NT}/ export RANKSperNODE=$((16/${NT})) echo "=================================" echo "Executable is : ${EXE}" echo "NP is : ${NP}" echo "NT is : ${NT}" echo "rank per node : ${RANKSperNODE}" echo "Blue gene size is: xBGsize" echo "Total Number of treads is: $((${NP}*${NT}))" echo "Total Number of BLUE GENE: $((128*${RANKSoperNODE}*${NT}))" echo "PAR file is : ${PAR}" echo "Working Dir is: ${WDIR}" echo "=================================" mkdir -p ${WDIR} cd ${WDIR} pwd ls echo "=================================" # -mode SMP # -mode DUAL # -mode VN # -np `${nodes}*${taskspernode}` echo runjob --np ${NP} --ranks-per-node $RANKSperNODE --envs OMP_NUM_THREADS=4 --exe $EXE --args ${PAR} echo "**************************************************" echo "**(TIME) START: " $(date) echo "**************************************************" export start=$(date +"%s" ) runjob --np ${NP} --ranks-per-node $RANKSperNODE --envs OMP_NUM_THREADS=4 --exe $EXE --args ${PAR} | tee OUTPUT export end=$(date +"%s" ) export timeSEC=$(($end - $start)) echo "**************************************************" echo "**(TIME) STOP: " $(date) echo "**************************************************" echo "**(WALL TIME) SEC:" $timeSEC echo "**************************************************" ====== SIMFACTORY on FERMI ====== Just to start > SIM setup resulted in Determining local machine name: fen08.fermi.cineca.it Creating machine fen08.fermi.cineca.it from generic: machine fen08.fermi.cineca.it [/fermi/home/userexternal/rdepietr/DEV/Cactus/simfactory/mdb/machines/fen08.fermi.cineca.it.ini] created successfully enter value for key user [rdepietr]: enter value for key email [rdepietr]: roberto.depietri@unipr.it enter value for key allocation []: INFN_OG51 enter value for key sourcebasedir (the parent directory containing the Cactus sourcetree) [/fermi/home/userexternal/rdepietr/DEV]: enter value for key basedir (the location of simfactory simulations) [/fermi/home/userexternal/rdepietr/simulations]: /gpfs/scratch/userexternal/rdepietr/simulations would you like to enter key/value pairs for a specific machine? [Y/N*]: --------------------SUMMARY---------------------[default] user = rdepietr email = roberto.depietri@unipr.it allocation = INFN_OG51 sourcebasedir = /fermi/home/userexternal/rdepietr/DEV basedir = /gpfs/scratch/userexternal/rdepietr/simulations ------------------END SUMMARY------------------: Save contents [Y*/N]: Contents successfully written to /fermi/home/userexternal/rdepietr/DEV/Cactus/simfactory/etc/defs.local.ini To properly define the setting I decided to name the system as BlueGeneQfermi