====== CHROMA ====== **//HYBRED MONTECARLO EVOLUTION (CHROMA)//** The step here refer to the compilation and execution of the Hybrid-Montecarlo simulations program. It targets the Pure Gauge SU(3) Theory using the publicly available USQCD collaboration and it is based on the "chroma" library (http://usqcd.jlab.org/usqcd-docs/chroma/). ===== Download CHROMA and its dependencies ===== The CHROMA application can be downloaded from the following web pages: wget http://usqcd.jlab.org/usqcd-software/chroma/chroma-3.38.0.tar.gz It has three main dependence. Two are from the same collaboration and provide the encapsulation of the MPI combination and the definition of the basic C++ class used: wget http://usqcd.jlab.org/usqcd-software/qmp/qmp-2.1.6.tar.gz wget http://usqcd.jlab.org/usqcd-software/qdp++/qdp++-1.36.1.tar.gz The last dependence is on the libxml2 library (http://xmlsoft.org/) that is used to parse the simulation parameter file. Usually this library is already installed on a machine but in our case (on turing) it is not. That will give as an example of the thinks that one needs to do when a required library is not installed. ===== Compiling CHROMA ===== The CHROMA application need has prerequisite to have installed the * The qmp wrapper library for MPI comunications. * The qdp++ C++ Library And the compilation can be achieved with the following commands #!/bin/bash export COPT=`mpicc -showme:compile` export LINKOPT=`mpicc -showme:link` export MYDIR=`pwd` export PATH=$PATH:$MYDIR/local/bin/ if [ "a${Nd}" = "a" ] ; then export Nd=4 ;fi if [ "a${Nc}" = "a" ] ; then export Nc=3 ;fi if [ "a${NT}" = "a" ] ; then export NT=1 ;fi ##-------------------------------------- First we have to check if there is a working installation of the xml2 library. In case is not installed we have to compile a local copy ##-------------------------------------- if [ "a$(which xml2-config)" = "a" ] ; then echo "** XML2 not found!" echo "** COMPILING libxml2-2.7.2" tar xzvf $MYDIR/libxml2-2.7.2.tar.gz cd $MYDIR/libxml2-2.7.2 ./configure --prefix=${MYDIR}/local --disable-shared make -j ${NT} make -j ${NT} install cd $MYDIR/ else echo "**Found xml2-config in " echo "**[$(which xml2-config)]" fi ##-------------------------------------- The next step is to install the qmp wrapper library for MPI communications ##-------------------------------------- if [ "a$(which qmp-config)" = "a" ] ; then echo "** NOT Found qmp-config" echo "** Compiling qmp-2.1.6" tar xzvf $MYDIR/qmp-2.1.6.tar.gz cd $MYDIR/qmp-2.1.6 ./configure --with-qmp-comms-type=MPI \ --with-qmp-comms-ldflags="$LINKOPT" \ --with-qmp-comms-cflags="$COPT" \ --prefix=${MYDIR}/local make -j ${NT} make -j ${NT} install cd ${MYDIR} else echo "** Found qmp-config in " echo "** [$(which qmp-config)]" fi ##-------------------------------------- Here the main part of the compilation procedure ##-------------------------------------- export DirCHROMA=${MYDIR} echo "=======================================================" echo "About to compile chroma for " echo "Nd = ${Nd}" echo "Nc = ${Nc}" echo "DIR =${DIR}" echo "MYDIR= " $MYDIR echo "COPT= " $COPT echo "LINKOPT=" $LINKOPT echo "DirCHROMA = ${DirCHROMA}" echo "=======================================================" tar xzf ${MYDIR}/qdp++-1.36.1.tar.gz tar xzf ${MYDIR}/chroma-3.38.0.tar.gz Then we have the first real component of the QCD package echo "=======================================================" echo " Compiling QDP for --enable-Nd=${Nd} --enable-Nc=${Nc} " echo "=======================================================" cd ${MYDIR}/qdp++-1.36.1 ./configure --prefix=${MYDIR}/local \ --with-qmp=${MYDIR}/local/ \ --enable-parallel-arch=parscalar \ --with-qmp-comms-type=MPI \ --enable-sse2 --enable-precision=double \ --enable-Nd=${Nd} --enable-Nc=${Nc} \ &> ${MYDIR}/qdp_configure.out make -j ${NT} &> ${MYDIR}/qdp_make.out make -j ${NT} install &> ${MYDIR}/qdp_make_install.out An finally we produce the executable to run the real simulations echo "=======================================================" echo " Compiling CHROMA --with-qdp=${MYDIR}" echo "=======================================================" cd ${MYDIR}/chroma ./configure --with-qdp=${MYDIR}/local \ --prefix=${MYDIR}/local \ &> ${MYDIR}/chroma__configure.out make -j ${NT} &> ${MYDIR}/chroma_make.out make -j ${NT} install &> ${MYDIR}/chroma_make_install.out cd ${MYDIR} ===== Executing CHROMA ===== The execution of a CHROMA simulation is done using the sintax purgaug -i IN.xml -o OUT.xml where IN.xml contains the parameter of the simulation and OUT.xml the corresponding results. It also allow to have checkpoint/restart but we are not interested to this possibilities for this notes. An example of an xml file that control the execution of our simulation is ''8_8_8_8_B6.0.xml'': WEAK_FIELD dummy testautocorr.sh 17 13 240 30 0 200 2000 2000 200 8_8_8_8_B6.00.data SINGLEFILE WILSON_GAUGEACT 6.00 false 3 1 1 SIMPLE_GAUGE_STATE PERIODIC_GAUGEBC 3 1 8 8 8 8 ====== Executing CHROMA on the GRID ====== We have create a small bash script command to simplify our copy operation to the grid and we called it "''CopyToGrid.sh''" #!/bin/bash lcg-cr -v --vo theophys -d srm://gridsrm.pi.infn.it/theophys/IS_OG51/Parma/CorsoGrid/$1 \ -l lfn:/grid/theophys/IS_OG51/Parma/CorsoGrid/$1 file://$(pwd)/$1 And then we evaluate the following commands: ../CopyToGrid.sh qdp++-1.36.1.tar.gz ../CopyToGrid.sh libxml2-2.7.2.tar.gz ../CopyToGrid.sh qmp-2.1.6.tar.gz ../CopyToGrid.sh chroma-3.38.0.tar.gz Now that we have all the data we can create a JOB script (''CompileChroma.jdl'') and a JOB descriptor (''CompileChroma.jdl'') that will do the compilation for us. The actual form of this two files is: ==== CompileChroma.jdl ==== # CompileChroma.jdl JobType = "Normal"; CPUNumber = 8; Executable = "CompileChroma.sh"; Arguments = ""; StdOutput = "std.out"; StdError = "std.err"; PerusalFileEnable = true; PerusalTimeInterval = 10; InputSandbox = {"CompileChroma.sh"}; OutputSandbox = {"std.out", "std.err"}; MyProxyServer = "myproxy.cnaf.infn.it"; Requirements =(other.GlueCEInfoHostName == "gridce3.pi.infn.it"); CeRequirements = "hostsmpsize==8 && wholenodes==\"true\" && hostnumber==1"; ==== CompileChroma.sh ==== #!/bin/bash ## ---------------------------------------- ## First we get the tra file we previously ## saved on the GRID storage ## ---------------------------------------- export LFN=lfn:/grid/theophys/IS_OG51/Parma/CorsoGrid lcg-cp -v --vo theophys ${LFN}/qdp++-1.36.1.tar.gz file://$(pwd)/qdp++-1.36.1.tar.gz lcg-cp -v --vo theophys ${LFN}/libxml2-2.7.2.tar.gz file://$(pwd)/libxml2-2.7.2.tar.gz lcg-cp -v --vo theophys ${LFN}/qmp-2.1.6.tar.gz file://$(pwd)/qmp-2.1.6.tar.gz lcg-cp -v --vo theophys ${LFN}/chroma-3.38.0.tar.gz file://$(pwd)/chroma-3.38.0.tar.gz ## ---------------------------------------- ## We can now start the procedure we tested ## on our local machine ## ---------------------------------------- export COPT=`mpicc -showme:compile` export LINKOPT=`mpicc -showme:link` export MYDIR=`pwd` export PATH=$PATH:$MYDIR/local/bin/ export Nd=$1 export Nc=$2 export NT=$3 if [ "a${Nd}" = "a" ] ; then export Nd=4 ;fi if [ "a${Nc}" = "a" ] ; then export Nc=3 ;fi if [ "a${NT}" = "a" ] ; then export NT=1 ;fi ##-------------------------------------- ##-------------------------------------- if [ "a$(which xml2-config)" = "a" ] ; then echo "** XML2 not found!" echo "** COMPILING libxml2-2.7.2" tar xzvf $MYDIR/libxml2-2.7.2.tar.gz cd $MYDIR/libxml2-2.7.2 ./configure --prefix=${MYDIR}/local --disable-shared make -j ${NT} make -j ${NT} install cd $MYDIR/ else echo "**Found xml2-config in " echo "**[$(which xml2-config)]" fi ##-------------------------------------- ##-------------------------------------- if [ "a$(which qmp-config)" = "a" ] ; then echo "** NOT Found qmp-config" echo "** Compiling qmp-2.1.6" tar xzvf $MYDIR/qmp-2.1.6.tar.gz cd $MYDIR/qmp-2.1.6 ./configure --with-qmp-comms-type=MPI \ --with-qmp-comms-ldflags="$LINKOPT" \ --with-qmp-comms-cflags="$COPT" \ --prefix=${MYDIR}/local make -j ${NT} make -j ${NT} install cd ${MYDIR} else echo "** Found qmp-config in " echo "** [$(which qmp-config)]" fi ##-------------------------------------- ##-------------------------------------- export DirCHROMA=${MYDIR} echo "=======================================================" echo "About to compile chroma for " echo "Nd = ${Nd}" echo "Nc = ${Nc}" echo "DIR =${DIR}" echo "MYDIR= " $MYDIR echo "COPT= " $COPT echo "LINKOPT=" $LINKOPT echo "DirCHROMA = ${DirCHROMA}" echo "=======================================================" tar xzf ${MYDIR}/qdp++-1.36.1.tar.gz tar xzf ${MYDIR}/chroma-3.38.0.tar.gz echo "=======================================================" echo " Compiling QDP for --enable-Nd=${Nd} --enable-Nc=${Nc} " echo "=======================================================" cd ${MYDIR}/qdp++-1.36.1 ./configure --prefix=${MYDIR}/local \ --with-qmp=${MYDIR}/local/ \ --enable-parallel-arch=parscalar \ --with-qmp-comms-type=MPI \ --enable-sse2 --enable-precision=double \ --enable-Nd=${Nd} --enable-Nc=${Nc} \ &> ${MYDIR}/qdp_configure.out make -j ${NT} &> ${MYDIR}/qdp_make.out make -j ${NT} install &> ${MYDIR}/qdp_make_install.out echo "=======================================================" echo " Compiling CHROMA --with-qdp=${MYDIR}" echo "=======================================================" cd ${MYDIR}/chroma ./configure --with-qdp=${MYDIR}/local \ --prefix=${MYDIR}/local \ &> ${MYDIR}/chroma__configure.out make -j ${NT} &> ${MYDIR}/chroma_make.out make -j ${NT} install &> ${MYDIR}/chroma_make_install.out cd ${MYDIR} ## ---------------------------------------- ## We now just have to copy back the executable ## and save it on the GRID storage ## ---------------------------------------- lcg-cr -v --vo theophys -d srm://gridsrm.pi.infn.it/theophys/IS_OG51/Parma/CorsoGrid/purgaug \ -l lfn:/grid/theophys/IS_OG51/Parma/CorsoGrid/purgaug file://$(pwd)/local/bin/purgaug ## ----------------------------------------------- ## We now show some basic fact on where the ## compilation wes run ## ----------------------------------------------- echo "**************************************" echo "*** command: df" df echo "*** command: pwd" pwd echo "*** command: ls" ls ===== Compilation on TRAMONTANA ===== We can now execute the compilation submitting the job glite-wms-job-submit -a -o 1.job CompileChroma.jdl ===== Execution on TRAMONTANA ===== The "RUN_RANKED_chroma.sh" that allow the execution of the CHROMA executable may look like as follows: ## ------------------------------------ ## CHECK local options ## ------------------------------------ export XMLparFILE=$1 export RUNid=$2 export EXE=purgaug export MYDIR=`pwd` echo " ========================== =================================" echo "XMLparFILE=" $XMLparFILE echo "MYDIR=" $MYDIR echo "PATH= " $PATH echo " ========================== =================================" ## Here we produce various file needed by mpirun ## In future all will be achived using the ## "mpi-start" hawk NODEFILE1=/tmp/hf1.$(date +"%m%d%y%H%M%S") NODEFILE2=/tmp/hf2.$(date +"%m%d%y%H%M%S") RANKFILE=/tmp/hf3.$(date +"%m%d%y%H%M%S") AWKcommand=/tmp/hf4.$(date +"%m%d%y%H%M%S") touch $NODEFILE1 touch $NODEFILE2 touch $AWKcommand echo "{print \"rank \" i++ \"=\" \$1 \" slot=0\"}" >> $AWKcommand echo "{print \"rank \" i++ \"=\" \$1 \" slot=1\"}" >> $AWKcommand echo "{print \"rank \" i++ \"=\" \$1 \" slot=2\"}" >> $AWKcommand echo "{print \"rank \" i++ \"=\" \$1 \" slot=3\"}" >> $AWKcommand echo "{print \"rank \" i++ \"=\" \$1 \" slot=4\"}" >> $AWKcommand echo "{print \"rank \" i++ \"=\" \$1 \" slot=5\"}" >> $AWKcommand echo "{print \"rank \" i++ \"=\" \$1 \" slot=6\"}" >> $AWKcommand echo "{print \"rank \" i++ \"=\" \$1 \" slot=7\"}" >> $AWKcommand for host in $LSB_HOSTS; do echo $host >> $NODEFILE1; done; sort -u ${NODEFILE1} > ${NODEFILE2} awk -f $AWKcommand ${NODEFILE2} > ${RANKFILE} NP=$[`cat ${RANKFILE} | wc --lines`] echo "." cmdmpirun="mpirun -np $NP -hostfile $NODEFILE2 --rankfile ${RANKFILE}" echo "========= MPI PARAMETER ================" echo "NP = ${NP}" echo "mpirun = $(which mpirun)" echo " => ${cmdmpirun}" echo "========================================" ## ------------------------------------ # copy the input file from the SE to the WN ## ------------------------------------ cmd1="lcg-cp -v --vo theophys lfn:/grid/theophys/IS_OG51/Parma/CorsoGrid/${XMLparFILE}.xml file://$(pwd)/${XMLparFILE}.xml" cmd2="lcg-cp -v --vo theophys lfn:/grid/theophys/IS_OG51/Parma/CorsoGrid/${EXE} file://$(pwd)/${EXE}" echo "===================== 1(a) ====================" echo $cmd1 echo $cmd2 echo "===================== 1(b) ====================" eval $cmd1 eval $cmd2 ls echo "===================== 1(c) ====================" ## ------------------------------------ # Now we the the exectuable and the # Parameter File ## ------------------------------------ echo "===============================" echo "RANKFILE:" cat ${RANKFILE} echo "===============================" echo "===============================" echo "NODEFILE2:" cat ${NODEFILE2} echo "===============================" echo "----------------------------- Command ----------------" cmd="(${cmdmpirun} ${MYDIR}/$EXE -i ${XMLparFILE}.xml -o ${XMLparFILE}_OUT.xml 2> stderror )" cmd2="time (${cmd}) 2> ${XMLparFILE}.timing" echo $cmd echo $cmd2 echo "------------------------------------------------------" ls -l ${MYDIR}/$EXE chmod +x ${MYDIR}/$EXE ls -l ${MYDIR}/$EXE echo "**************************************************" echo "**Started at $(date)" time (${cmdmpirun} ${MYDIR}/$EXE -i ${XMLparFILE}.xml -o ${XMLparFILE}_OUT.xml 2> stderror) 2> ${XMLparFILE}.timing echo "**Ended at $(date)" rm -f $NODEFILE1 rm -f $NODEFILE2 rm -f $RANKFILE echo "************************************" ls -l tar czf localRESULTS.tgz * echo "************************************" ## ------------------------------------ # Save the result file from the WN to the SE ## ------------------------------------ FnameToSave="${XMLparFILE}_${RUNid}.tgz" lcg-cr -v --vo theophys -d srm://gridsrm.pi.infn.it/theophys/IS_OG51/Parma/CorsoGrid/${FnameToSave} \ -l lfn:/grid/theophys/IS_OG51/Parma/CorsoGrid/${FnameToSave} \ file://$(pwd)/localRESULTS.tgz echo "=====================" The corresponding jdl file is: # RUN_RANKED_chroma.jdl JobType = "Normal"; CPUNumber = 64; Executable = "RUN_RANKED_chroma.sh"; Arguments = "16_16_16_16_B6.0 id8x4"; StdOutput = "std.out"; StdError = "std.err"; PerusalFileEnable = true; PerusalTimeInterval = 10; InputSandbox = {"RUN_RANKED_chroma.sh"}; OutputSandbox = {"std.out", "std.err"}; MyProxyServer = "myproxy.cnaf.infn.it"; Requirements =(other.GlueCEInfoHostName == "gridce3.pi.infn.it"); CeRequirements = "hostsmpsize==8 && wholenodes==\"true\" && hostnumber==4"; One should note a whole section that create auxiliary file for the mpirun (openmp flavour) command to create the actual list of execution nodes and, very important, to manage MEMORY-CORE affinity. In future all of this will be dealt with using the mpi-start command. (...commet in details...) One should note that doubling the number of processor the execution time is better than linear scaling ... 2000 sweep on a 16^4 hypercube. real 164m 2.279s 16 cores real 67m40.271s 32 cores