UNIVERSITÀ DI PARMA
DIPARTIMENTO DI SCIENZE MATEMATICHE, FISICHE E INFORMATICHE

Strumenti Utente

Strumenti Sito

Traccia:
roberto.depietri:pub:chroma

Differenze

Queste sono le differenze tra la revisione selezionata e la versione attuale della pagina.

Link a questa pagina di confronto

Entrambe le parti precedenti la revisioneRevisione precedente
Prossima revisione		Revisione precedente
roberto.depietri:pub:chroma [27/09/2011 09:17] – roberto.depietriroberto.depietri:pub:chroma [27/09/2011 15:15] (versione attuale) – roberto.depietri
Linea 1: Linea 1:
 +====== CHROMA ======
 +
 +
 +**//HYBRED MONTECARLO EVOLUTION (CHROMA)//** 
 +
 +The step here refer to the compilation and execution of the Hybrid-Montecarlo 
 +simulations program. It targets the Pure Gauge SU(3) Theory using the 
 +publicly available USQCD collaboration and it is based on the "chroma" library (http://usqcd.jlab.org/usqcd-docs/chroma/).
 +
 +
 +
 +===== Download CHROMA and its dependencies =====
 +
 +The CHROMA application can be downloaded from the following web pages:
 +
 +  wget http://usqcd.jlab.org/usqcd-software/chroma/chroma-3.38.0.tar.gz
 +
 +It has three main dependence. Two are from the same collaboration
 +and provide the encapsulation of the MPI combination and the 
 +definition of the basic C++ class used:
 +
 +  wget http://usqcd.jlab.org/usqcd-software/qmp/qmp-2.1.6.tar.gz
 +  wget http://usqcd.jlab.org/usqcd-software/qdp++/qdp++-1.36.1.tar.gz
 +
 +The last dependence is on the libxml2 library (http://xmlsoft.org/) that 
 +is used to parse the simulation parameter file. Usually this library
 +is already installed on a machine but in our case (on turing) it is not.
 +That will give as an example of the thinks that one needs to do when
 +a required library is not installed.   
 +
 +===== Compiling CHROMA  =====
 +
 +The CHROMA application need has prerequisite to have installed the 
 +  * The qmp wrapper library for MPI comunications.
 +  * The qdp++ C++ Library
 +
 +
 +
 +And the compilation can be achieved with the following commands
 +
 +  #!/bin/bash
 +  export COPT=`mpicc -showme:compile`
 +  export LINKOPT=`mpicc -showme:link`
 +  export MYDIR=`pwd`
 +  export PATH=$PATH:$MYDIR/local/bin/
 +  if [ "a${Nd}" = "a" ] ; then export Nd=4 ;fi
 +  if [ "a${Nc}" = "a" ] ; then export Nc=3 ;fi
 +  if [ "a${NT}" = "a" ] ; then export NT=1 ;fi
 +  ##--------------------------------------
 +
 +First we have to check if there is a working installation of
 +the xml2 library. In case is not installed we have to compile a
 +local copy
 +
 +  ##--------------------------------------
 +  if [ "a$(which xml2-config)" = "a" ] ; then
 +    echo "** XML2 not found!"
 +    echo "** COMPILING libxml2-2.7.2"
 +    tar xzvf $MYDIR/libxml2-2.7.2.tar.gz
 +    cd $MYDIR/libxml2-2.7.2
 +    ./configure --prefix=${MYDIR}/local --disable-shared
 +    make -j ${NT}
 +    make -j ${NT} install
 +    cd $MYDIR/
 +  else
 +    echo "**Found xml2-config in "
 +    echo "**[$(which xml2-config)]"
 +  fi
 +  ##--------------------------------------
 +
 +The next step is to install the qmp wrapper
 +library for MPI communications
 +
 +  ##--------------------------------------
 +  if [ "a$(which qmp-config)" = "a" ] ; then
 +    echo "** NOT Found qmp-config"
 +    echo "** Compiling qmp-2.1.6"
 +    tar xzvf $MYDIR/qmp-2.1.6.tar.gz
 +    cd $MYDIR/qmp-2.1.6
 +    ./configure --with-qmp-comms-type=MPI  \
 +       --with-qmp-comms-ldflags="$LINKOPT" \
 +       --with-qmp-comms-cflags="$COPT" \
 +       --prefix=${MYDIR}/local
 +    make -j ${NT}
 +    make -j ${NT} install
 +    cd ${MYDIR}
 +  else
 +    echo "** Found qmp-config in "
 +    echo "** [$(which qmp-config)]"
 +  fi
 +  ##--------------------------------------
 +
 +Here the main part of the compilation procedure
 +
 +  ##--------------------------------------
 +  export DirCHROMA=${MYDIR}
 +  echo "======================================================="
 +  echo "About to compile chroma for "
 +  echo "Nd = ${Nd}"
 +  echo "Nc = ${Nc}"
 +  echo "DIR =${DIR}"
 +  echo "MYDIR=  " $MYDIR
 +  echo "COPT=   " $COPT
 +  echo "LINKOPT=" $LINKOPT
 +  echo "DirCHROMA = ${DirCHROMA}" 
 +  echo "======================================================="
 +  tar xzf ${MYDIR}/qdp++-1.36.1.tar.gz
 +  tar xzf ${MYDIR}/chroma-3.38.0.tar.gz
 +
 +Then we have the first real component of the QCD package 
 +
 +  echo "======================================================="
 +  echo " Compiling QDP for --enable-Nd=${Nd}  --enable-Nc=${Nc} "
 +  echo "======================================================="
 +  cd ${MYDIR}/qdp++-1.36.1
 +  ./configure --prefix=${MYDIR}/local    \
 +              --with-qmp=${MYDIR}/local/   \
 +              --enable-parallel-arch=parscalar     \
 +              --with-qmp-comms-type=MPI            \
 +              --enable-sse2  --enable-precision=double \
 +              --enable-Nd=${Nd}  --enable-Nc=${Nc} \
 +                        &> ${MYDIR}/qdp_configure.out
 +  make -j ${NT}         &> ${MYDIR}/qdp_make.out
 +  make -j ${NT} install &> ${MYDIR}/qdp_make_install.out
 +
 +
 +An finally we produce the executable to run the real simulations 
 +
 +  echo "======================================================="
 +  echo " Compiling CHROMA --with-qdp=${MYDIR}"
 +  echo "======================================================="
 +  cd ${MYDIR}/chroma
 +  ./configure  --with-qdp=${MYDIR}/local    \
 +               --prefix=${MYDIR}/local \
 +                        &> ${MYDIR}/chroma__configure.out   
 +  make -j ${NT}         &> ${MYDIR}/chroma_make.out      
 +  make -j ${NT} install &> ${MYDIR}/chroma_make_install.out
 +  cd ${MYDIR}
 +
 +
 +
 +
 +===== Executing CHROMA  =====
 +
 +The execution of a CHROMA simulation is done using the sintax
 +
 +  purgaug -i IN.xml -o OUT.xml
 +
 +where IN.xml contains the parameter of the simulation and OUT.xml
 +the corresponding results. It also allow to have checkpoint/restart
 +but we are not interested to this possibilities for this notes. 
 +
 +An example of an xml file that control the execution of our simulation
 +is ''8_8_8_8_B6.0.xml'':
 +
 +  <?xml version="1.0"?>
 +  <purgaug>
 +    <Cfg>
 +      <cfg_type>WEAK_FIELD</cfg_type>
 +      <cfg_file>dummy</cfg_file>
 +    </Cfg>
 +    <MCControl>
 +      <RNG>
 +        <Seed>testautocorr.sh
 +          <elem>17</elem>
 +          <elem>13 </elem>
 +          <elem>240 </elem>
 +          <elem>30 </elem>
 +        </Seed>
 +      </RNG>
 +      <StartUpdateNum>0</StartUpdateNum>
 +      <NWarmUpUpdates>200</NWarmUpUpdates>
 +      <NProductionUpdates>2000</NProductionUpdates>
 +      <NUpdatesThisRun>2000</NUpdatesThisRun>
 +      <SaveInterval>200</SaveInterval>
 +      <SavePrefix>8_8_8_8_B6.00.data</SavePrefix>
 +      <SaveVolfmt>SINGLEFILE</SaveVolfmt>
 +    </MCControl>
 +    <InlineMeasurements>
 +    </InlineMeasurements>
 +    <HBItr>
 +      <GaugeAction>
 +         <Name>WILSON_GAUGEACT</Name>
 +         <beta>6.00</beta>
 +         <AnisoParam>
 +              <anisoP>false</anisoP>
 +              <t_dir>3</t_dir>
 +              <xi_0>1</xi_0>
 +              <nu>1</nu>
 +         </AnisoParam>
 +         <GaugeState>
 +           <Name>SIMPLE_GAUGE_STATE</Name>
 +           <GaugeBC>
 +             <Name>PERIODIC_GAUGEBC</Name>
 +           </GaugeBC>
 +         </GaugeState>
 +      </GaugeAction>
 +      <HBParams>
 +        <nOver>3</nOver>
 +        <NmaxHB>1</NmaxHB>
 +      </HBParams>
 +      <nrow>8 8 8 8</nrow>
 +    </HBItr>
 +  </purgaug>
 +
 +====== Executing CHROMA on the GRID ======
 +
 +We have create a small bash script command to simplify our copy operation 
 +to the grid and we called it "''CopyToGrid.sh''"
 +
 +  #!/bin/bash                                                                                                     
 +  lcg-cr -v --vo theophys -d srm://gridsrm.pi.infn.it/theophys/IS_OG51/Parma/CorsoGrid/$1  \
 +         -l lfn:/grid/theophys/IS_OG51/Parma/CorsoGrid/$1 file://$(pwd)/$1
 +
 +
 +And then we evaluate the following commands:
 +
 +  ../CopyToGrid.sh qdp++-1.36.1.tar.gz
 +  ../CopyToGrid.sh libxml2-2.7.2.tar.gz
 +  ../CopyToGrid.sh qmp-2.1.6.tar.gz
 +  ../CopyToGrid.sh chroma-3.38.0.tar.gz
 +
 +Now that we have all the data we can create a JOB script (''CompileChroma.jdl'') and a 
 +JOB descriptor (''CompileChroma.jdl'') that will do the compilation for us. The actual
 +form of this two files is:
 +
 +
 +==== CompileChroma.jdl ====
 +
 +  # CompileChroma.jdl
 +  JobType = "Normal";
 +  CPUNumber = 8;
 +  Executable    = "CompileChroma.sh";
 +  Arguments     = "";
 +  StdOutput     = "std.out";
 +  StdError      = "std.err";
 +  PerusalFileEnable = true;
 +  PerusalTimeInterval = 10;
 +  InputSandbox  = {"CompileChroma.sh"}; 
 +  OutputSandbox = {"std.out", "std.err"};
 +  MyProxyServer = "myproxy.cnaf.infn.it";
 +  Requirements =(other.GlueCEInfoHostName == "gridce3.pi.infn.it");
 +  CeRequirements = "hostsmpsize==8 &&  wholenodes==\"true\"  && hostnumber==1";
 +
 +
 +
 +==== CompileChroma.sh ====
 +
 +  #!/bin/bash
 +  ## ----------------------------------------
 +  ## First we get the tra file we previously
 +  ## saved on the GRID storage 
 +  ## ----------------------------------------
 +  export LFN=lfn:/grid/theophys/IS_OG51/Parma/CorsoGrid
 +  lcg-cp -v --vo theophys ${LFN}/qdp++-1.36.1.tar.gz  file://$(pwd)/qdp++-1.36.1.tar.gz 
 +  lcg-cp -v --vo theophys ${LFN}/libxml2-2.7.2.tar.gz file://$(pwd)/libxml2-2.7.2.tar.gz
 +  lcg-cp -v --vo theophys ${LFN}/qmp-2.1.6.tar.gz     file://$(pwd)/qmp-2.1.6.tar.gz    
 +  lcg-cp -v --vo theophys ${LFN}/chroma-3.38.0.tar.gz file://$(pwd)/chroma-3.38.0.tar.gz
 +  ## ----------------------------------------
 +  ## We can now start the procedure we tested
 +  ## on our local machine
 +  ## ----------------------------------------
 +  export COPT=`mpicc -showme:compile`
 +  export LINKOPT=`mpicc -showme:link`
 +  export MYDIR=`pwd`
 +  export PATH=$PATH:$MYDIR/local/bin/
 +  export Nd=$1
 +  export Nc=$2 
 +  export NT=$3
 +  if [ "a${Nd}" = "a" ] ; then export Nd=4 ;fi
 +  if [ "a${Nc}" = "a" ] ; then export Nc=3 ;fi
 +  if [ "a${NT}" = "a" ] ; then export NT=1 ;fi
 +  ##--------------------------------------
 +  ##--------------------------------------
 +  if [ "a$(which xml2-config)" = "a" ] ; then
 +    echo "** XML2 not found!"
 +    echo "** COMPILING libxml2-2.7.2"
 +    tar xzvf $MYDIR/libxml2-2.7.2.tar.gz
 +    cd $MYDIR/libxml2-2.7.2
 +    ./configure --prefix=${MYDIR}/local --disable-shared
 +    make -j ${NT}
 +    make -j ${NT} install
 +    cd $MYDIR/
 +  else
 +    echo "**Found xml2-config in "
 +    echo "**[$(which xml2-config)]"
 +  fi 
 +  ##--------------------------------------
 +  ##--------------------------------------
 +  if [ "a$(which qmp-config)" = "a" ] ; then
 +    echo "** NOT Found qmp-config"
 +    echo "** Compiling qmp-2.1.6"
 +    tar xzvf $MYDIR/qmp-2.1.6.tar.gz
 +    cd $MYDIR/qmp-2.1.6
 +    ./configure --with-qmp-comms-type=MPI  \
 +       --with-qmp-comms-ldflags="$LINKOPT" \
 +       --with-qmp-comms-cflags="$COPT" \
 +       --prefix=${MYDIR}/local
 +    make -j ${NT}
 +    make -j ${NT} install
 +    cd ${MYDIR}
 +  else
 +    echo "** Found qmp-config in "
 +    echo "** [$(which qmp-config)]"
 +  fi
 +  ##--------------------------------------
 +  ##--------------------------------------
 +  export DirCHROMA=${MYDIR}
 +  echo "======================================================="
 +  echo "About to compile chroma for " 
 +  echo "Nd = ${Nd}"
 +  echo "Nc = ${Nc}"
 +  echo "DIR =${DIR}"
 +  echo "MYDIR=  " $MYDIR
 +  echo "COPT=   " $COPT
 +  echo "LINKOPT=" $LINKOPT
 +  echo "DirCHROMA = ${DirCHROMA}" 
 +  echo "======================================================="
 +  tar xzf ${MYDIR}/qdp++-1.36.1.tar.gz
 +  tar xzf ${MYDIR}/chroma-3.38.0.tar.gz
 +  echo "======================================================="
 +  echo " Compiling QDP for --enable-Nd=${Nd}  --enable-Nc=${Nc} "
 +  echo "======================================================="
 +  cd ${MYDIR}/qdp++-1.36.1
 +  ./configure --prefix=${MYDIR}/local    \
 +            --with-qmp=${MYDIR}/local/   \
 +            --enable-parallel-arch=parscalar     \
 +            --with-qmp-comms-type=MPI            \
 +            --enable-sse2  --enable-precision=double \
 +            --enable-Nd=${Nd}  --enable-Nc=${Nc} \
 +                      &> ${MYDIR}/qdp_configure.out
 +  make -j ${NT}         &> ${MYDIR}/qdp_make.out
 +  make -j ${NT} install &> ${MYDIR}/qdp_make_install.out
 +  echo "======================================================="
 +  echo " Compiling CHROMA --with-qdp=${MYDIR}"
 +  echo "======================================================="
 +  cd ${MYDIR}/chroma
 +  ./configure  --with-qdp=${MYDIR}/local    \
 +             --prefix=${MYDIR}/local \
 +                      &> ${MYDIR}/chroma__configure.out   
 +  make -j ${NT}         &> ${MYDIR}/chroma_make.out      
 +  make -j ${NT} install &> ${MYDIR}/chroma_make_install.out
 +  cd ${MYDIR}
 +  ## ----------------------------------------
 +  ## We now just have to copy back the executable
 +  ## and save it on the GRID storage 
 +  ## ----------------------------------------
 +  lcg-cr -v --vo theophys -d srm://gridsrm.pi.infn.it/theophys/IS_OG51/Parma/CorsoGrid/purgaug  \
 +     -l lfn:/grid/theophys/IS_OG51/Parma/CorsoGrid/purgaug file://$(pwd)/local/bin/purgaug
 +  ## -----------------------------------------------
 +  ## We now show some basic fact on where the
 +  ## compilation wes run
 +  ## -----------------------------------------------
 +  echo "**************************************"
 +  echo "*** command:  df"
 +  df
 +  echo "*** command:  pwd"
 +  pwd
 +  echo "*** command:  ls"
 +  ls
 +
 +
 +===== Compilation on TRAMONTANA =====
 +
 +We can now execute the compilation submitting the job
 +
 +   glite-wms-job-submit -a -o 1.job CompileChroma.jdl
 +
 +
 +
 +
 +
 +
 +===== Execution on TRAMONTANA =====
 +
 +The "RUN_RANKED_chroma.sh" that allow the execution of the CHROMA
 +executable may look like as follows:
 +
 +  ## ------------------------------------
 +  ##  CHECK local options
 +  ## ------------------------------------
 +  export XMLparFILE=$1
 +  export RUNid=$2
 +  export EXE=purgaug
 +  export MYDIR=`pwd`
 +  echo " ========================== ================================="
 +  echo "XMLparFILE=" $XMLparFILE
 +  echo "MYDIR=" $MYDIR
 +  echo "PATH= " $PATH
 +  echo " ========================== ================================="
 +  ## Here we produce various file needed by mpirun
 +  ## In future all will be achived using the
 +  ## "mpi-start" hawk
 +  NODEFILE1=/tmp/hf1.$(date +"%m%d%y%H%M%S")
 +  NODEFILE2=/tmp/hf2.$(date +"%m%d%y%H%M%S")
 +  RANKFILE=/tmp/hf3.$(date +"%m%d%y%H%M%S")
 +  AWKcommand=/tmp/hf4.$(date +"%m%d%y%H%M%S")
 +  touch $NODEFILE1
 +  touch $NODEFILE2
 +  touch $AWKcommand
 +  echo "{print \"rank \" i++ \"=\" \$1 \" slot=0\"}"   >> $AWKcommand
 +  echo "{print \"rank \" i++ \"=\" \$1 \" slot=1\"}"   >> $AWKcommand
 +  echo "{print \"rank \" i++ \"=\" \$1 \" slot=2\"}"   >> $AWKcommand
 +  echo "{print \"rank \" i++ \"=\" \$1 \" slot=3\"}"   >> $AWKcommand
 +  echo "{print \"rank \" i++ \"=\" \$1 \" slot=4\"}"   >> $AWKcommand
 +  echo "{print \"rank \" i++ \"=\" \$1 \" slot=5\"}"   >> $AWKcommand
 +  echo "{print \"rank \" i++ \"=\" \$1 \" slot=6\"}"   >> $AWKcommand
 +  echo "{print \"rank \" i++ \"=\" \$1 \" slot=7\"}"   >> $AWKcommand
 +  for host in $LSB_HOSTS; do echo $host >> $NODEFILE1; done;
 +  sort -u  ${NODEFILE1} > ${NODEFILE2}
 +  awk -f $AWKcommand  ${NODEFILE2} > ${RANKFILE}
 +  NP=$[`cat ${RANKFILE} | wc --lines`]
 +  echo "."
 +  cmdmpirun="mpirun -np $NP -hostfile $NODEFILE2 --rankfile ${RANKFILE}"
 +  echo "========= MPI PARAMETER ================"
 +  echo "NP = ${NP}"
 +  echo "mpirun = $(which mpirun)"
 +  echo "      => ${cmdmpirun}"
 +  echo "========================================" 
 +  ## ------------------------------------
 +  # copy the input file from the SE to the WN
 +  ## ------------------------------------
 +  cmd1="lcg-cp -v --vo theophys lfn:/grid/theophys/IS_OG51/Parma/CorsoGrid/${XMLparFILE}.xml file://$(pwd)/${XMLparFILE}.xml"
 +  cmd2="lcg-cp -v --vo theophys lfn:/grid/theophys/IS_OG51/Parma/CorsoGrid/${EXE}            file://$(pwd)/${EXE}"
 +  echo "=====================  1(a) ===================="
 +  echo $cmd1
 +  echo $cmd2
 +  echo "=====================  1(b) ===================="
 +  eval $cmd1
 +  eval $cmd2
 +  ls
 +  echo "=====================  1(c) ===================="
 +  ## ------------------------------------
 +  #  Now we the the exectuable and the 
 +  #  Parameter File
 +  ## ------------------------------------
 +  echo "==============================="
 +  echo "RANKFILE:"
 +  cat ${RANKFILE}
 +  echo "==============================="
 +  echo "==============================="
 +  echo "NODEFILE2:"
 +  cat ${NODEFILE2}
 +  echo "==============================="
 +  echo "----------------------------- Command ----------------"
 +  cmd="(${cmdmpirun} ${MYDIR}/$EXE -i ${XMLparFILE}.xml -o ${XMLparFILE}_OUT.xml 2> stderror )"
 +  cmd2="time (${cmd}) 2> ${XMLparFILE}.timing" 
 +  echo $cmd
 +  echo $cmd2
 +  echo "------------------------------------------------------"
 +  ls -l ${MYDIR}/$EXE
 +  chmod +x  ${MYDIR}/$EXE
 +  ls -l ${MYDIR}/$EXE
 +  echo "**************************************************"
 +  echo "**Started at $(date)"
 +  time (${cmdmpirun} ${MYDIR}/$EXE -i ${XMLparFILE}.xml -o ${XMLparFILE}_OUT.xml 2> stderror)  2> ${XMLparFILE}.timing
 +  echo "**Ended at   $(date)"
 +  rm -f $NODEFILE1
 +  rm -f $NODEFILE2
 +  rm -f $RANKFILE
 +  echo "************************************"
 +  ls -l 
 +  tar czf localRESULTS.tgz *
 +  echo "************************************"
 +  ## ------------------------------------
 +  # Save the result file from the WN  to the SE
 +  ## ------------------------------------
 +  FnameToSave="${XMLparFILE}_${RUNid}.tgz"
 +  lcg-cr -v --vo theophys -d srm://gridsrm.pi.infn.it/theophys/IS_OG51/Parma/CorsoGrid/${FnameToSave}  \
 +         -l lfn:/grid/theophys/IS_OG51/Parma/CorsoGrid/${FnameToSave} \
 +          file://$(pwd)/localRESULTS.tgz
 +  echo "====================="
 +
 +
 +The corresponding jdl file is:
 +
 +  # RUN_RANKED_chroma.jdl
 +  JobType = "Normal";
 +  CPUNumber = 64;
 +  Executable    = "RUN_RANKED_chroma.sh";
 +  Arguments     = "16_16_16_16_B6.0 id8x4";
 +  StdOutput     = "std.out";
 +  StdError      = "std.err";
 +  PerusalFileEnable = true;
 +  PerusalTimeInterval = 10;
 +  InputSandbox  = {"RUN_RANKED_chroma.sh"}; 
 +  OutputSandbox = {"std.out", "std.err"};
 +  MyProxyServer = "myproxy.cnaf.infn.it";
 +  Requirements =(other.GlueCEInfoHostName == "gridce3.pi.infn.it");
 +  CeRequirements = "hostsmpsize==8 &&  wholenodes==\"true\"  && hostnumber==4";
 + 
 +One should note a whole section that create auxiliary file for the mpirun 
 +(openmp flavour) command to create the actual list of execution nodes and,
 +very important, to manage MEMORY-CORE affinity.
 +
 +In future all of this will be dealt with using the mpi-start command.
 +(...commet in details...) 
 +
 +
 +One should note that doubling the number of processor the execution time is better
 +than linear scaling ... 2000 sweep on a 16^4 hypercube.
 +
 +
 +  real 164m 2.279s   16 cores
 +  real 67m40.271s   32 cores
  
Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki