UNIVERSITÀ DI PARMA
DIPARTIMENTO DI SCIENZE MATEMATICHE, FISICHE E INFORMATICHE

Questa pagina è in sola lettura. Puoi visualizzare il sorgente, ma non puoi modificarlo. Contatta l'amministratore se pensi che ci sia un errore.
====== CHROMA ======


**//HYBRED MONTECARLO EVOLUTION (CHROMA)//** 

The step here refer to the compilation and execution of the Hybrid-Montecarlo 
simulations program. It targets the Pure Gauge SU(3) Theory using the 
publicly available USQCD collaboration and it is based on the "chroma" library (http://usqcd.jlab.org/usqcd-docs/chroma/).



===== Download CHROMA and its dependencies =====

The CHROMA application can be downloaded from the following web pages:

  wget http://usqcd.jlab.org/usqcd-software/chroma/chroma-3.38.0.tar.gz

It has three main dependence. Two are from the same collaboration
and provide the encapsulation of the MPI combination and the 
definition of the basic C++ class used:

  wget http://usqcd.jlab.org/usqcd-software/qmp/qmp-2.1.6.tar.gz
  wget http://usqcd.jlab.org/usqcd-software/qdp++/qdp++-1.36.1.tar.gz

The last dependence is on the libxml2 library (http://xmlsoft.org/) that 
is used to parse the simulation parameter file. Usually this library
is already installed on a machine but in our case (on turing) it is not.
That will give as an example of the thinks that one needs to do when
a required library is not installed.   

===== Compiling CHROMA  =====

The CHROMA application need has prerequisite to have installed the 
  * The qmp wrapper library for MPI comunications.
  * The qdp++ C++ Library



And the compilation can be achieved with the following commands

  #!/bin/bash
  export COPT=`mpicc -showme:compile`
  export LINKOPT=`mpicc -showme:link`
  export MYDIR=`pwd`
  export PATH=$PATH:$MYDIR/local/bin/
  if [ "a${Nd}" = "a" ] ; then export Nd=4 ;fi
  if [ "a${Nc}" = "a" ] ; then export Nc=3 ;fi
  if [ "a${NT}" = "a" ] ; then export NT=1 ;fi
  ##--------------------------------------

First we have to check if there is a working installation of
the xml2 library. In case is not installed we have to compile a
local copy

  ##--------------------------------------
  if [ "a$(which xml2-config)" = "a" ] ; then
    echo "** XML2 not found!"
    echo "** COMPILING libxml2-2.7.2"
    tar xzvf $MYDIR/libxml2-2.7.2.tar.gz
    cd $MYDIR/libxml2-2.7.2
    ./configure --prefix=${MYDIR}/local --disable-shared
    make -j ${NT}
    make -j ${NT} install
    cd $MYDIR/
  else
    echo "**Found xml2-config in "
    echo "**[$(which xml2-config)]"
  fi
  ##--------------------------------------

The next step is to install the qmp wrapper
library for MPI communications

  ##--------------------------------------
  if [ "a$(which qmp-config)" = "a" ] ; then
    echo "** NOT Found qmp-config"
    echo "** Compiling qmp-2.1.6"
    tar xzvf $MYDIR/qmp-2.1.6.tar.gz
    cd $MYDIR/qmp-2.1.6
    ./configure --with-qmp-comms-type=MPI  \
       --with-qmp-comms-ldflags="$LINKOPT" \
       --with-qmp-comms-cflags="$COPT" \
       --prefix=${MYDIR}/local
    make -j ${NT}
    make -j ${NT} install
    cd ${MYDIR}
  else
    echo "** Found qmp-config in "
    echo "** [$(which qmp-config)]"
  fi
  ##--------------------------------------

Here the main part of the compilation procedure

  ##--------------------------------------
  export DirCHROMA=${MYDIR}
  echo "======================================================="
  echo "About to compile chroma for "
  echo "Nd = ${Nd}"
  echo "Nc = ${Nc}"
  echo "DIR =${DIR}"
  echo "MYDIR=  " $MYDIR
  echo "COPT=   " $COPT
  echo "LINKOPT=" $LINKOPT
  echo "DirCHROMA = ${DirCHROMA}" 
  echo "======================================================="
  tar xzf ${MYDIR}/qdp++-1.36.1.tar.gz
  tar xzf ${MYDIR}/chroma-3.38.0.tar.gz

Then we have the first real component of the QCD package 

  echo "======================================================="
  echo " Compiling QDP for --enable-Nd=${Nd}  --enable-Nc=${Nc} "
  echo "======================================================="
  cd ${MYDIR}/qdp++-1.36.1
  ./configure --prefix=${MYDIR}/local    \
              --with-qmp=${MYDIR}/local/   \
              --enable-parallel-arch=parscalar     \
              --with-qmp-comms-type=MPI            \
              --enable-sse2  --enable-precision=double \
              --enable-Nd=${Nd}  --enable-Nc=${Nc} \
                        &> ${MYDIR}/qdp_configure.out
  make -j ${NT}         &> ${MYDIR}/qdp_make.out
  make -j ${NT} install &> ${MYDIR}/qdp_make_install.out


An finally we produce the executable to run the real simulations 

  echo "======================================================="
  echo " Compiling CHROMA --with-qdp=${MYDIR}"
  echo "======================================================="
  cd ${MYDIR}/chroma
  ./configure  --with-qdp=${MYDIR}/local    \
               --prefix=${MYDIR}/local \
                        &> ${MYDIR}/chroma__configure.out   
  make -j ${NT}         &> ${MYDIR}/chroma_make.out	     
  make -j ${NT} install &> ${MYDIR}/chroma_make_install.out
  cd ${MYDIR}




===== Executing CHROMA  =====

The execution of a CHROMA simulation is done using the sintax

  purgaug -i IN.xml -o OUT.xml

where IN.xml contains the parameter of the simulation and OUT.xml
the corresponding results. It also allow to have checkpoint/restart
but we are not interested to this possibilities for this notes. 

An example of an xml file that control the execution of our simulation
is ''8_8_8_8_B6.0.xml'':

  <?xml version="1.0"?>
  <purgaug>
    <Cfg>
      <cfg_type>WEAK_FIELD</cfg_type>
      <cfg_file>dummy</cfg_file>
    </Cfg>
    <MCControl>
      <RNG>
        <Seed>testautocorr.sh
          <elem>17</elem>
          <elem>13 </elem>
          <elem>240 </elem>
          <elem>30 </elem>
        </Seed>
      </RNG>
      <StartUpdateNum>0</StartUpdateNum>
      <NWarmUpUpdates>200</NWarmUpUpdates>
      <NProductionUpdates>2000</NProductionUpdates>
      <NUpdatesThisRun>2000</NUpdatesThisRun>
      <SaveInterval>200</SaveInterval>
      <SavePrefix>8_8_8_8_B6.00.data</SavePrefix>
      <SaveVolfmt>SINGLEFILE</SaveVolfmt>
    </MCControl>
    <InlineMeasurements>
    </InlineMeasurements>
    <HBItr>
      <GaugeAction>
         <Name>WILSON_GAUGEACT</Name>
         <beta>6.00</beta>
         <AnisoParam>
              <anisoP>false</anisoP>
              <t_dir>3</t_dir>
              <xi_0>1</xi_0>
              <nu>1</nu>
         </AnisoParam>
         <GaugeState>
           <Name>SIMPLE_GAUGE_STATE</Name>
           <GaugeBC>
             <Name>PERIODIC_GAUGEBC</Name>
           </GaugeBC>
         </GaugeState>
      </GaugeAction>
      <HBParams>
        <nOver>3</nOver>
        <NmaxHB>1</NmaxHB>
      </HBParams>
      <nrow>8 8 8 8</nrow>
    </HBItr>
  </purgaug>

====== Executing CHROMA on the GRID ======

We have create a small bash script command to simplify our copy operation 
to the grid and we called it "''CopyToGrid.sh''"

  #!/bin/bash                                                                                                     
  lcg-cr -v --vo theophys -d srm://gridsrm.pi.infn.it/theophys/IS_OG51/Parma/CorsoGrid/$1  \
         -l lfn:/grid/theophys/IS_OG51/Parma/CorsoGrid/$1 file://$(pwd)/$1


And then we evaluate the following commands:

  ../CopyToGrid.sh qdp++-1.36.1.tar.gz
  ../CopyToGrid.sh libxml2-2.7.2.tar.gz
  ../CopyToGrid.sh qmp-2.1.6.tar.gz
  ../CopyToGrid.sh chroma-3.38.0.tar.gz

Now that we have all the data we can create a JOB script (''CompileChroma.jdl'') and a 
JOB descriptor (''CompileChroma.jdl'') that will do the compilation for us. The actual
form of this two files is:


==== CompileChroma.jdl ====

  # CompileChroma.jdl
  JobType = "Normal";
  CPUNumber = 8;
  Executable    = "CompileChroma.sh";
  Arguments     = "";
  StdOutput     = "std.out";
  StdError      = "std.err";
  PerusalFileEnable = true;
  PerusalTimeInterval = 10;
  InputSandbox  = {"CompileChroma.sh"}; 
  OutputSandbox = {"std.out", "std.err"};
  MyProxyServer = "myproxy.cnaf.infn.it";
  Requirements =(other.GlueCEInfoHostName == "gridce3.pi.infn.it");
  CeRequirements = "hostsmpsize==8 &&  wholenodes==\"true\"  && hostnumber==1";



==== CompileChroma.sh ====

  #!/bin/bash
  ## ----------------------------------------
  ## First we get the tra file we previously
  ## saved on the GRID storage 
  ## ----------------------------------------
  export LFN=lfn:/grid/theophys/IS_OG51/Parma/CorsoGrid
  lcg-cp -v --vo theophys ${LFN}/qdp++-1.36.1.tar.gz  file://$(pwd)/qdp++-1.36.1.tar.gz 
  lcg-cp -v --vo theophys ${LFN}/libxml2-2.7.2.tar.gz file://$(pwd)/libxml2-2.7.2.tar.gz
  lcg-cp -v --vo theophys ${LFN}/qmp-2.1.6.tar.gz     file://$(pwd)/qmp-2.1.6.tar.gz    
  lcg-cp -v --vo theophys ${LFN}/chroma-3.38.0.tar.gz file://$(pwd)/chroma-3.38.0.tar.gz
  ## ----------------------------------------
  ## We can now start the procedure we tested
  ## on our local machine
  ## ----------------------------------------
  export COPT=`mpicc -showme:compile`
  export LINKOPT=`mpicc -showme:link`
  export MYDIR=`pwd`
  export PATH=$PATH:$MYDIR/local/bin/
  export Nd=$1
  export Nc=$2 
  export NT=$3
  if [ "a${Nd}" = "a" ] ; then export Nd=4 ;fi
  if [ "a${Nc}" = "a" ] ; then export Nc=3 ;fi
  if [ "a${NT}" = "a" ] ; then export NT=1 ;fi
  ##--------------------------------------
  ##--------------------------------------
  if [ "a$(which xml2-config)" = "a" ] ; then
    echo "** XML2 not found!"
    echo "** COMPILING libxml2-2.7.2"
    tar xzvf $MYDIR/libxml2-2.7.2.tar.gz
    cd $MYDIR/libxml2-2.7.2
    ./configure --prefix=${MYDIR}/local --disable-shared
    make -j ${NT}
    make -j ${NT} install
    cd $MYDIR/
  else
    echo "**Found xml2-config in "
    echo "**[$(which xml2-config)]"
  fi 
  ##--------------------------------------
  ##--------------------------------------
  if [ "a$(which qmp-config)" = "a" ] ; then
    echo "** NOT Found qmp-config"
    echo "** Compiling qmp-2.1.6"
    tar xzvf $MYDIR/qmp-2.1.6.tar.gz
    cd $MYDIR/qmp-2.1.6
    ./configure --with-qmp-comms-type=MPI  \
       --with-qmp-comms-ldflags="$LINKOPT" \
       --with-qmp-comms-cflags="$COPT" \
       --prefix=${MYDIR}/local
    make -j ${NT}
    make -j ${NT} install
    cd ${MYDIR}
  else
    echo "** Found qmp-config in "
    echo "** [$(which qmp-config)]"
  fi
  ##--------------------------------------
  ##--------------------------------------
  export DirCHROMA=${MYDIR}
  echo "======================================================="
  echo "About to compile chroma for " 
  echo "Nd = ${Nd}"
  echo "Nc = ${Nc}"
  echo "DIR =${DIR}"
  echo "MYDIR=  " $MYDIR
  echo "COPT=   " $COPT
  echo "LINKOPT=" $LINKOPT
  echo "DirCHROMA = ${DirCHROMA}" 
  echo "======================================================="
  tar xzf ${MYDIR}/qdp++-1.36.1.tar.gz
  tar xzf ${MYDIR}/chroma-3.38.0.tar.gz
  echo "======================================================="
  echo " Compiling QDP for --enable-Nd=${Nd}  --enable-Nc=${Nc} "
  echo "======================================================="
  cd ${MYDIR}/qdp++-1.36.1
  ./configure --prefix=${MYDIR}/local    \
            --with-qmp=${MYDIR}/local/   \
            --enable-parallel-arch=parscalar     \
            --with-qmp-comms-type=MPI            \
            --enable-sse2  --enable-precision=double \
            --enable-Nd=${Nd}  --enable-Nc=${Nc} \
                      &> ${MYDIR}/qdp_configure.out
  make -j ${NT}         &> ${MYDIR}/qdp_make.out
  make -j ${NT} install &> ${MYDIR}/qdp_make_install.out
  echo "======================================================="
  echo " Compiling CHROMA --with-qdp=${MYDIR}"
  echo "======================================================="
  cd ${MYDIR}/chroma
  ./configure  --with-qdp=${MYDIR}/local    \
             --prefix=${MYDIR}/local \
                      &> ${MYDIR}/chroma__configure.out   
  make -j ${NT}         &> ${MYDIR}/chroma_make.out	     
  make -j ${NT} install &> ${MYDIR}/chroma_make_install.out
  cd ${MYDIR}
  ## ----------------------------------------
  ## We now just have to copy back the executable
  ## and save it on the GRID storage 
  ## ----------------------------------------
  lcg-cr -v --vo theophys -d srm://gridsrm.pi.infn.it/theophys/IS_OG51/Parma/CorsoGrid/purgaug  \
     -l lfn:/grid/theophys/IS_OG51/Parma/CorsoGrid/purgaug file://$(pwd)/local/bin/purgaug
  ## -----------------------------------------------
  ## We now show some basic fact on where the
  ## compilation wes run
  ## -----------------------------------------------
  echo "**************************************"
  echo "*** command:  df"
  df
  echo "*** command:  pwd"
  pwd
  echo "*** command:  ls"
  ls


===== Compilation on TRAMONTANA =====

We can now execute the compilation submitting the job

   glite-wms-job-submit -a -o 1.job CompileChroma.jdl






===== Execution on TRAMONTANA =====

The "RUN_RANKED_chroma.sh" that allow the execution of the CHROMA
executable may look like as follows:

  ## ------------------------------------
  ##  CHECK local options
  ## ------------------------------------
  export XMLparFILE=$1
  export RUNid=$2
  export EXE=purgaug
  export MYDIR=`pwd`
  echo " ========================== ================================="
  echo "XMLparFILE=" $XMLparFILE
  echo "MYDIR=" $MYDIR
  echo "PATH= " $PATH
  echo " ========================== ================================="
  ## Here we produce various file needed by mpirun
  ## In future all will be achived using the
  ## "mpi-start" hawk
  NODEFILE1=/tmp/hf1.$(date +"%m%d%y%H%M%S")
  NODEFILE2=/tmp/hf2.$(date +"%m%d%y%H%M%S")
  RANKFILE=/tmp/hf3.$(date +"%m%d%y%H%M%S")
  AWKcommand=/tmp/hf4.$(date +"%m%d%y%H%M%S")
  touch $NODEFILE1
  touch $NODEFILE2
  touch $AWKcommand
  echo "{print \"rank \" i++ \"=\" \$1 \" slot=0\"}"   >> $AWKcommand
  echo "{print \"rank \" i++ \"=\" \$1 \" slot=1\"}"   >> $AWKcommand
  echo "{print \"rank \" i++ \"=\" \$1 \" slot=2\"}"   >> $AWKcommand
  echo "{print \"rank \" i++ \"=\" \$1 \" slot=3\"}"   >> $AWKcommand
  echo "{print \"rank \" i++ \"=\" \$1 \" slot=4\"}"   >> $AWKcommand
  echo "{print \"rank \" i++ \"=\" \$1 \" slot=5\"}"   >> $AWKcommand
  echo "{print \"rank \" i++ \"=\" \$1 \" slot=6\"}"   >> $AWKcommand
  echo "{print \"rank \" i++ \"=\" \$1 \" slot=7\"}"   >> $AWKcommand
  for host in $LSB_HOSTS; do echo $host >> $NODEFILE1; done;
  sort -u  ${NODEFILE1} > ${NODEFILE2}
  awk -f $AWKcommand  ${NODEFILE2} > ${RANKFILE}
  NP=$[`cat ${RANKFILE} | wc --lines`]
  echo "."
  cmdmpirun="mpirun -np $NP -hostfile $NODEFILE2 --rankfile ${RANKFILE}"
  echo "========= MPI PARAMETER ================"
  echo "NP = ${NP}"
  echo "mpirun = $(which mpirun)"
  echo "      => ${cmdmpirun}"
  echo "========================================" 
  ## ------------------------------------
  # copy the input file from the SE to the WN
  ## ------------------------------------
  cmd1="lcg-cp -v --vo theophys lfn:/grid/theophys/IS_OG51/Parma/CorsoGrid/${XMLparFILE}.xml file://$(pwd)/${XMLparFILE}.xml"
  cmd2="lcg-cp -v --vo theophys lfn:/grid/theophys/IS_OG51/Parma/CorsoGrid/${EXE}            file://$(pwd)/${EXE}"
  echo "=====================  1(a) ===================="
  echo $cmd1
  echo $cmd2
  echo "=====================  1(b) ===================="
  eval $cmd1
  eval $cmd2
  ls
  echo "=====================  1(c) ===================="
  ## ------------------------------------
  #  Now we the the exectuable and the 
  #  Parameter File
  ## ------------------------------------
  echo "==============================="
  echo "RANKFILE:"
  cat ${RANKFILE}
  echo "==============================="
  echo "==============================="
  echo "NODEFILE2:"
  cat ${NODEFILE2}
  echo "==============================="
  echo "----------------------------- Command ----------------"
  cmd="(${cmdmpirun} ${MYDIR}/$EXE -i ${XMLparFILE}.xml -o ${XMLparFILE}_OUT.xml 2> stderror )"
  cmd2="time (${cmd}) 2> ${XMLparFILE}.timing" 
  echo $cmd
  echo $cmd2
  echo "------------------------------------------------------"
  ls -l ${MYDIR}/$EXE
  chmod +x  ${MYDIR}/$EXE
  ls -l ${MYDIR}/$EXE
  echo "**************************************************"
  echo "**Started at $(date)"
  time (${cmdmpirun} ${MYDIR}/$EXE -i ${XMLparFILE}.xml -o ${XMLparFILE}_OUT.xml 2> stderror)  2> ${XMLparFILE}.timing
  echo "**Ended at   $(date)"
  rm -f $NODEFILE1
  rm -f $NODEFILE2
  rm -f $RANKFILE
  echo "************************************"
  ls -l 
  tar czf localRESULTS.tgz *
  echo "************************************"
  ## ------------------------------------
  # Save the result file from the WN  to the SE
  ## ------------------------------------
  FnameToSave="${XMLparFILE}_${RUNid}.tgz"
  lcg-cr -v --vo theophys -d srm://gridsrm.pi.infn.it/theophys/IS_OG51/Parma/CorsoGrid/${FnameToSave}  \
         -l lfn:/grid/theophys/IS_OG51/Parma/CorsoGrid/${FnameToSave} \
          file://$(pwd)/localRESULTS.tgz
  echo "====================="


The corresponding jdl file is:

  # RUN_RANKED_chroma.jdl
  JobType = "Normal";
  CPUNumber = 64;
  Executable    = "RUN_RANKED_chroma.sh";
  Arguments     = "16_16_16_16_B6.0 id8x4";
  StdOutput     = "std.out";
  StdError      = "std.err";
  PerusalFileEnable = true;
  PerusalTimeInterval = 10;
  InputSandbox  = {"RUN_RANKED_chroma.sh"}; 
  OutputSandbox = {"std.out", "std.err"};
  MyProxyServer = "myproxy.cnaf.infn.it";
  Requirements =(other.GlueCEInfoHostName == "gridce3.pi.infn.it");
  CeRequirements = "hostsmpsize==8 &&  wholenodes==\"true\"  && hostnumber==4";
 
One should note a whole section that create auxiliary file for the mpirun 
(openmp flavour) command to create the actual list of execution nodes and,
very important, to manage MEMORY-CORE affinity.

In future all of this will be dealt with using the mpi-start command.
(...commet in details...) 


One should note that doubling the number of processor the execution time is better
than linear scaling ... 2000 sweep on a 16^4 hypercube.


  real	164m 2.279s   16 cores
  real	 67m40.271s   32 cores
UNIVERSITÀ DI PARMADIPARTIMENTO DI SCIENZE MATEMATICHE, FISICHE E INFORMATICHE

Strumenti Utente

Strumenti Sito

Strumenti Pagina

UNIVERSITÀ DI PARMA
DIPARTIMENTO DI SCIENZE MATEMATICHE, FISICHE E INFORMATICHE
