Two Threads and One Core on Coprocessor

In order to run on two threads, we use the following OpenMP setup calls,

omp_set_num_threads(2);
kmp_set_defaults("KMP_AFFINITY=compact");

where the first call sets the number of threads we will use and and second call sets the affinity variable, KMP_AFFINITY of the threads across the cores. To compile the code, use the following command,

$ icc -openmp -mmic -vec-report=3 -O3 helloflops2.c -o helloflops2

moving this over to the coprocessor and executing results in the following output,

%  ./helloflops2
Initializing
Starting Compute on 2 threads
GFlops =     51.200, Secs =      1.301, GFlops per sec =     39.358

This result is sensible and is nearly exactly in line with the maximum achievable computation rate of two threads on a single core with single precision accuracy. Specifically,

Peak Single Precision FLOPs = Clock Frequency x Number of Cores x 16 Lanes x 2 (FMA) FLOPs/cycle

and for our Intel Xeon Phi processor,

1.238 GHz x 1 Core x 16 Lanes x 2 (FMA) FLOPs/cycle = 39.616 GFlops/s

and therefore, we have achieved 99.3% of the theoretical maximum computation rate.

122 Threads and 61 Cores on Coprocessor

So to proceed, compile the code on the host,

$ icc -openmp -mmic -vec-report=3 -O3 helloflops3.c -o helloflops3

and transfer the output to the coprocessor. Then on the coprocessor, we may change the environment variables with the following commands,

% export OMP_NUM_THREADS=122
% export KMP_AFFINITY=scatter

where we are requesting that the threads be "scattered" across the 61 available cores, rather than completely filling a single core with threads before proceeding to the next. Now, on the coprocessor, run the executable,

%  ./helloflops3
Initializing
Starting Compute on 122 threads
GFlops =   3123.200, Secs =      1.388, GFlops per sec =   2250.830

which is 93.1% of the theoretical maximum calculation rate for single precision floats.

References

• Open MP Tutorials
• Automatic Vectorization

* OpenMP? Tutorials: Tutorial

One Thread and One Core on Coprocessor

move to the relevant directory and launch the executable

% ./helloflops1
Initializing
Starting Compute
GFlops =     25.600, Secs =      3.900, GFlops per sec =      6.563

This test continues to be an anomaly for our system, however, and only achieves 6.5 GFlop/s. Theoretically, we should expect 17.5 GFlop/s (half the 34.9 GFlop/s performance of a single core since scheduling skips every other clock cycle with only one thread running).

Vectorization

We can immediately see the impact of vectorization by disabling it via the compile option -no-vec

%  icc -mmic -no-vec -vec-report=3 -O3 helloflops1.c -o helloflops1novec

which results in the output

%  ./helloflops1novec
Initializing
Starting Compute
GFlops =     25.600, Secs =     62.399, GFlops per sec =      0.410

Which we immediately see is the expected factor of 16 slow-down from our vectorized code.

Scaling to Multiple Cores

$  cd liamb315

And from the host, we may access the 0th Many Integrated Core (MIC) chip via

fa[k] = a * fa[k] + fb[k];

To compile the helloflops1.c code using the Intel C compiler, use the following command

$  icc -mmic -vec -report=3 -O3 helloflops1.c -o helloflops1

Here, -mmic requests the code be generated for the Intel Many Integrated Core (MIC) architecture, -vec-report=3 indicates to generate a vector report and -O3 indicates to use standard optimization techniques.

Now, copy the helloflops1 executable object over to the mic0 via the command

$  scp helloflops1 mic0:~/liamb315 

Now, ssh to the mic0 chip

$  ssh mic0

move to the relevant directory and launch the executable

%  cd liamb315
%  ./helloflops1

Vectorization

Seed Algorithm

Setting Up

Logging Onto Processor and Coprocessor

ssh root@phiphi.t2.ucsd.edu

ssh mic0

Examples

Simple Calculation

fa[k] = a * fa[k] + fb[k];

which is a fused multiply and add (FMA) instruction, representing the primary compute capability of the coprocessor.

Intel Xeon Phi Coprocessor

Overview

The main goal of this twiki is to document a series of training sessions to teach the basics of doing a particle physics analysis from a practical perspective. The loose set of topics that will be covered are:

• ROOT -- the C++ based data analysis package that is used in HEP
• CMSSW -- the software framework used by the collaboration
• CMS2 -- the software sub-framework used by the UCSD/UCSB/FNAL group (a.k.a. SNT)
• A full analysis example -- measuring the Z cross section

These topics are not necessarily ordered in any particular way and are only loosely related.

Order of topics (Subject to change)

ROOT

• Documentation
  • Home: http://root.cern.ch/drupal/
  • User's guide: http://root.cern.ch/drupal/content/users-guide
  • Classes: http://root.cern.ch/root/html534/ClassIndex.html
  • Tutorial Example Code: http://root.cern.ch/root/html/tutorials/
• To Check out the example code:

git clone https://github.com/kelleyrw/AnalysisTutorial

Lesson 1

• Studied the basics of TTree and made efficiency plots for some tracking variables
• Reading: ROOT user's guide: read ch 1-3,7,12
• Example code: Lesson 1

TTree example

This tree was filled per event and contains a unfiltered list (in the form of a std::vector) of TrackingParticles per event:

Events
  |
  --> list of TrackingParticles
             |
             --> Tracking particle information (p4, # sim hits, d0, dz, ...) 
             --> Matching reconstructed Track info (bogus values filled if no matching track).

The tree is small (1000 events) and I was able to check into the repository (https://github.com/kelleyrw/AnalysisTutorial/blob/master/week1/trees/tracking_ntuple.root). All the branches should be the same size:

// TrakingParticle info
std::vector<LorentzVector> tps_p4:  four momentum
std::vector<int> tps_pdgid:         pdg particle ID code: http://pdg.lbl.gov/2007/reviews/montecarlorpp.pdf
std::vector<double> tps_d0:         transverse impact parameter
std::vector<double> tps_dz:         longitudinal impact parameter
std::vector<bool> tps_matched:      matched to track?  true/false 
std::vector<int> tps_charge:        charge
std::vector<int> tps_nhits:         # of simulated hits

// reco track info
std::vector<LorentzVector> trks_p4:  four momentum
std::vector<double> trks_tip:        transverse impact parameter  (from the TrackingParticle vertex)
std::vector<double> trks_lip:        longitudinal impact parameter  (from the TrackingParticle vertex)
std::vector<double> trks_d0:         transverse impact parameter (using the trajectory builder)
std::vector<double> trks_dz:         longitudinal impact parameter (using the trajectory builder)
std::vector<double> trks_pterr:      pt uncertainty
std::vector<double> trks_d0err:      d0 uncertainty
std::vector<double> trks_dzerr:      dz uncertainty
std::vector<double> trks_chi2:       chi^2 of the track's fit
std::vector<int> trks_ndof:          # degrees of freedom
std::vector<int> trks_nlayers:       # number of valid layers with a measurement
std::vector<bool> trks_high_purity:  # track passes high purity quality requirement