Practical Parallel Programming

• 26 hrs lectures
• 16 hrs pc labs
• 10 hrs projects

• 60 pts final exam (written part)
• 10 pts mid-term test (8 pts written part, 2 pts test part)
• 30 pts projects

To get familiar with the architecture of distributed supercomputing systems, their interconnection networks and storage. To orientate oneself in parallel systems on the market, be able to assess communication and computing possibilities of a particular architecture and to predict the performance of parallel applications. Learn how to write portable programs using standardized interfaces and languages, specify parallelism and process communication. To learn how to practically use supercoputer for solving complex engineering problems.
Overview of principles of current parallel system design and of interconnection networks, communication techniques and algorithms. Survey of parallelization techniques of fundamental scientific problems, knowledge of parallel programming in MPI. Knowledge of basic parallel programming patterns. Practical experience with the work on supercomputers, ability to identify performance issues and propose their solution.
Knowledge of capabilities and limitations of parallel processing, ability to estimate performance of parallel applications. Language means for process/thread communication and synchronization. Competence in hardware-software platforms for high-performance computing and simulations.

Von-Neumann computer architecture, computer memory hierarchy, cache memories and their organization, programming in assembly and in C/C++. Knowledge gained in courses PRL and AVS.

• MPI Tutoriál: http://mpitutorial.com/
• Alternativní kurz o paralelním programování http://www.cs.kent.edu/~jbaker/ParallelProg-Sp11/

• Pacecho, P.: Introduction to Parallel Programming. Morgan Kaufman Publishers, 2011, 392 s., ISBN: 9780123742605 URL: download
• Ananth Grama, Anshul Gupta, George Karypis, Vipin Kumar: Introduction to Parallel Computing, Addison-Wesley, 2003, 978-0201648652.Slides: download
• Victor Eijkhout: Parallel Programming in MPI and OpenMP Full book: download web version: https://theartofhpc.com/pcse/
• William Gropp, Ewing Lusk, Anthony Skjellum: Using MPI - 2nd Edition: Portable Parallel Programming with the Message Passing InterfaceUsing MPI - 2nd Edition: Portable Parallel Programming with the Message Passing Interface, MIT Press, 978-0262571326
• Hennessy, J.L., Patterson, D.A.: Computer Architecture - A Quantitative Approach. 5. vydání, Morgan Kaufman Publishers, Inc., 2012, 1136 s., ISBN 1-55860-596-7.

1. Introduction to parallel processing.
2. Architectures with distributed memory,
3. Interconnection networks: topology and routing algorithms, switching, flow control.
4. Technologies of interconnection networks (Infiniband).
5. Distributed file systems (Lustre, HPFS).
6. Message passing interface, pair-wise communications, data types
7. Collective communications and communicators,
8. Hybrid programming OpenMP/MPI and one-sided communications.
9. Parallel code debugging, profiling and tracing. 
10. Programming patterns for parallel programming.
11. Case studies: matrix calculations, linear equation systems 
12. Case studies: solution of PDE systems, finite difference, spectral methods
13. Case studies: Fluid dynamics, N-Body systems, Monte-Carlo.

1. MPI: Point-to-point communications
2. MPI: Collective communications
3. MPI: Communicators
4. MPI: Data types, reduction
5. MPI: Parallel input and output
6. Profiling and tracing of parallel applications
7. Matrix calculations.
8. Finite difference methods.

• A parallel program in MPI on the supercomputer.

Assessment of a project, 10 hours in total and a midterm examination.

• Missed labs can be substituted in alternative dates (monday or friday)
• There will be a place for missed labs in the last week of the semester.