Parallel Processing

Description

Parallel Processing may refer to Parallel Computing. Parallel computing is a type of computation in which many calculations are carried out simultaneously, operating on the principle that large problems can often be divided into smaller ones, which are then solved at the same time. There are several different forms of parallel computing: bit-level, instruction-level, data, and task parallelism. Parallelism has been employed for many years, mainly in high-performance computing, but interest in it has grown lately due to the physical constraints preventing frequency scaling. As power consumption (and consequently heat generation) by computers has become a concern in recent years, parallel computing has become the dominant paradigm in computer architecture, mainly in the form of multi-core processors. - From Wikipedia, the free encyclopedia

 

This is a Master's course in Computer Engineering(Software, Hardware and Artificial Intelligence). So, This course is recommended for students in Computer Engineering or Computer Science who are interested in scientific computing and would like to obtain a first `hands-on experience' in parallel programming. In this course, students will learn how to design a parallel algorithm for a problem from the area of scientific computing and how to write a parallel program that solves the problem.

Lectures

Lecture #	Title	Status
00	Preface	Completed
01	Introduction	Completed
02	Parallel Processing Architecture	Completed
03	Interconnection Networks	Completed
04	Parallel Techniques	Completed
05	Parallel Algorithm Complexity	Completed
06	Parallel Sorting	Completed
07	Parallel Matrix Algorithms	Completed
08	Parallel Graph Algorithms	Completed
09	Parallel Programming Paradigms	Completed
10	Advanced Topics	Under Review
--	References	Completed

Grading

Your performance will be assessed by assignments, project, class participation and final exam. Their weights are:

•     Final Exam: 70%
•     Project Report: 30%

Assignments have to be typed, if you want them corrected. All of students must work on assignments individually. The assignments play a crucial part in understanding the course material and must be turned in by class time on the due date for full credit. Participation means coming to class, asking questions, taking part in discussions and so on.

Languages and Tools

MPI:

• MPICH2
• MPI Tools
• MPI Implementations
• Simple MPI Programs
• MPI 3.0 Standardization Effort
• LAM/MPI Parallel Computing
• Message Passing Interface (MPI)
• Message Passing Interface - Tutorials
• Message Passing Interface From Wikipedia
• The Message Passing Interface (MPI) standard
• MPI: A Message-Passing Interface Standard
• Parallel Programming with MPI Using MPICH
• Parallel Programming with MPI , by Peter Pacheco
• ARGO-NEW: The Message Passing Interface (MPI)
• MPI-2 : Extensions to the Message-Passing Interface
• How to run program on multiple machines by mpiexec
• Intel® MPI Library 3.2 for Windows* OS Installation Guide
• Tutorial on MPI: The Message-Passing Interface William Gropp
• C++ - Boost.MPI: What's received isn't what was sent! - Stack Overflow

MPI.NET and C#:

• Implementing parallel programming using MPI and C#
• MPI.NET: High-Performance C# Library for Message Passing

MATLAB and MPI:

• MATLAB MPI
• Linux Lab Parallel Matlab FAQ
• MIT Lincoln Laboratory: MatlabMPI
• Writing MATLAB programs using MPI

Hadoop:

• Apache Hadoop
• Writing an Hadoop MapReduce Program in Python

Java and Parallel Computing:

• JPPF
• jPVM
• mpiJava
• MPJ Express
• Parallel Java Library
• How to run JPVM
• JPVM (The Java Parallel Virtual Machine)
• Parallel Java on the RIT CS Parallel Computers

Python and Parallel Processing:

• MPI for Python
• Distributed and Parallel Computing with/for Python
• Python Introduction to Parallel Programming with MPI
• Parallel Processing and Multiprocessing in Python

References

Recommended Textbooks

[1] B. Parhami, "Introduction to Parallel Processing: Algorithms and Architectures" , Kluwer Academic Publishers, 2002.
[2] A. Grama, A. Gupta, G. Karypis and V. Kumar , "Introduction to Parallel Computing" , Second Edition, Addison Wesley, 2003. (PDF files)
[3] H. Casanova, A. Legrand, and Y. Robert "Parallel Algorithms" , CRC Press, 2009.

Other Textbooks

[4] R. Neapolitan , "Foundations Of Algorithms" , 5th Edition, Jones & Bartlett Learning, 2014.
[5] T. Cormen, C. Leiserson, R. Rivest and C. Stein , (CLRS) "Introduction to Algorithms" , Third Edition, MIT press, 2009.
[6] M. J. Quinn, "Designing efficient algorithms for parallel computers" , McGraw-Hill, 1987.
[7] S. G. Akl, "The Design and Analysis of Parallel Algorithms" , Prentice Hall, 1989.
[8] J. JaJa , "Introduction to parallel algorithms" , Addison Wesley, 1992.
[9] F. Gebali, "Algorithms and Parallel Computing" , A John Wiley & Sons, Inc., 2011.
[10] P. Pacheco, “An Introduction to Parallel Programming” , Morgan Kaufmann, 2011.

Useful Links

Laboratories:

• Parallel Algorithms Lab
• Parallel Programming Laboratory
• Parallel Algorithm and Parallel Software
• Computer Architecture and Parallel Processing Lab
• Parallel Algorithms For Solving Stochastic Linear Programs

Parallel Programing:

• Performance Analysis Tools
• A Library of Parallel Algorithms
• Parallel Numerical Algorithms
• Parallel Programming (IN 2147)
• Introduction to Parallel Computing
• SP Parallel Programming Workshop
• Parallel Programming - Using MPI
• Writing Parallel Programs using MPI standard
• Parallel Programming with Microsoft Visual C++
• Parallelization of a Polynomial Extrapolation Code
• Multithreaded Programming with ThreadMentor: A Tutorial
• Tutorial on MPI: The Message-Passing Interface William Gropp

If you have any comments and suggestions about the class and my teaching course, or you know any links about this course, you can send me a mail. I would really appreciate your feedback. Also, If you happen to find an incorrect or non-functional link, please inform me.