Christian’s corner on HPC

As announced in a previous post already, I was involved in two workshops attached to the HPCS 2009, hosted by the HPCVL in Kinston, ON, Canada. Being back in the office now I found some time to upload my slide sets. Obviously I can only make my own slides public.

Using OpenMP 3.0 for Parallel Programming on Multicore Systems [abstract]

Ruud van der Pas, Sun Microsystems; Dieter an Mey and Christian Terboven, RWTH Aachen University.

• Tasking in OpenMP 3.0 (Christian Terboven).
• Data Race Detection in OpenMP programs using the Sun Thread Analyzer (Christian Terboven).
• OpenMP in the Real World (Christian Terboven and Dieter an Mey).

Parallel Programming in Visual Studio 2008 on Windows HPC Server 2008 [abstract]

Christian Terboven, RWTH Aachen University.

• Windows HPC Server 2008: Overview (Christian Terboven).
• Windows HPC Server 2008: user’s point of view (Christian Terboven).
• Using Microsoft Visual Studio 2008 (Christian Terboven).
• HPC Tools Portfolio: Shared-Memory Parallelization on Windows (Christian Terboven).
• HPC Tools Portfolio: Message-Passing with MPI on Windows (Christian Terboven).
• Case Studies … and an Outlook into the Future (Christian Terboven).

I was told that the book I covered in my last review can be get way cheaper over at Packt Publishing. There is also an eBook version available.

Just recently – in May 2009 – I gave two lectures on Multithreading with C# for Desktop Applications. I found there are quite a few books available that cover the .NET Thread class when talking about Windows programming in general, but the book C# 2008 and 2005 Threaded Programming: Beginner’s Guide is only about, well, Multithreading with C#. The subtitle Exploit the power of multiple processors for faster, more responsive software also states that both algorithmic parallelization as well as the separation of computation from a graphical user interface (GUI) is covered in here, and this is exactly what I was looking for. The book is clearly marked as a Beginner’s Guide and is well-written for that aspect, so if you already know about Multithreading and just want to learn about how to do this with C#, you might find the book to proceed too slowly. If you are uncertain or clearly new to this subject, then this book might do it’s job very well for you.

Chapters one and two start with a brief motivation of why the shift towards multicore processors has such an important influence on how software has to be designed and written nowadays and also contain a brief description of the typical pitfalls you may run into when parallelizing software. Chapter three describes the BackgroundWorker component, which is the simplest facility to separate the computation from the user interface in order to keep it responsible. Chapters four and five cover the most important aspects of the Thread class as well as how to use Visual Studio to debug multithreaded programs. Chapters six to nine describe how to apply parallelization to a range of common problems and design cases, for example howobject-oriented features of C# and the garbage collector of .NET play along with the Thread class and what to take care for when doing Input/Output and Data Access. Chapter ten explains in detail how GUIs and Threads work together (or not) and how to design you GUI and your application to report progress to the GUI from threads, for example. When doing so there are some rules one has to obey and I found the issues that I was not aware of before very well-explained. Chapter eleven gives a brief overview of the .NET Parallel Extensions – which will be part of .NET 4.0 – such as the Parallel class and PLINQ. The final chapter twelve tries to put all things together into a single application.

Most aspects of Multithreading with C# are introduced by first stating a problem / motivation (with respect to the example code), then showing the solution in C# code and discussing the effects of it and finally explaining the concept in some more detail, if needed. The two example codes, a text message encryption and decryption software and an image analysis tool, are consistently extended with the new features that have been introduced. I personally did not like that there is so much example code shown in the book, although people new to Multithreading might find studying the source code helpfull. With a strong focus on explaining and discussing example the book is not well-suited as a reference, but it does not say to do so. Actually I think that once you are familiar with certain aspects of Multithreading with C#, MSDN does a good job of serving as a reference.

The book is published by Packt Publishing and has been released in January 2009. The price of about 30 Euro for about 420 pages at amazon.de in Germany is affordable for students, I think. Regards to Radha Iyer at Packt Publishing for making this book available for me in time.

Let me point you to some HPC events in June 2009.

5th International Workshop on OpenMP (IWOMP 2009) in Dresden, Germany. The IWOMP workshop series focuses on the development and usage of OpenMP. This year’s conference is titled Evolving OpenMP in an Age of Extreme Parallelism – I think this phrase is a but funny, but nevertheless one can clearly observe a trend towards Shared-Memory parallelization on the node of even the extremely parallel machines. Attached to the conference is a two day meeting of the OpenMP language committee. The language committee is currently discussing a long list of possible items for a future OpenMP 3.1 or 4.0 specification, including but not limited to my favorites Composability (especially for C++) and Performance on cc-NUMA system. Bronis de Supinski, the recently appointed Chair of the OpenMP Language Committee, will give a talk on the current activities of the LC and how the future of OpenMP might look like – I hope the slides will be made public soon after the talk. Right before the conference there will also be a one day tutorial for all people interested in learning OpenMP (mainly given by Ruud van der Pas – strongly recommended).

High Performance Computing Symposium 2009 (HPCS) in Kingston, Canada. HPCS is a multidisciplinary conference that focuses on research involving High Performance Computing and this year it takes place in Kingston. I’ve never been at that conference series, so I am pretty curious how it will look like. Attached to the conference are a couple of workshops, including Using OpenMP 3.0 for Parallel Programming on Multicore Systems – run again by Ruud van der Pas and us, and Parallel Programming in Visual Studio 2008 on Windows HPC Server 2008 – organized by us as well. Here in Aachen, the interest in our Windows-HPC compute service is still growing fine and thus we have usually around 50 new participants in our bi-yearly training events. The HPCVL people asked explicitly to cover parallel programming on Windows in the OpenMP workshop, so we separated this aspect out without further ado to serve it well. The workshop program can be found here.

International Supercomputing Conference (ISC 2009) in Hamburg, Germany. ISC titles itself as Europe’s premier HPC event – while this is probably true it is of course smaller than the SC events in the US, but usually better organized. Without question you will find numerous interesting exhibits and can listen to several talks (mostly by invited speakers), so please excuse the self-marketing of me pointing to the Jülich Aachen Research Alliance (JARA) booth in the research space where we will show an interactive visualization of large-scale numerical simulation (damage of blood cells by a ventricular device – pretty cool) as well as give an overview of our research activities focused on Shared-Memory parallelization (we will distribute OpenMP syntax references again). If you are interested in HPC software development on Windows, feel invited to stop by at our demo station at the Microsoft booth where we will have many demos regarding HPC Application Development on Windows (Visual Studio, Allinea DDTlite and Vampir are confirmed, maybe more …). And if you are closely monitoring the HPC market, you have probably heard about ScaleMP already, the company aggregating multiple x86 system into a single (virtual) system over InfiniBand – obviously very interesting for Shared-Memory parallelization. If you are interested, you can hear about our experiences with this architecture for HPC.

If you want to meet up during any of these events just drop me an email.

The Second Meeting of the German Windows-UPC User Group took place on March 30th and 31st in Dresden with about 80 participants. The ZIH of TU Dresden was the kind host of this event, which included both a series of talks and several booth places for various vendors. Having the event begin on Monday afternoon allowed for comfortable travel and after the first part was over the social event took place in the Luisenhof am Elbhang restaurant, with a great view over the Elbtal. The second part of the presentation program ended on Tuesday afternoon. According to the feedback gathered so far, this format is very well accepted and will also be used for the next meeting on March 08th and 09th at Schloss Birlinghoven to be hosted by the Fraunhofer Institute SCAI.

Personally, I think this event was successful. We had a good mix of technical talks as well as presentations of how and where Windows-HPC is used today. Some of these users have presented at last year’s event already – they made progress, but most were new. The versions of the HPC ISV-Codes that will be released this year will be well-integrated into HPC Server 2008 (for some products it will be the second version supporting Windows-HPC), but it seems most codes still have quite some potential for out-of-the-box performance improvements. The service providers – such as cluster integrators – that were presenting and / or had booths at the event have made their first experiences (good and bad, of course) with HPC on Windows. Many thanks to the sponsors for supporting this event, in alphabetical ordering: GNS Systems, Intel, Microsoft, Megware, Myricom, Sun and Transtec! The presentation program consisted of several sessions of which I will provide a brief summary in the rest of this post.

The event was opened by a brief welcome talk of Dieter an Mey (RWTH Aachen) and Wolfgang Dreyer (Microsoft) outlining the agenda of the next 1.5 days. The first main presentation was given by Matej Ciesko (Microsoft) and titled Highly-productive Computing with Windows-HPC (no slides). With respect to cc-NUMA awareness and thread scheduling, he explained the new features of the upcoming Windows Server 2008 R2, which will also be the foundation of the next HPC Server (v3). He also gave a brief overview of the activities Microsoft is undertaking to push parallel programming into mainstream and which tools will be made available with Visual Studio 2010.

The second session was titled HPC Infrastructure and the first talk was on the evaluation of integrating Windows-HPC into existing computing environments (slides (de)), given by Harry Schlagenhauf (Science+Computing). It was followed by Jürgen Gretzschel (MegWare) with an overview of MegWare’s activities and developments (slides (de)) in and for the HPC market. Markus Fischer (Myricom) presented on Myricom’s low-latency support of the Network Direct interface on both Myrinet and 10 Gb Ethernet networks (slides (en)). This session as well as the first day was closed by Wolfgang Nagel (TU Dresden), who talked about the HPC and tool development projects at the ZIH of TU Dresden as well as the goals and activities of the Gauß alliance in Germany (no slides).

The first session of the second day was captioned Performance Analysis and the first talk was given by Xavier Pillons (Microsoft) on exactly this topic (slides (en)). He presented and demoed on various build-in tools of Windows to measure the performance of the whole system and single applications as well, putting a focus on the xperf tool of the Windows Performance Toolkit. He also talked about his best practices of troubleshooting system performance issues and about his experience of running LINPACK benchmarks for Top500 submissions on Windows. He was followed by Holger Brunst (TU Dresden) who presented the Vampir GUI on Windows (slides (de)) and how to collect MPI traces. The last talk of this session was given by Christian Terboven (RWTH Aachen); he discussed the HPC tools portfolio for debugging, performance tuning and Shared-Memory parallelization on Windows (slides (en)).

The next session was on Running Windows-HPC Clusters and also included the project presentations of the Microsoft academic program of last year. Michael Wirtz (RWTH Aachen) presented on how the cluster at RWTH Aachen University has been designed and is managed today (slides (de)) and what the current challenges look like. Johannes Habich (University of Erlangen) talked about some of their projects on the Windows-HPC platform and about how they are doing the resource accounting for their HPC customers (slides (de)). Thomas Blümel (TU Dresden) reported on the problems of getting their cluster, sponsored by Microsoft and Dell, up and running and what they would like to see in order to make the setup process more straight-forward (slides (de)).

The 2008 Microsoft academic program was a student competition about bringing HPC codes to the Windows platform. Christopher Schleiden (RWTH Aachen) presented how a C++ 3D Navier-Stokes solver making use of libraries such as ParMETIS and DDD has been ported (slides (de)) and how to get good performance. Roman Parys (University of Tübingen) reported on a project about data compression and volumetric rendering of giga-voxel data sets (slides (en)) to drive a large-scale video cluster. Johannes Hoppe and Johannes Hofmeister (both University of Applied Sciences at Heidelberg) demoed their clustered neural network for pattern recognition in image and video data, written in .NET (no slides).

After the lunch break at the student cafeteria, the next session was titled HPC Applications. Karsten Reineck and Horst Schwichtenberg (both Fraunhofer SCAI) compared the performance of different ISV-Codes on Windows and Linux and, well, I got the impression that some ISVs still have quite a way to go (slides (de)). Peter Kirsch and Markus Kirsch (both ICT AG) compared the performance of Fluent for Windows-HPC on Myrinet versus Gb Ethernet and instructed how to setup this application for a cluster (slides (de)). Sorin Serban (Visual Numerics) demoed the IMSL numeric library and explained how the library has been parallelized with OpenMP and what performance improvements the user can expect from different parts of the library (slides (de)).

The last session was named HPC Perspectives. Mario Deilmann (Intel) gave a talk that was split into two parts (slides (en + de)): The first was on the features and capabilities of Intel Parallel Studio and how it integrates in the Windows-HPC development story as an extension to Visual Studio; the second was on the Ct research project, a language-in-the-C++-language for throughput computing that was just announced to become available at Intel’s IDF. The closing talk was given by Torsten Langner (Microsoft), who introduced Microsoft’s offerings for Service Oriented Architectures (SOA) in the HPC world. With a very interesting example from the financial business, he explained what a WCF broker node is doing and what class of applications is expected to make use of this technology (slides (en)).

Although it is high time to deliver the second part of this blog post series, I decided to squeeze in one additional post which I named part “1.5″, as it will cover some experiments with SMXV in C#. Since I am currently preparing a lecture named Multi-Threading for Desktop Systems (it will be held in German, though) in which C# plays an important role, we took a closer look into how parallelism has made it’s way into the .NET framework version 3.5 and 4.0. The final post will then cover some more tools and performance experiments (especially regarding cc-NUMA architectures) with the focus back on native coding.

First, let us briefly recap how the SMXV was implemented and examine how this can look like in C#. As explained in my previous post, the CRS format stores just the nonzero elements of the matrix in three vectors: The val-vector contains the values of all nonzero elements, the col-vector contains the column indices for each nonzero element and the row-vector points to the first nonzero element index (in val and col) for each matrix row. Having one class to represent a CRS matrix and using an array of doubles to represent a vector, the SMXV operation encapsulated by the operator* can be implemented like this, independent of whether you use managed or unmanaged arrays:

public static double[] operator *(matrix_crs lhs, double[] rhs)
{

   double[] result = new double[lhs.getNumRows()];

   for (long i = 0; i < lhs.getNumRows(); ++i)

   {

      double sum = 0;

      long rowbeg = lhs.row(i);

      long rowend = lhs.row(i + 1);

      for (long nz = rowbeg; nz < rowend; ++nz)

         sum += lhs.val(nz) * rhs[ lhs.col(nz) ];

      result[i] = sum;

   }

   return result;
}

We have several options to parallelize this code, which I wil present and briefly discuss in the rest of this post.

Threading. In this approach, the programmer is responsible for managing the threads and distributing the work onto the threads. It is not too hard to implement a static work-distribution for any given number of threads, but implementing a dynamic or adaptive work-distribution is a lot of work and also error-prone. In order to implement the static approach, we need an array of threads, have to compute the iteration chunk for each thread, put the threads to work and finally wait for the threads to finish their computation.

//Compute chunks of work:Thread[] threads = new Thread[lhs.NumThreads];

long chunkSize = lhs.getNumRows() / lhs.NumThreads;

//Start threads with respective chunks:

for (int t = 0; t < threads.Length; ++t)

{

   threads[t] = new Thread(delegate(object o)

   {

      int thread = (int)o;

      long firstRow = thread * chunkSize;

      long lastRow = (thread + 1) * chunkSize;

      if (thread == lhs.NumThreads - 1) lastRow = lhs.getNumRows();

      for (long i = firstRow; i < lastRow; ++i)

      { /* ... SMXV ... */ }

   });

   //Start the thread and pass the ID:
   threads[t].Start(t);

}

//Wait for all threads to complete:

for(int t = 0; t < threads.Length; ++t) threads[t].Join();
return result;

Instead of managing the threads on our own, we could use the thread pool of the runtime system. From a usage point of view, this is equivalent to the version shown above, so I will not discuss this any further.

Tasks. The problem of the approach discussed above is the static work-distribution that may lead to load imbalances, and implementing a dynamic work-distribution is error-prone and depending on the code it also may be a lot of work. The goal should be to distribute the workload into smaller packages, but doing this with threads is not optimal: Threads are quite costly in the sense that creating or destroying a thread takes quite a lot of time (in computer terms) since the OS is involved, and threads also need some amount of memory. A solution for this problem are Tasks. Well, tasks are quite “in” nowadays with many people thinking on how to program multicore systems and therefore there are many definitions of what a task really is. I have given mine in previous posts on OpenMP and repeat it here briefly: A task is a small package consisting of some code to execute and some private data (access to shared data is possible, of course) which the runtime schedules for execution by a team of threads. Actually it is pretty simple to parallelize the code from above using tasks: We have to manage a list of tasks and have to decide how much work a task should do (in terms of matrix lines), and of course we have to create and start the tasks and finally wait for them to finish. See below:

//Set the size of the tasks:

List<Task> taskList = new List<Task>();

int chunkSize = 1000;

//Create the tasks that calculate the parts of the result:

for (long i = 0; i < lhs.getNumRows(); i += chunkSize)

{

   taskList.Add(Task.Create(delegate(object o)

   {

      long chunkStart = (long)o;

      for(long index = (long)chunkStart;

      index < System.Math.Min(chunkStart + chunkSize, lhs.getNumRows()); index++)

      { /* ... SMXV ... */ }

   }, i));

}

//Wait for all tasks to finish:

Task.WaitAll(taskList.ToArray());
return result;

Using the TPL. The downside of the approach discussed so far is that we (= the programmer) has to distribute the work manually. In OpenMP, this is done by the compiler + runtime – at least when Worksharing constructs can be employed. In the case of for-loops, one would use Worksharing in OpenMP, With the upcoming .NET Framework version 4.0 there will be something similar (but not so powerful) available for C#: The Parallel class allows for the parallelization of for-loops, when certain conditions are fulfilled (always think about possible Data Races!). Using it is pretty simple thanks to support for delegates / lambda expressions in C#, as you can see below:

Parallel.For(0, (int)lhs.getNumRows(), delegate(int i)

{
   /* ... SMXV ... */

});
return result;

Nice? I certainly like this! It is very similar to Worksharing in the sense that you instrument your code with further knowledge to (incrementally) add parallelization, while it is also nicely integrated in the core language (which OpenMP isn’t). But you have to note that this Worksharing-like functionality is different from OpenMP in certain important aspects:

• Tasks are used implicitly. There is a significant difference between using tasks underneath to implement this parallel for-loop, and Worksharing in OpenMP: Worksharing uses explicit threads that can be bound to cores / numa nodes, while tasks are scheduled onto threads on the behalf of the runtime system. Performance will be discussed in my next blog post, but tasks can easily be moved between numa nodes and that can spoil your performance really. OpenMP has no built-in support for affinity, but the tricks how to deal with Worksharing on cc-NUMA architectures are well-known.
• Runtime system has full control. To my current knowledge, there is no reliably way of influencing how many threads will be used to execute the implicit tasks. Even more: I think this is by design. While it is probably nice for many users and applications when the runtime figures out how many threads should be used, this is bad for the well-educated programmer as he often has better knowledge of the application than the compiler + runtime could ever figure out (about data access pattern, for instance). If you want to fine-tune this parallelization, you have hardly any option (note: this is still beta and the options may change until .NET 4.0 will be released). In OpenMP, you can influence the work-distribution in many aspects.

PLINQ. LINQ stands for language-integrated query and allows for declarative data access. When I first heard about this technology, it was demonstrated in the context of data access and I found it interesting, but not closely related to the parallelism I am interested in. Well, it turned out that PLINQ (+ parallel) can be used to parallelize a SMXV code as well (the matrix_crs class has to implement the IEnumerable / IParallelEnumerable interface):

public static double[] operator *(matrix_crs_plinq lhs, double[] rhs)

{

   var res = from rowIndices in lhs.AsParallel().AsOrdered()

             select RowSum(rowIndices, lhs, rhs);

   double[] result = res.ToArray();

   return result;
}

public static double RowSum(long[] rowIndices, matrix_crs_plinq lhs, double[] rhs)

{

   double rowSum = 0;

   for (long i = rowIndices[0]; i < rowIndices[1]; i++)

   {

      rowSum += lhs.val(i) * rhs[lhs.col(i)];

   }

   return rowSum;
}

Did you recognized the AsParallel() in there? That is all you have to do, once the required interfaces have been implemented. Would I recommend using PLINQ for this type of code? No, it is meant to parallelize queries on object collections and more general data sources (think of databases). But (for me at least) it is certainly interesting to see this paradigm applied to a code snippet from the scientific-technical world. As PLINQ uses TPL internally, you will probably have the same issues regarding locality, although I did not look into this too closely yet.

Let me give credit to Ashwani Mehlem, who is one of the student workers in our group. He did some of the implementation work (especially the PLINQ version) and code maintenance of the experiment framework.

Let me use this well-linked position to announce three HPC events in March 2009.

Future Directions in High Performance Computing: 2009 – 20018. On Monday, March 23th 2009, Dr. Horst Simon (Associate Laboratory Director for Computing Sciences at Lawrence Berkeley National Laboratory) will give a presentation on future directions in HPC. The talk will be given in room 637 in the SuperC building of RWTH Aachen University at 11:00h. More information can be found at the associated webpage.

Parallel Programming in Computational Engineering and Science (PPCES). This event will continue the tradition of previous annual SunHPC events taking place in Aachen since 2001, which have been organized by the RWTH Aachen University and Sun Microsystems. We are changing the format a little bit with each year, reflecting changes in the hardware architecture (Sun UltraSparc to Intel Nehalem) and the operating systems (Solaris-only to Linux plus Windows) and programming languages (Fortran-dominated examples are disappearing). This year we intend to cover Linux, Windows and Solaris and offer all demos, exercises and examples on both Linux and Windows, demonstrating that Windows-HPC is a first class citizen with respect to our user support. The event will take place in the lecture room and the pc pool of the Center for Computing and Communication at RWTH Aachen University. More information and the option to register can be found at the event website.

2nd Meeting of the German Windows-HPC User Group (Win-HPC UG). The second meeting of the German Windows-HPC User Group will be hosted by the Technical University of Dresden and take place on March 30th and 31th. The agenda is not yet online (but will be shortly), but I am looking forward to some interesting talks. The event is sponsored by Intel, Microsoft, Megware, Sun and Transtec – we are hoping and doing our best to repeat the success of the predecessor event that took place last year in Aachen. It was harder than usual to get technical experts from these companies for a talk, but in the end we are quite happy (if you think you have something interesting to share, please contact us immediately). More information can be found at the event website and our plan is to schedule all talks between Monday 14:00h and Tuesday 17:00h to allow for comfortable travel.

Because of conflicting dates and my intent of keeping some scheduled events as they were planned for quite some time I decided against attending Microsoft’s PDC 2008 – which probably was the wrong decision . The mainstream media is talking a lot about Windows Azure and Windows 7 – which are interesting news items for sure – but not closely related to my “business”, maybe except for the fact that Windows Server 2008 R2 will be capable of handling more cores than just 64.

According to my opinion PDC 2008 brought us great news regarding the feature set of Visual Studio 2010 and Microsoft’s activities on multi-core (aka Shared-Memory parallel) programming and I hope to see and learn more about this during SC08 in about two weeks from now. Meanwhile you can do the following, if you are like me interesting in this stuff:

• Grab the Visual Studio 2010 and .NET Framework 4.0 CTP virtual machine image from this download site and play with it. From my experience with the Visual Studio 2008 beta program I know that although my laptop is dual-core and has 2 GB of memory, running the VM on that hardware does not make a lot of fun. I am so happy that just recently I virtually “found” a two-socket quad-core (Clovertown) machine that is not suited for production as it is a Intel Software Development test machine with a pre-series chipset and pre-series CPU. I released that machine instantly from doing some stupid software and system testing task .
• Watch the PDC 2008 videos that are available online. Channel 9 has a section covering PDC 2008. That is nice if you are in front of a PC and connected to the Internet, but downloading all the interesting videos from that site is a pain. Greg Duncan has put together a site containing the download links to the videos (in various formats) and the PowerPoint slides here. Grep the content for “parallel” or “concur” or “Studio”!

Last week we had the Windows HPC Server 2008 launch event in Germany. Microsoft is doing a lot of marketing for this product, which one may like or dislike. However, HPC just recently got the attention by some Channel8 activists. Channel8 is a forum with (video) content produced by students meant for students, covering all sorts of Microsoft products and technologies as well as other interesting “hot” technologies topics.

Following this link you will find five videos (as of October 21, 2008) on HPC (in German only). The first two videos are interviews with Wolfgang Nagel and Matthias Müller from the Center for Information Services and High Performance Computing at Dresden University covering HPC in general. The next two videos are interviews with Dieter an Mey and me from the Center for Computing and Communication at RWTH Aachen and we are talking about our experiences and activities with HPC on Windows. The fifth video shows excerpts of our interview again, but covers parallel programming with MPI and (especially) OpenMP in general.

Just this week Allinea released it’s DDTLite plugin for Visual Studio 2008. I have been using a beta version for a couple of weeks now and in my humble opinion, DDTLite extends the MPI cluster debugger of Visual Studio 2008 with a must-have feature for any parallel debugger: Individual process control for MPI programs. With the capabilities provided by the MPI cluster debugger of Visual Studio, debugging MPI programs can be a pain as it is not possible to control MPI processes individually. That means if you select one process and execute it step-by-step, the other process will continue as well and there is no chance of stopping it from doing so (e.g. freezing as you can do with threads). This blog post is not intended to become an Allinea commercial, but I want to briefly demonstrate what DDTLite can do for you.

In order to debug MPI programs, you have to go to the project properties, choose Debugging in the left column, and select the MPI Cluster Debugger as the debugger to launch. Additionally you have to provide the following options (listed below along with my advices):

• MPIRun Command: The location of mpirun. Specify the full path to the mpiexec program here, do not use “”, and do not omit the .exe extension.
• MPIRun Arguments: Arguments to pass to mpirun, such as number of processes to start.
• MPIShim Location: Location of mpishim.exe. As far as my experience goes, you avoid trouble if you copy mpishim.exe to a path that does not contain any white space (the original location is C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE\Remote Debugger\x86 on a 32-bit system), again do not omit the .exe extension.

That said, your configuration could look like this:

If you then start the debugger (e.g. via F5), two MPI processes will be started and you can switch between them using the Processes window (you can enable that window via the menu: Debug –> Windows –> Processes):

From the menu via Tools –> Options… –> Debugging (in the left column) you can set the option Break all processes when one process breaks to influence what happens when a breakpoint is encountered. For the case of debugging MPI programs, you probably want this option to be enabled! But – as already mentioned above – when all processes were interrupted after a breakpoint has been hit, you cannot continue with just one process step-by-step, as the other process will always do a step as well. And this is where DDTLite comes into play…

After the plugin has been enabled (via the menu: Tools –> Add-in Manager…) you are presented with several additional windows, among this is the Selected Processes and Threads window to select and switch between processes and threads, as shown above. Via the Groups – Parallel View window you can select individual processes (in the screenshot above you can see that only the MPI process with rank 0, out of two MPI processes, is selected) and then control the selection (selecting a group of processes is possible as well) using the Visual Studio debugger as you do with a serial program. All MPI processes not currently selected stand still!

There is more in DDTLite: For example you can select a variable and go to the Variable – Parallel View window to receive a list of variable values by MPI rank (the screenshot below shows the iMyRank member of a struct type named data, which denotes the MPI rank).

Of course there are even more capabilities provided by DDTLite, but you can go to the product homepage and find out for yourself by grabbing a 30-day trial version (I used that trial to create the screenshots shown in this blog post). But I would like to add one additional note on the question of how many MPI processes you should use for debugging. Most parallel debuggers (including DDTLite and DDT) are advertised that they are capable of controlling hundreds (and even thousands) of MPI processes. I think that you will hardly ever need that! Instead, I bet that in 99% of the cases in which your MPI programs works fine with one and two processes but fails when using more, you will find the issue by using three or maybe five processes with your debugger. That is all you need for finding the usual off-by-one work-distribution error and similar things .