Tools to visualize multithreaded C++ application call graph, multithreaded code coverage?

I would like to know if there are tools that can

• Help visualize call graph of a large multi-threaded application.
• Specifically I want to see how multiple threads interleaves on one core / executes simu开发者_开发知识库ltaneously on multiple cores.
• The tool would ideally identify possible wait/deadlock/race conditions.

• Ultimately I want to do code coverage in terms of how threads interacts with each other during runtime (multi-thread-wise code coverage tool) so as to find potential multi-threaded bugs.

  I apologize if I haven't explained my question clearly and I would love to provide any details.

---

The VTune Profiler from Intel can do some of what you ask. From the VTune site:

Locks and Waits: Use the Intel® performance profiling tools to quickly find a common cause of slow performance in parallel programs: waiting too long on a lock while the cores are underutilized during the wait.

Timeline Visualizes Thread Behavior: See when threads are running and waiting, and when transitions occur.

If you were looking for something that is open source/free, then Valgrind has an experimental tool called Helgrind that supposedly finds races in multi-threaded programs. I can't comment on it, I haven't used it.

I should note that I haven't been successful in utilizing these or other profilers for multi-threaded debugging and optimizations, and instead I have developed my own techniques.

For identifying lock contention my preferred technique is to use an extended Mutex class that records all the operations done on each instance. I do this in a very lightweight way, so that the application performance doesn't change in a big way.

To identify race conditions I find the brute force approach the best. I just design a test that can be run for an extended period of time, some times this is hours, or days, depending on the case. And I always run my test on at least two different platforms (more if I can), since different OSes use different schedulers and that gives you better coverage.

---

While I can't help (yet!) on most of your issues, I think our C++ Test Coverage tool could provide you with multithreaded test coverage data pretty easily.

This tool instruments your source code; you compile and run that. You end up with (cheap) instrumentation probes in your code representing various blocks. The instrumentation records which parts of your program execute, nominally as a bit vector with one bit per instrumentation probe. At the end of execution (or whenever you like), this bit vector is dumped out and a viewer will show it to you superimposed on the code.

The trick to getting multihread test coverage is to know that we provide you complete control over defining how the instrument probes work; they are macros. So rather than using the default macro of essentially

  probe[n]=true;

on a boolean array, you can instead implement

  probe[n]|=1<<threadid;

on an int array (or something cleverly cheaper by precomputing this value). This likely takes only a few lines of code to implement.

Folks might note this technically has synchronization troubles. That's true, but at most it loses a bit of coverage data, and the odds against it are pretty high. Most people are happy with "pretty good" data rather than perfect. If you insist on perfection, you'll pay a high synchonization price using some atomic update instruction.

We also provide you control over the probe dumping logic; you can revise it to write out thread-specific coverage data (in the tens of lines of custom code range). The test coverage data viewer will then let you see thread-specific coverage (just choose the right coverage vector); it also has built-in facility for easily computing/displaying intersection/union/diff on coverage vectors, which gives you exactly your relation of coverage-per-thread.

---

Concurrency Visualizer (a free add on to visual studio) is really nice visualizer of parallel threads. Including visualizing mutex locks, preemption and callstacks. https://msdn.microsoft.com/en-us/library/dd537632.aspx