Tracking memory usage in embedded real-time MPI executable

I have been doing some digging lately, and I am trying to see what the best approach is to track memory usage in real time software that is always running on Linux (in this case on a bunch of nodes, using MPI). Since there is no end-condition for the code, I want to be able to track the memory usage in real time, to see which MPI image is using the most, which functions are using the most, how much each node is using, and how much the entire executable is using across the entire system.

Are there tools already out there for such a thing? It seems like most things I find are profilers that give results after the fact. Hopefully th开发者_JAVA技巧e answer will be able to track the memory usage of my software, and 3rd party libs my software is using, which I have no access to source code.

---

The OS can usually provide the data dynamically during the run time per executable/thread (see the linux ps command for example).

If you want to trace the callers, then you can override the new and delete operators to gather run-time allocation/release data and trace which function called it and log it somewhere.

---

As littleadv proposed: I would override new/delete (probably malloc/free) and trace the allocations and the size of them. A good way to keep track of such allocations is to just push them to a file or through a network stream to allow another program to analyze the logs. It's fairly simple to write a python/lua script to parse logs (if you format them well) that can present the information to you in a good way.

Overriding 3rd-party software memory allocations can be rough depending on how you link them. Memory overriding through DLL's are often a rough pain so see if you can link them statically to force them to use your allocations.

---

Take a look into vstat. (man pages for vstat)