• KernelTraceSystems

Here are some links to information about different kernel tracing systems:

Table of Contents:

General Purpose tracing systems

Some major Linux general-purpose tracing systems are:

• ptrace - ability to trace syscall entry and exit, and signal delivery, to a process (also used for debugging a process)
  • see "man ptrace" and "man strace"

• SystemTap - SystemTap is a system for building and executing tracing and sampling systems that can be applied to a running Linux system

  • see SystemTap

• LTTng - Linux Trace Toolkit, next generation

  • see LinuxTraceToolkit

• LKST - Linux Kernel State Tracer

  • see LinuxKernelStateTracer

Special Purpose tracing systems

There are some other notable special-purpose kernel tracing systems:

• KFT - Kernel Function Trace - traces functions to show function durations and call graphs

  • see KernelFunctionTrace

• latency trace - RT-preempt tool for measuring interrupt and mutex latency

  • The latency tracer is embedded in the RT-preempt patch - see RealtimePreemption and http://lwn.net/Articles/97811/

• block tracer (blktrace) - allows you to see exactly what is going on in the block layer for a given queue

  • Introduction by Jens Axboe: http://lwn.net/Articles/148761/

  • Execellent presentation: http://www.gelato.org/pdf/apr2006/gelato_ICE06apr_blktrace_brunelle_hp.pdf

  • Guide to using is at: https://secure.engr.oregonstate.edu/wiki/CS411/index.php/Blktrace_Guide

  • This appears to have been mainlined as of 2.6.17

  • Timeline utility (blktrace post-processing tool): described here

• delay accounting patches - collect statistics about the delays that are experienced by each task on the system

  • see http://lkml.org/lkml/2006/5/2/30

Trace Infrastructure

• KProbes - grew out of dprobes, with information at: http://dprobes.sourceforge.net/

  • see an excellent tutorial at: http://www-users.cs.umn.edu/~boutcher/kprobes/

  • The mainline version of the KProbes supports x86,Alpha and PPC64 architectures. A MIPS implementation has been completed on the 2.6.16 kernel and tested on the Toshiba TX49 platform. Patch is available in the CELF archive.
  • Kprobes implementation for ARM has been completed on the 2.6.16 kernel and tested on the OMAP5912 OSK platform. Patch is available in the CELF archive.
• [would be nice to have some djprobe stuff here]

Sampling Systems

Note that profile systems (or "sampling systems") are slightly different, in that they involve sampling instead of event tracing. Some major ones for Linux are:

• top - provides a dynamic real-time view of a running system, including processes
  • see "man top"

  • see also ksysguard, Gnome system Monitor

• OProfile - system-wide profiler for Linux systems

  • see http://oprofile.sourceforge.net/about/

  • also: http://www-128.ibm.com/developerworks/linux/library/l-oprof.html

• BootChart - samples bootup and provides visualization of process startup and system utilization

  • see BootChart

Related facilities

• in-kernel statistics infrastructure - proposal for a generic implementation of statistics facilities inside the kernel

  • see http://lkml.org/lkml/2006/5/19/106

• perfmon2 - interfaces to hardware performance monitoring features of the CPU

  • see http://perfmon2.sourceforge.net/

• inotify - http://www-128.ibm.com/developerworks/linux/library/l-inotify.html

Other Systems

Here are some systems I haven't classified yet:

• Datastreams - a system for creating and monitoring tracepoints - see http://kusp.ittc.ku.edu/wiki/index.php/Main_Page

Collaboration Efforts

Some trace system project leaders are trying to collaborate: see TracingCollaborationProject