Systemtap: Tapping the Linux kernel for information.

Went to a Singapore Linux Meetup Group meeting on Wednesday night (22 Aug 07) held at Singapore Management University. About 30 people attended. This is an informal group which holds a meetup once every month where members makes a presentation on topics related to Linux.

This month, we had a presentation on SystemTap by Eugene Teo.

Here an attempt at a synopsis on SystemTap:

SystemTap is a software that provides kernel level instrumentation. With it, knowledgeable users can create programmable probes to peer into the inner workings of a live linux kernel. It comes with a C-like scripting language from which scripts are written and compiled into kernel modules. These kernel modules are then automatically loaded into the current running linux kernel. Within these modules, you can put in code to print/echo on everything you want to know about the current state of the kernel.

Here's an example of a Systemtap script which outputs the top 10 I/O intensive processes:



global reads, writes, total_io

probe kernel.function("vfs_read") {
    reads[execname()] += $count
}

probe kernel.function("vfs_write") {
    writes[execname()] += $count
}

# print top 10 IO processes every 5 seconds
probe timer.s(5) {
    foreach (name in writes)
        total_io[name] += writes[name]
    foreach (name in reads)
        total_io[name] += reads[name]
    printf ("%16s\t%10s\t%10s\n", "Process", "KB Read", "KB Written")
    foreach (name in total_io- limit 10)
        printf("%16s\t%10d\t%10d\n", name,
               reads[name]/1024, writes[name]/1024)
    delete reads
    delete writes
    delete total_io
    print("\n")
}

Here's the output:

     Process           KB Read      KB Written
            Xvnc             16831               3
            grep              5754               3
            sort              2046               0
           xterm               718              19
             twm               610              15
       vncserver               153               0
            sshd               128               0
            bash                52               0
             cat                33               0
            yast                29               0

For more examples and information about Systemtap, take a look at its wiki page.

Systemtap is mainly developed by RedHat but is open source and is also available to other linux variants such as SUSE and Dedian. It's been around as a tech preview since Red Hat 4US. (Tech preview means it's still under development and is unsupported). It requires a 2.6 kernel with the kprobes module enabled.

Systemtap is a good tool for instrumenting the Linux kernel but the participants at the meetup pointed the following points:

- Requires a high level of Linux knowledge to use effectively. For example, you have to know something about the actual system calls in order to probe about them. The average user/sysadmin is not likely to be motivated to learn to use it. "It's a typical geek tool".

- Yet another scripting language to learn. Why can't some other popular language, say Python, be used? (my 2 cents)

- Doesn't seem to offer much value since a lot of useful information can be gleaned from other tools. "Any real life examples of Systemtap being used to solve a problem?"

- Comparison were made with Solaris's Dtrace and IBM's PowerTap which were considered to be much more user friendly.

- Appeared to be too much of a Red Hat project. Efforts could be made to involve the wider Linux community.