Components of the SAT

The QNX Neutrino System Analysis Toolkit (SAT) consists of the following main components:

You can also trace and analyze events under control of the Integrated Development Environment.

Figure 1. Overall view of the SAT.

Instrumented kernel

The instrumented kernel is actually the regular QNX Neutrino microkernel with a small, highly efficient event-gathering module included. Except for the instrumentation, its operation is virtually indistinguishable—the instrumented kernel runs at 98% of the speed of our regular microkernel.

As threads run, the instrumented kernel continuously intercepts information about what the kernel is doing, generating time-stamped and CPU-stamped events that are stored in a ring of buffers. Because the tracing occurs at the kernel level, the SAT can track the performance of all processes, including the data-capturing program.

To check to see if your system is running the instrumented kernel, type:

ls /proc/boot

and then look for a file whose name includes procnto. If the file name is procnto-*instr, you're running the instrumented kernel; if the file name doesn't include instr, you're running the noninstrumented kernel.

To switch to the instrumented kernel, do the following:

  1. In your buildfile, replace the entry for procnto with the appropriate procnto-*instr. For more information, see the entry for procnto in the Utilities Reference.
  2. Add tracelogger and traceprinter to your buildfile or target.
  3. Run the mkifs utility to rebuild the image. For more information, see the entry for mkifs in the Utilities Reference.
  4. Replace your current boot image with the new one.

Kernel buffer management

The kernel buffer is composed of many small trace buffers. Although the number of buffers is limited only by the amount of system memory, it's important to understand that this space must be managed carefully. If all of the events are being traced on an active system, the number of events can be quite large.

To allow the instrumented kernel to write to one part of the kernel buffer and store another part of it simultaneously, the trace buffers are organized as a ring. As the buffer data reaches a high-water mark (about 70% full in linear mode, or 90% in ring mode), the instrumented kernel module raises an _NTO_HOOK_TRACE synthetic interrupt to notify the data-capture program, passing the index of the buffer. The data-capture program can then retrieve the buffer and save it in a storage location for offline processing or pass it to a data interpreter for realtime manipulation. In either case, once the buffer has been emptied, it's once again available for use by the kernel.

Data-capture program (tracelogger)

The QNX Neutrino RTOS includes a tracelogger that you can use to capture data. This service receives events from the instrumented kernel and saves them in a file or sends them to a device for later analysis.

Note: The data-capture utilities require the PROCMGR_AID_TRACE ability enabled in order to allocate buffer memory, and the PROCMGR_AID_IO and I/O privileges to use functions such as InterruptHookTrace(). For more information about abilities, see the entry procmgr_ability() in the QNX Neutrino C Library Reference.

Because the tracelogger may write data at rates well in excess of 20 MB/minute, running it for prolonged periods or running it repeatedly can use up a large amount of space. If disk space is low, erase old log files regularly. (In its default mode, tracelogger overwrites its previous default file.)

You can also control tracing from your application (e.g., to turn tracing on just for a problematic area) with the TraceEvent() kernel call. This function has over 30 different commands that let you:

For more information, see the Capturing Trace Data chapter in this guide, the entry for tracelogger in the Utilities Reference, and the entry for TraceEvent() in the QNX Neutrino C Library Reference.

Data interpretation (e.g., traceprinter)

To aid in processing the binary trace event data, we provide the libtraceparser library. The API functions let you set up a series of functions that are called when complete buffer slots of event data have been received/read from the raw binary event stream.

We also provide a linear trace event printer (traceprinter) that outputs all of the trace events ordered linearly by their timestamp as they're emitted by the kernel. This utility uses the libtraceparser library. You can also use the API to create an interface to do the following offline or in real time:

The traceparser library provides an API for parsing and interpreting the trace events that are stored in the event file. The library simplifies the parsing and interpretation process by letting you easily:

For more information, see the Interpreting Trace Data chapter in this guide, as well as the entry for traceprinter in the Utilities Reference.

Integrated Development Environment

The QNX Momentics Tool Suite's IDE provides a graphical interface that you can use to capture and examine tracing events. The IDE lets you filter events, zoom in on ranges of them, examine the associated data, save subsets of events, and more.

For more information, see the Analyzing Your System with Kernel Tracing chapter of the IDE User's Guide.