Capturing Trace Data

This chapter includes:


The program that captures data is the “messenger” between the instrumented kernel and the filesystem.

Data capture

Possible data capture configurations.

The main function of the data-capture program is to send the buffers given to it by the instrumented kernel to an output device (which may be a file or something else). In order to accomplish this function, the program must also:

You must configure the instrumented kernel before logging. The instrumented kernel configuration settings include:

Note: The instrumented kernel retains the settings, and multiple programs access a single instrumented kernel configuration. Changing the settings in one process supersedes the settings made in another.

We've provided tracelogger as the default data-capture utility. Although you can write your own utility, there's little need to.

You can control the capture of data via qconn (under the control of the IDE), tracelogger (from the command line), or directly from your application. All three approaches use the TraceEvent() function to control the instrumented kernel:

Trace control

Controlling the capture of trace data.

For information about controlling the trace from the IDE, see the Analyzing Your System with Kernel Tracing chapter of the IDE User's Guide.

Let's look first at using tracelogger, and then we'll describe how you can use TraceEvent() to control tracing from your application.

  • Don't run more than one instance of tracelogger at a time. Similarly, don't run tracelogger and trace events under control of the IDE at the same time.
  • On a multicore system, if the clocks on all processors aren't synchronized, then tracelogger will produce data with inconsistent timestamps, and the IDE won't be able to load the trace file. The IDE attempts to properly order the events in the trace file, and this can go awry if the timestamp data is incorrect.

    The traceprinter utility doesn't have any issues with such traces because it doesn't attempt to reorder the data and interpret it; it simply dumps the contents of each event.

Using tracelogger to control tracing

The options that you use when you start tracelogger affect the way that the instrumented kernel logs events and how tracelogger captures them. In this section, we'll look at:

Managing trace buffers

You can use tracelogger's command-line options to manage the instrumented kernel's buffers, specifying:

You can also specify the number of buffers that tracelogger itself uses.

For more information, see the entry for tracelogger in the Neutrino Utilities Reference.

tracelogger's modes of operation

You can run tracelogger in several modes — depending on how and what you want to trace — by specifying the following command-line options:

Mode Option The kernel: tracelogger:
Continuous -c Logs events Captures the events, and continues to do so until terminated
Daemon -d1 Doesn't log eventsa Waits passivelya
Iterations (the default) -n Logs events Captures num_buffers of data and then terminates
Ring -r Logs events Doesn't capture events until it gets a SIGINT signal, or an application calls TraceEvent(_NTO_TRACE_STOP)b
Time-based -s Logs events Captures events for the specified number of seconds

a In daemon mode, logging starts when an application calls TraceEvent(_NTO_TRACE_START) and continues until an application calls TraceEvent(_NTO_TRACE_STOP), or until you terminate tracelogger.

b When you terminate tracing in ring mode, tracelogger stops logging events, and then briefly restarts and stops it again so it can capture the state information that's emitted by the _NTO_TRACE_START command. This information includes the thread IDs and names of processes.

In the non-daemon modes, you configure, start, and stop the tracing from the command line. In daemon mode, your application must do everything from code, but if you also specify the -E option, tracelogger enables all events in all classes, and you can use the -F option to set up filtering.

Here's an outline of the strengths, weaknesses, and features of these modes:

Feature Non-daemon mode Daemon mode Daemon mode with -E
tracelogger support Full Limited Full
Controllability Limited Full Full
Events recorded by default All None All
Configuration difficulty Easy Harder Easy
Configuration method Command line only User program, using calls to TraceEvent() Command line and user program
Logging starts Instantaneously User program; using calls to TraceEvent() User program; using calls to TraceEvent()

Note: If an application has called TraceEvent(_NTO_TRACE_START), and you then try to start tracelogger, tracelogger might fail with a “resource busy” message. To help avoid this:
  • Start tracelogger before your application issues a _NTO_TRACE_START or _NTO_TRACE_STARTNOSTATE command.
  • Don't leave tracing on indefinitely; be sure to issue a _NTO_TRACE_STOP after each _NTO_TRACE_START or _NTO_TRACE_STARTNOSTATE command.

For a full description of the tracelogger utility and its options, see its entry in the Neutrino Utilities Reference.

Choosing between wide and fast modes

By default, the instrumented kernel and tracelogger collect data in fast mode; to switch to wide mode, specify the -w option when you start tracelogger.

Filtering events

The tracelogger utility gives you some basic control over filtering by way of its -F option. This filtering is limited to excluding entire classes of events at a time; if you need a finer granularity, you'll need to use TraceEvent(), as described in the Filtering chapter in this guide.

By default, tracelogger captures all events from all classes, but you can disable the tracing of events from the classes as follows:

To disable this class: Specify:
Kernel calls -F1
Interrupt -F2
Process -F3
Thread -F4
Virtual thread -F5
Communication -F6
System -F7

You can specify more than one filter by using multiple -F options. Note that you can't disable the Control or User classes with this option. For more information about classes, see the Events and the Kernel chapter of this guide.

Specifying where to send the output

Because the ring of buffers can't hope to store a complete log of event activity for any significant amount of time, the tracebuffer must be handed off to a data-capture program. Normally the data-capture program pipes the information to either an output device or a file.

By default, the tracelogger utility saves the output in the binary file /dev/shmem/tracebuffer.kev, but you can use the -f option to specify a different path. The .kev extension is short for “kernel events”; you can use a different extension, but the IDE recognizes .kev and automatically uses the System Profiler to open such files.

You can also map the file in shared memory (-M), but you must then also specify the maximum size for the file (-S).

Using TraceEvent() to control tracing

You don't have to use tracelogger to control all aspects of tracing; you can call TraceEvent() directly — which (after all) is what tracelogger does. Using TraceEvent() to control tracing means a bit more work for you, but you have much more control over specific details.

You could decide not to use tracelogger at all, and use TraceEvent() exclusively, but you'd then have to manage the buffers, collect the trace data, and save it in the appropriate form, which could be a significant amount of work.

In practical terms you'll likely use tracelogger and TraceEvent() together. For example, you might run tracelogger in daemon mode, to take advantage of its management of the trace data, but call TraceEvent() to control exactly which events to trace.

The TraceEvent() kernel call takes a variable number of arguments. The first is always a command and determines what (if any) additional arguments are required.

In this section, we'll discuss:

For reference information about TraceEvent(), see the QNX Neutrino Library Reference.

Managing trace buffers

As mentioned above, you can use TraceEvent() to manage the instrumented kernel's buffers, but it's probably easier to run tracelogger in daemon mode and let it look after the buffers. Nevertheless, here's a summary of how to do it with TraceEvent():

For examples of some of these commands, see Data-capture program in the Sample Programs appendix.

Modes of operation

TraceEvent() doesn't support the different modes of operation that tracelogger does; your application has to indicate when to start tracing, how long to trace for, and so on:

Note: If an application has called TraceEvent(_NTO_TRACE_START), and you then try to start tracelogger, tracelogger might fail with a “resource busy” message. To help avoid this:
  • Start tracelogger before your application issues a _NTO_TRACE_START or _NTO_TRACE_STARTNOSTATE command.
  • Don't leave tracing on indefinitely; be sure to issue a _NTO_TRACE_STOP after each _NTO_TRACE_START or _NTO_TRACE_STARTNOSTATE command.

Filtering events

You can select events in an additive or subtractive manner; you can start with no events, and then add specific classes or events, or you can start with all events, and then exclude specific ones. We'll discuss using TraceEvent() to filter events in the Filtering chapter.

Choosing between wide and fast modes

TraceEvent() gives you much finer control over wide and fast mode than you can get with tracelogger, which can simply set the mode for all events in all traced classes. Using TraceEvent(), you can set fast and wide mode for all classes, a specific class, or a specific event in a class:

Inserting trace events

You can even use TraceEvent() to insert your own events into the trace data. You can call TraceEvent() directly (see below), but it's much easier to use the following convenience functions:

Insert a trace event for the entry to a function
Insert a trace event for the exit from a function
Insert a trace event for the current address
Insert a user combine trace event
Insert a trace event of an arbitrary class and type with arbitrary data
Insert a user string trace event
Insert a user simple trace event
Insert a user string trace event, specifying a maximum string length
Insert a user string trace event, using a variable argument list

If you want to call TraceEvent() directly, use one of the following commands:

For more information, see the entry for TraceEvent() in the QNX Neutrino Library Reference.