Caution: This version of this document is no longer maintained. For the latest documentation, see http://www.qnx.com/developers/docs.

The Bones of a Resource Manager

This chapter includes:

Under the covers

Despite the fact that you'll be using a resource manager API that hides many details from you, it's still important to understand what's going on under the covers. For example, your resource manager is a server that contains a MsgReceive() loop, and clients send you messages using MsgSend*(). This means that you must reply either to your clients in a timely fashion, or leave your clients blocked but save the rcvid for use in a later reply.

To help you understand, we'll discuss the events that occur under the covers for both the client and the resource manager.

Under the client's covers

When a client calls a function that requires pathname resolution (e.g. open(), rename(), stat(), or unlink()), the function sends messages to both the process manager and the resource manager to obtain a file descriptor. Once the file descriptor is obtained, the client can use it to send messages to the device associated with the pathname, via the resource manager.

In the following, the file descriptor is obtained, and then the client writes directly to the device:

/*
 * In this stage, the client talks 
 * to the process manager and the resource manager.
 */
fd = open("/dev/ser1", O_RDWR);

/*
 * In this stage, the client talks directly to the
 * resource manager.
 */
for (packet = 0; packet < npackets; packet++)
{
    write(fd, packets[packet], PACKET_SIZE);
}
close(fd);

For the above example, here's the description of what happened behind the scenes. We'll assume that a serial port is managed by a resource manager called devc-ser8250, that's been registered with the pathname prefix /dev/ser1:


Under-the-cover diagram illustrating the client side


Under-the-cover communication between the client, the process manager, and the resource manager.

  1. The client's library sends a “query” message. The open() in the client's library sends a message to the process manager asking it to look up a name (e.g. /dev/ser1).
  2. The process manager indicates who's responsible and it returns the nd, pid, chid, and handle that are associated with the pathname prefix.

    Here's what went on behind the scenes...
    When the devc-ser8250 resource manager registered its name (/dev/ser1) in the namespace, it called the process manager. The process manager is responsible for maintaining information about pathname prefixes. During registration, it adds an entry to its table that looks similar to this:

    0, 47167, 1, 0, 0, /dev/ser1
          

    The table entries represent:

    0
    Node descriptor (nd).
    47167
    Process ID (pid) of the resource manager.
    1
    Channel ID (chid) that the resource manager is using to receive messages with.
    0
    Handle given in case the resource manager has registered more than one name. The handle for the first name is 0, 1 for the next name, etc.
    0
    The open type passed during name registration (0 is _FTYPE_ANY).
    /dev/ser1
    The pathname prefix.

    A resource manager is uniquely identified by a node descriptor, process ID, and a channel ID. The process manager's table entry associates the resource manager with a name, a handle (to distinguish multiple names when a resource manager registers more than one name), and an open type.

    When the client's library issued the query call in step 1, the process manager looked through all of its tables for any registered pathname prefixes that match the name. If another resource manager had previously registered the name /, more than one match would be found. So, in this case, both / and /dev/ser1 match. The process manager will reply to the open() with the list of matched servers or resource managers. The servers are queried in turn about their handling of the path, with the longest match being asked first.

  3. The client's library sends a “connect” message to the resource manager. To do so, it must create a connection to the resource manager's channel:
    fd = ConnectAttach(nd, pid, chid, 0, 0);
          

    The file descriptor that's returned by ConnectAttach() is also a connection ID and is used for sending messages directly to the resource manager. In this case, it's used to send a connect message (_IO_CONNECT defined in <sys/iomsg.h>) containing the handle to the resource manager requesting that it open /dev/ser1.


    Note: Typically, only functions such as open() call ConnectAttach() with an index argument of 0. Most of the time, you should OR _NTO_SIDE_CHANNEL into this argument, so that the connection is made via a side channel, resulting in a connection ID that's greater than any valid file descriptor.

    When the resource manager gets the connect message, it performs validation using the access modes specified in the open() call (e.g. are you trying to write to a read-only device?).

  4. The resource manager generally responds with a pass (and open() returns with the file descriptor) or fail (the next server is queried).
  5. When the file descriptor is obtained, the client can use it to send messages directly to the device associated with the pathname.

    In the sample code, it looks as if the client opens and writes directly to the device. In fact, the write() call sends an _IO_WRITE message to the resource manager requesting that the given data be written, and the resource manager responds that it either wrote some of all of the data, or that the write failed.

Eventually, the client calls close(), which sends an _IO_CLOSE_DUP message to the resource manager. The resource manager handles this by doing some cleanup.

Under the resource manager's covers

The resource manager is a server that uses the Neutrino send/receive/reply messaging protocol to receive and reply to messages. The following is pseudo-code for a resource manager:

initialize the resource manager
register the name with the process manager
DO forever
    receive a message
    SWITCH on the type of message
        CASE _IO_CONNECT:
            call io_open handler
            ENDCASE
        CASE _IO_READ:
            call io_read handler
            ENDCASE
        CASE _IO_WRITE:
            call io_write handler
            ENDCASE
        .   /* etc. handle all other messages */
        .   /* that may occur, performing     */
        .   /* processing as appropriate      */
    ENDSWITCH
ENDDO

Many of the details in the above pseudo-code are hidden from you by a resource manager library that you'll use. For example, you won't actually call a MsgReceive*() function — you'll call a library function, such as resmgr_block() or dispatch_block(), that does it for you. If you're writing a single-threaded resource manager, you might provide a message handling loop, but if you're writing a multithreaded resource manager, the loop is hidden from you.

You don't need to know the format of all the possible messages, and you don't have to handle them all. Instead, you register “handler functions,” and when a message of the appropriate type arrives, the library calls your handler. For example, suppose you want a client to get data from you using read() — you'll write a handler that's called whenever an _IO_READ message is received. Since your handler handles _IO_READ messages, we'll call it an io_read handler.

The resource manager library:

  1. Receives the message.
  2. Examines the message to verify that it's an _IO_READ message.
  3. Calls your io_read handler.

However, it's still your responsibility to reply to the _IO_READ message. You can do that from within your io_read handler, or later on when data arrives (possibly as the result of an interrupt from some data-generating hardware).

The library does default handling for any messages that you don't want to handle. After all, most resource managers don't care about presenting proper POSIX filesystems to the clients. When writing them, you want to concentrate on the code for talking to the device you're controlling. You don't want to spend a lot of time worrying about the code for presenting a proper POSIX filesystem to the client.

Layers in a resource manager

A resource manager is composed of some of the following layers:

Let's look at these from the bottom up.

The iofunc layer

This layer consists of a set of functions that take care of most of the POSIX filesystem details for you — they provide a POSIX personality. If you're writing a device resource manager, you'll want to use this layer so that you don't have to worry too much about the details involved in presenting a POSIX filesystem to the world.

This layer consists of default handlers that the resource manager library uses if you don't provide a handler. For example, if you don't provide an io_open handler, iofunc_open_default() is called.

The iofunc layer also contains helper functions that the default handlers call. If you override the default handlers with your own, you can still call these helper functions. For example, if you provide your own io_read handler, you can call iofunc_read_verify() at the start of it to make sure that the client has access to the resource.

The names of the functions and structures for this layer have the form iofunc_*. The header file is <sys/iofunc.h>. For more information, see the QNX Neutrino Library Reference.

The resmgr layer

This layer manages most of the resource manager library details. It:

If you don't use this layer, then you'll have to parse the messages yourself. Most resource managers use this layer.

The names of the functions and structures for this layer have the form resmgr_*. The header file is <sys/resmgr.h>. For more information, see the QNX Neutrino Library Reference.


Resource manager layer


You can use the resmgr layer to handle _IO_* messages.

The dispatch layer

This layer acts as a single blocking point for a number of different types of things. With this layer, you can handle:

_IO_* messages
It uses the resmgr layer for this.
select()
Processes that do TCP/IP often call select() to block while waiting for packets to arrive, or for there to be room for writing more data. With the dispatch layer, you register a handler function that's called when a packet arrives. The functions for this are the select_*() functions.
Pulses
As with the other layers, you register a handler function that's called when a specific pulse arrives. The functions for this are the pulse_*() functions.
Other messages
You can give the dispatch layer a range of message types that you make up, and a handler. So if a message arrives and the first few bytes of the message contain a type in the given range, the dispatch layer calls your handler. The functions for this are the message_*() functions.

Dispatch layer


You can use the dispatch layer to handle _IO_* messages, select, pulses, and other messages.

The following describes the manner in which messages are handled via the dispatch layer (or more precisely, through dispatch_handler()). Depending on the blocking type, the handler may call the message_*() subsystem. A search is made, based on the message type or pulse code, for a matching function that was attached using message_attach() or pulse_attach(). If a match is found, the attached function is called.

If the message type is in the range handled by the resource manager (I/O messages) and pathnames were attached using resmgr_attach(), the resource manager subsystem is called and handles the resource manager message.

If a pulse is received, it may be dispatched to the resource manager subsystem if it's one of the codes handled by a resource manager (UNBLOCK and DISCONNECT pulses). If a select_attach() is done and the pulse matches the one used by select, then the select subsystem is called and dispatches that event.

If a message is received and no matching handler is found for that message type, MsgError(ENOSYS) is returned to unblock the sender.

The thread pool layer

This layer allows you to have a single- or multithreaded resource manager. This means that one thread can be handling a write() while another thread handles a read().

You provide the blocking function for the threads to use as well as the handler function that's to be called when the blocking function returns. Most often, you give it the dispatch layer's functions. However, you can also give it the resmgr layer's functions or your own.

You can use this layer independently of the resource manager layer.

Simple examples of device resource managers

The following are complete but simple examples of a device resource manager:


Note: As you read through this guide, you'll encounter many code snippets. Most of these code snippets have been written so that they can be combined with either of these simple resource managers.

The first two of these simple device resource managers model their functionality after that provided by /dev/null (although they use /dev/sample to avoid conflict with the “real” /dev/null):

The chapters that follow describe how to add more functionality to these simple resource managers.


Note: The QNX Momentics Integrated Development Environment (IDE) includes a sample /dev/sample resource manager that's very similar to the single-threaded one given below. To get the sample in the IDE, choose Help-->Welcome, and then click the Samples icon.

Single-threaded device resource manager

Here's the complete code for a simple single-threaded device resource manager:

#include <errno.h>
#include <stdio.h>
#include <stddef.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/iofunc.h>
#include <sys/dispatch.h>

static resmgr_connect_funcs_t    connect_funcs;
static resmgr_io_funcs_t         io_funcs;
static iofunc_attr_t             attr;

main(int argc, char **argv)
{
    /* declare variables we'll be using */
    resmgr_attr_t        resmgr_attr;
    dispatch_t           *dpp;
    dispatch_context_t   *ctp;
    int                  id;

    /* initialize dispatch interface */
    if((dpp = dispatch_create()) == NULL) {
        fprintf(stderr,
                "%s: Unable to allocate dispatch handle.\n",
                argv[0]);
        return EXIT_FAILURE;
    }

    /* initialize resource manager attributes */
    memset(&resmgr_attr, 0, sizeof resmgr_attr);
    resmgr_attr.nparts_max = 1;
    resmgr_attr.msg_max_size = 2048;

    /* initialize functions for handling messages */
    iofunc_func_init(_RESMGR_CONNECT_NFUNCS, &connect_funcs, 
                     _RESMGR_IO_NFUNCS, &io_funcs);

    /* initialize attribute structure used by the device */
    iofunc_attr_init(&attr, S_IFNAM | 0666, 0, 0);

    /* attach our device name */
    id = resmgr_attach(
            dpp,            /* dispatch handle        */
            &resmgr_attr,   /* resource manager attrs */
            "/dev/sample",  /* device name            */
            _FTYPE_ANY,     /* open type              */
            0,              /* flags                  */
            &connect_funcs, /* connect routines       */
            &io_funcs,      /* I/O routines           */
            &attr);         /* handle                 */
    if(id == -1) {
        fprintf(stderr, "%s: Unable to attach name.\n", argv[0]);
        return EXIT_FAILURE;
    }

    /* allocate a context structure */
    ctp = dispatch_context_alloc(dpp);

    /* start the resource manager message loop */
    while(1) {
        if((ctp = dispatch_block(ctp)) == NULL) {
            fprintf(stderr, "block error\n");
            return EXIT_FAILURE;
        }
        dispatch_handler(ctp);
    }
}

Note: Include <sys/dispatch.h> after <sys/iofunc.h> to avoid warnings about redefining the members of some functions.

Let's examine the sample code step-by-step.

Here's an outline of the steps we followed:

Initialize the dispatch interface

/* initialize dispatch interface */
if((dpp = dispatch_create()) == NULL) {
    fprintf(stderr, "%s: Unable to allocate dispatch handle.\n",
            argv[0]);
    return EXIT_FAILURE;
}

We need to set up a mechanism so that clients can send messages to the resource manager. This is done via the dispatch_create() function which creates and returns the dispatch structure. This structure contains the channel ID. Note that the channel ID isn't actually created until you attach something, as in resmgr_attach(), message_attach(), and pulse_attach().


Note: The dispatch structure (of type dispatch_t) is opaque; you can't access its contents directly. Use message_connect() to create a connection using this hidden channel ID.

Initialize the resource manager attributes

When you call resmgr_attach(), you pass a resmgr_attr_t control structure to it. Our sample code initializes this structure like this:

/* initialize resource manager attributes */
memset(&resmgr_attr, 0, sizeof resmgr_attr);
resmgr_attr.nparts_max = 1;
resmgr_attr.msg_max_size = 2048;

In this case, we're configuring:

For more information, see resmgr_attach() in the QNX Neutrino Library Reference.

Initialize functions used to handle messages

/* initialize functions for handling messages */
iofunc_func_init(_RESMGR_CONNECT_NFUNCS, &connect_funcs, 
                 _RESMGR_IO_NFUNCS, &io_funcs);

Here we supply two tables that specify which function to call when a particular message arrives:

Instead of filling in these tables manually, we call iofunc_func_init() to place the iofunc_*_default() handler functions into the appropriate spots.

Initialize the attribute structure used by the device

/* initialize attribute structure used by the device */
iofunc_attr_init(&attr, S_IFNAM | 0666, 0, 0);

The attribute structure contains information about our particular device associated with the name /dev/sample. It contains at least the following information:

Effectively, this is a per-name data structure. In the Extending the POSIX-Layer Data Structures chapter, we'll see how you can extend the structure to include your own per-device information.

Put a name into the namespace

To register our resource manager's path, we call resmgr_attach() like this:

/* attach our device name */
id = resmgr_attach(dpp,            /* dispatch handle        */
                   &resmgr_attr,   /* resource manager attrs */
                   "/dev/sample",  /* device name            */
                   _FTYPE_ANY,     /* open type              */
                   0,              /* flags                  */
                   &connect_funcs, /* connect routines       */
                   &io_funcs,      /* I/O routines           */
                   &attr);         /* handle                 */
if(id == -1) {
    fprintf(stderr, "%s: Unable to attach name.\n", argv[0]);
    return EXIT_FAILURE;
}

Before a resource manager can receive messages from other programs, it needs to inform the other programs (via the process manager) that it's the one responsible for a particular pathname prefix. This is done via pathname registration. When the name is registered, other processes can find and connect to this process using the registered name.

In this example, a serial port may be managed by a resource manager called devc-xxx, but the actual resource is registered as /dev/sample in the pathname space. Therefore, when a program requests serial port services, it opens the /dev/sample serial port.

We'll look at the parameters in turn, skipping the ones we've already discussed.

device name
Name associated with our device (i.e. /dev/sample).
open type
Specifies the constant value of _FTYPE_ANY. This tells the process manager that our resource manager will accept any type of open request — we're not limiting the kinds of connections we're going to be handling.

Some resource managers legitimately limit the types of open requests they handle. For instance, the POSIX message queue resource manager accepts only open messages of type _FTYPE_MQUEUE.

flags
Controls the process manager's pathname resolution behavior. By specifying a value of zero, we indicate that we'll accept only requests for the name /dev/sample.

Note: The bits that you use in this argument are the _RESMGR_FLAG_* flags (e.g. _RESMGR_FLAG_BEFORE) defined in <sys/resmgr.h>. We'll discuss some of these flags in this guide, but you can find a full list in the entry for resmgr_attach() in the QNX Neutrino Library Reference.

There are some other flags whose names don't start with an underscore, but they're for the flags member of the resmgr_attr_t structure, which we'll look at in more detail in Setting resource manager attributes in the Fleshing Out the Skeleton chapter.


Allocate the context structure

/* allocate a context structure */
ctp = dispatch_context_alloc(dpp);

The context structure contains a buffer where messages will be received. The size of the buffer was set when we initialized the resource manager attribute structure. The context structure also contains a buffer of IOVs that the library can use for replying to messages. The number of IOVs was set when we initialized the resource manager attribute structure.

For more information, see dispatch_context_alloc() in the QNX Neutrino Library Reference.

Start the resource manager message loop

/* start the resource manager message loop */
while(1) {
    if((ctp = dispatch_block(ctp)) == NULL) {
        fprintf(stderr, "block error\n");
        return EXIT_FAILURE;
    }
    dispatch_handler(ctp);
}

Once the resource manager establishes its name, it receives messages when any client program tries to perform an operation (e.g. open(), read(), write()) on that name.

In our example, once /dev/sample is registered, and a client program executes:

fd = open ("/dev/sample", O_RDONLY);

the client's C library constructs an _IO_CONNECT message and sends it to our resource manager. Our resource manager receives the message within the dispatch_block() function. We then call dispatch_handler(), which decodes the message and calls the appropriate handler function based on the connect and I/O function tables that we passed in previously. After dispatch_handler() returns, we go back to the dispatch_block() function to wait for another message.


Note: Note that dispatch_block() returns a pointer to a dispatch context (dispatch_context_t) structure — the same type of pointer you pass to the routine:
  • If dispatch_block() returns a non-NULL context pointer, it could be different from the one passed in, as it's possible for the ctp to be reallocated to a larger size. In this case, the old ctp is no longer valid.
  • If dispatch_block() returns NULL (for example, because a signal interrupted the MsgReceive()), the old context pointer is still valid. Typically, a resource manager targets any signals to a thread that's dedicated to handling signals. However, if a signal can be targeted to the thread doing dispatch_block(), you could use the following code:
    dispatch_context_t   *ctp, *new_ctp;
    
    ctp = dispatch_context_alloc( … );
    while (1) {
       new_ctp = dispatch_block( ctp );
       if ( new_ctp ) {
          ctp = new_ctp
          }
       else {
          /* handle the error condition */
          …
          }
    }
      

At some later time, when the client program executes:

read (fd, buf, BUFSIZ);

the client's C library constructs an _IO_READ message, which is then sent directly to our resource manager, and the decoding cycle repeats.

Multithreaded device resource manager

Here's the complete code for a simple multithreaded device resource manager:

#include <errno.h>
#include <stdio.h>
#include <stddef.h>
#include <stdlib.h>
#include <unistd.h>

/*
 * Define THREAD_POOL_PARAM_T such that we can avoid a compiler
 * warning when we use the dispatch_*() functions below
 */
#define THREAD_POOL_PARAM_T dispatch_context_t

#include <sys/iofunc.h>
#include <sys/dispatch.h>

static resmgr_connect_funcs_t    connect_funcs;
static resmgr_io_funcs_t         io_funcs;
static iofunc_attr_t             attr;

main(int argc, char **argv)
{
    /* declare variables we'll be using */
    thread_pool_attr_t   pool_attr;
    resmgr_attr_t        resmgr_attr;
    dispatch_t           *dpp;
    thread_pool_t        *tpp;
    dispatch_context_t   *ctp;
    int                  id;

    /* initialize dispatch interface */
    if((dpp = dispatch_create()) == NULL) {
        fprintf(stderr, "%s: Unable to allocate dispatch handle.\n",
                argv[0]);
        return EXIT_FAILURE;
    }

    /* initialize resource manager attributes */
    memset(&resmgr_attr, 0, sizeof resmgr_attr);
    resmgr_attr.nparts_max = 1;
    resmgr_attr.msg_max_size = 2048;

    /* initialize functions for handling messages */
    iofunc_func_init(_RESMGR_CONNECT_NFUNCS, &connect_funcs, 
                     _RESMGR_IO_NFUNCS, &io_funcs);

    /* initialize attribute structure used by the device */
    iofunc_attr_init(&attr, S_IFNAM | 0666, 0, 0);

    /* attach our device name */
    id = resmgr_attach(dpp,            /* dispatch handle        */
                       &resmgr_attr,   /* resource manager attrs */
                       "/dev/sample",  /* device name            */
                       _FTYPE_ANY,     /* open type              */
                       0,              /* flags                  */
                       &connect_funcs, /* connect routines       */
                       &io_funcs,      /* I/O routines           */
                       &attr);         /* handle                 */
    if(id == -1) {
        fprintf(stderr, "%s: Unable to attach name.\n", argv[0]);
        return EXIT_FAILURE;
    }

    /* initialize thread pool attributes */
    memset(&pool_attr, 0, sizeof pool_attr);
    pool_attr.handle = dpp;
    pool_attr.context_alloc = dispatch_context_alloc;
    pool_attr.block_func = dispatch_block; 
    pool_attr.unblock_func = dispatch_unblock;
    pool_attr.handler_func = dispatch_handler;
    pool_attr.context_free = dispatch_context_free;
    pool_attr.lo_water = 2;
    pool_attr.hi_water = 4;
    pool_attr.increment = 1;
    pool_attr.maximum = 50;

    /* allocate a thread pool handle */
    if((tpp = thread_pool_create(&pool_attr, 
                                 POOL_FLAG_EXIT_SELF)) == NULL) {
        fprintf(stderr, "%s: Unable to initialize thread pool.\n",
                argv[0]);
        return EXIT_FAILURE;
    }

    /* start the threads; will not return */
    thread_pool_start(tpp);
}

Most of the code is the same as in the single-threaded example, so we'll cover only those parts that aren't described above. Also, we'll go into more detail on multithreaded resource managers later in this guide, so we'll keep the details here to a minimum.

Here's an outline of the steps we'll cover:

For this code sample, the threads are using the dispatch_*() functions (i.e. the dispatch layer) for their blocking loops.

Define THREAD_POOL_PARAM_T

/*
 * Define THREAD_POOL_PARAM_T such that we can avoid a compiler
 * warning when we use the dispatch_*() functions below
 */
#define THREAD_POOL_PARAM_T dispatch_context_t

#include <sys/iofunc.h>
#include <sys/dispatch.h>

The THREAD_POOL_PARAM_T manifest tells the compiler what type of parameter is passed between the various blocking/handling functions that the threads will be using. This parameter should be the context structure used for passing context information between the functions. By default it's defined as a resmgr_context_t, but since this sample is using the dispatch layer, we need it to be a dispatch_context_t. We define it prior to the include directives above, since the header files refer to it.

Initialize thread pool attributes

/* initialize thread pool attributes */
memset(&pool_attr, 0, sizeof pool_attr);
pool_attr.handle = dpp;
pool_attr.context_alloc = dispatch_context_alloc;
pool_attr.block_func = dispatch_block;
pool_attr.unblock_func = dispatch_unblock;
pool_attr.handler_func = dispatch_handler;
pool_attr.context_free = dispatch_context_free;
pool_attr.lo_water = 2;
pool_attr.hi_water = 4;
pool_attr.increment = 1;
pool_attr.maximum = 50;

The thread pool attributes tell the threads which functions to use for their blocking loop and control how many threads should be in existence at any time. We'll go into more detail on these attributes when we talk about multithreaded resource managers in more detail later in this guide.

Allocate a thread pool handle

/* allocate a thread pool handle */
if((tpp = thread_pool_create(&pool_attr, 
                             POOL_FLAG_EXIT_SELF)) == NULL) {
    fprintf(stderr, "%s: Unable to initialize thread pool.\n",
            argv[0]);
    return EXIT_FAILURE;
}

The thread pool handle is used to control the thread pool. Among other things, it contains the given attributes and flags. The thread_pool_create() function allocates and fills in this handle.

Start the threads

/* start the threads; will not return */
thread_pool_start(tpp);

The thread_pool_start() function starts up the thread pool. Each newly created thread allocates a context structure of the type defined by THREAD_POOL_PARAM_T using the context_alloc function we gave above in the attribute structure. They'll then block on the block_func and when the block_func returns, they'll call the handler_func, both of which were also given through the attributes structure. Each thread essentially does the same thing that the single-threaded resource manager above does for its message loop.

From this point on, your resource manager is ready to handle messages. Since we gave the POOL_FLAG_EXIT_SELF flag to thread_pool_create(), once the threads have been started up, pthread_exit() will be called and this calling thread will exit.

Using MsgSend() and MsgReply()

You don't have to use read() and write() to interact with a resource manager; you can use the path that a resource manager registers to get a connection ID (coid) that you can use with MsgSend() to send messages to the server.

This example consists of simple client and server programs that you can use as the starting point for any similar project. There are two source files: one for the server, and one for the client. Note that you must run server as root — a requirement in order to use the resmgr_attach() function.

A bit of history

QNX Neutrino and our earlier QNX 4 RTOS both use a notion of Send/Receive/Reply for messaging. This IPC mechanism is (generally) used in a synchronous manner; the sending process waits for a reply from the receiver, and a receiver waits for a message to be sent. This provides a very easy call-response synchronization.

Under QNX 4, the Send() function needed only the process ID (pid) of the receiving process. QNX 4 also provided a very simple API for giving a process a name and, in turn, looking up that name to get a process ID. So you could name your server process, and then your client process could look up that name, get a process ID (pid), and then send the server some data and wait for a reply. This model worked well in a non-threaded environment.

Since QNX Neutrino includes proper thread support, the notion of having a single conduit into a process doesn't make a lot of sense, so a more flexible system was designed. To perform a MsgSend() under QNX Neutrino, you no longer need a pid, but rather a connection ID (coid).

This coid is obtained from opening a connection to a channel. Processes can create multiple channels and can have different threads service any (or all) of them. The issue now becomes: how does a client get a coid in the first place so it can open a connection to get the coid it needs to perform the MsgSend()?

There are many different ways this kind of information-sharing can occur, but the method that falls in line with the QNX Neutrino design ideals is for the server to also be a resource manager.

Under QNX Neutrino — and other POSIX systems — when you call open(), you get back a file descriptor (fd). But this fd is also a coid. So instead of registering a name, as in QNX 4, your server process registers a path in the filesystem, and the client opens that path to get the coid to talk to the server.

The server

Let's begin with the server:

/*
 * ResMgr and Message Server Process
 */

#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <errno.h>
#include <string.h>
#include <sys/neutrino.h>
#include <sys/iofunc.h>
#include <sys/dispatch.h>

resmgr_connect_funcs_t  ConnectFuncs;
resmgr_io_funcs_t       IoFuncs;
iofunc_attr_t           IoFuncAttr;


typedef struct
{
    uint16_t msg_no;
    char     msg_data[255];
} server_msg_t;


int message_callback( message_context_t *ctp, int type, unsigned flags, 
                      void *handle )
{
    server_msg_t *msg;
    int num;
    char msg_reply[255];

    /* Cast a pointer to the message data */
    msg = (server_msg_t *)ctp->msg;

    /* Print some useful information about the message */
    printf( "\n\nServer Got Message:\n" );
    printf( "  type: %d\n" , type );
    printf( "  data: %s\n\n", msg->msg_data );

    /* Build the reply message */
    num = type - _IO_MAX;
    snprintf( msg_reply, 254, "Server Got Message Code: _IO_MAX + %d", num );
   
    /* Send a reply to the waiting (blocked) client */ 
    MsgReply( ctp->rcvid, EOK, msg_reply, strlen( msg_reply ) + 1 );

    return 0;
}



int main( int argc, char **argv )
{
    resmgr_attr_t        resmgr_attr;
    message_attr_t       message_attr;
    dispatch_t           *dpp;
    dispatch_context_t   *ctp, *ctp_ret;
    int                  resmgr_id, message_id;

    /* Create the dispatch interface */
    dpp = dispatch_create();
    if( dpp == NULL )
    {
        fprintf( stderr, "dispatch_create() failed: %s\n", 
                 strerror( errno ) );
        return EXIT_FAILURE;
    }

    memset( &resmgr_attr, 0, sizeof( resmgr_attr ) );
    resmgr_attr.nparts_max = 1;
    resmgr_attr.msg_max_size = 2048;

    /* Setup the default I/O functions to handle open/read/write/... */
    iofunc_func_init( _RESMGR_CONNECT_NFUNCS, &ConnectFuncs,
                      _RESMGR_IO_NFUNCS, &IoFuncs );

    /* Setup the attribute for the entry in the filesystem */
    iofunc_attr_init( &IoFuncAttr, S_IFNAM | 0666, 0, 0 );

    resmgr_id = resmgr_attach( dpp, &resmgr_attr, "serv", _FTYPE_ANY, 
                               0, &ConnectFuncs, &IoFuncs, &IoFuncAttr );
    if( resmgr_id == -1 )
    {
        fprintf( stderr, "resmgr_attach() failed: %s\n", strerror( errno ) );
        return EXIT_FAILURE;
    }

    /* Setup our message callback */
    memset( &message_attr, 0, sizeof( message_attr ) );
    message_attr.nparts_max = 1;
    message_attr.msg_max_size = 4096;

    /* Attach a callback (handler) for two message types */
    message_id = message_attach( dpp, &message_attr, _IO_MAX + 1,
                                 _IO_MAX + 2, message_callback, NULL );
    if( message_id == -1 )
    {
        fprintf( stderr, "message_attach() failed: %s\n", strerror( errno ) );
        return EXIT_FAILURE;
    }

    /* Setup a context for the dispatch layer to use */
    ctp = dispatch_context_alloc( dpp );
    if( ctp == NULL )
    {
        fprintf( stderr, "dispatch_context_alloc() failed: %s\n", 
                 strerror( errno ) );
        return EXIT_FAILURE;
    }


    /* The "Data Pump" - get and process messages */
    while( 1 )
    {
        ctp_ret = dispatch_block( ctp );
        if( ctp_ret )
        {
            dispatch_handler( ctp );
        }
        else
        {
            fprintf( stderr, "dispatch_block() failed: %s\n", 
                     strerror( errno ) );
            return EXIT_FAILURE;
        }
    }

    return EXIT_SUCCESS;
}

The first thing the server does is create a dispatch handle (dpp) using dispatch_create(). This handle will be used later when making other calls into the dispatch portion of the library. The dispatch layer takes care of receiving incoming messages and routing them to the appropriate layer (resmgr, message, pulse).

After the dispatch handle is created, the server sets up the variables needed to make a call into resmgr_attach(). But since we're not using the resmgr functionality for anything more than getting a connection ID to use with MsgSend(), the server sets up everything to the defaults.

We don't need (or want) to worry about I/O and connection messages right now (like the messages that open(), close(), read(), write() and so on generate); we just want them to work and do the right thing. Luckily, there are defaults built into the C library to handle these types of messages for you, and iofunc_func_init() sets up these defaults. The call to iofunc_attr_init() sets up the attribute structure so that the entry in the filesystem has the specified attributes.

Finally, the call to resmgr_attach() is made. For our purposes, the most important parameter is the third. In this case we're registering the filesystem entry serv. Since an absolute path wasn't given, the entry will appear in the same directory where the server was run. All of this gives us a filesystem entry that can be opened and closed, but generally behaves the same as /dev/null. But that's fine, since we want to be able to MsgSend() data to our server, not write() data to it.

Now that the resmgr portion of the setup is complete, we need to tell the dispatch layer that we'll be handling our own messages in addition to the standard I/O and connection messages handled by the resmgr layer. In order to let the dispatch layer know the general attributes of the messages we'll be receiving, we fill in the message_attr structure. In this case we're telling it that the number of message parts we're going to receive is 1 with a maximum message size of 4096 bytes.

Once we have these attributes defined, we can register our intent to handle messages with the dispatch layer by invoking message_attach(). With this call, we're setting up our message_callback() routine to be the handler of messages of type _IO_MAX + 1 up to and including messages of type _IO_MAX + 2. There's even the option of having a pointer to arbitrary data passed into the callback, but we don't need that so we're setting it to NULL.

You might now be asking, “Message type _IO_MAX + 1! I don't see anything in the MsgSend() docs for setting a message type!” This is true. However, in order to play nicely with the dispatch layer, all incoming messages must have a 32-bit integer at the start indicating the message type. Although this may seem restrictive to a new QNX developer, the reason it's in place is that most designs will end up using some sort of message identification anyway, and this just forces you into a particular style. This will become clearer when we look at the client. But now let's finish the server.

Now that we've registered both the resmgr and message handlers with the dispatch layer, we simply create a context for the dispatch layer to use while processing messages by calling dispatch_context_alloc(), and then start receiving and processing data. This is a two-step process:

  1. The server calls dispatch_block(), which waits for incoming messages and pulses.
  2. Once there's data available, we call into dispatch_handler() to do the right thing based on the message data. It's inside the dispatch_handler() call that our message_callback() routing will be invoked, when messages of the proper type are received.

Finally, let's look at what our message_callback() routine actually does when a proper message is received. When a message of type _IO_MAX + 1 or _IO_MAX + 2 is received, our callback is invoked. We get the message type passed in via the type parameter. The actual message data can be found in ctp->msg. When the message comes in, the server prints the message type and the string that was sent from the client. It then prints the offset from _IO_MAX of the message type, and then finally formats a reply string and sends the reply back to the client via ctp->rcvid using MsgReply().

The client

The client is much simpler:

/* 
 * Message Client Process 
 */ 

#include <stdio.h> 
#include <unistd.h> 
#include <stdlib.h> 
#include <fcntl.h> 
#include <errno.h> 
#include <string.h> 
#include <sys/neutrino.h> 
#include <sys/iofunc.h> 
#include <sys/dispatch.h> 

typedef struct 
{ 
    uint16_t msg_no; 
    char msg_data[255]; 
} client_msg_t; 

int main( int argc, char **argv ) 
{ 
    int fd; 
    int c; 
    client_msg_t msg; 
    int ret; 
    int num; 
    char msg_reply[255]; 
    
    num = 1; 
    
    /* Process any command line arguments */ 
    while( ( c = getopt( argc, argv, "n:" ) ) != -1 ) 
    { 
        if( c == 'n' ) 
        { 
            num = strtol( optarg, 0, 0 ); 
        } 
    } 
    /* Open a connection to the server (fd == coid) */ 
    fd = open( "serv", O_RDWR ); 
    if( fd == -1 ) 
    { 
        fprintf( stderr, "Unable to open server connection: %s\n", 
            strerror( errno ) ); 
        return EXIT_FAILURE; 
    } 
    
    /* Clear the memory for the msg and the reply */ 
    memset( &msg, 0, sizeof( msg ) ); 
    memset( &msg_reply, 0, sizeof( msg_reply ) ); 
    
    /* Set up the message data to send to the server */ 
    msg.msg_no = _IO_MAX + num; 
    snprintf( msg.msg_data, 254, "client %d requesting reply.", getpid() ); 
    
    printf( "client: msg_no: _IO_MAX + %d\n", num ); 
    fflush( stdout ); 
    
    /* Send the data to the server and get a reply */ 
    ret = MsgSend( fd, &msg, sizeof( msg ), msg_reply, 255 ); 
    if( ret == -1 ) 
    { 
        fprintf( stderr, "Unable to MsgSend() to server: %s\n", strerror( errno ) ); 
        return EXIT_FAILURE; 
    } 
    
    /* Print out the reply data */ 
    printf( "client: server replied: %s\n", msg_reply ); 
    
    close( fd ); 
    
    return EXIT_SUCCESS; 
} 

Note: Remember that since the server registers a relative pathname, the client must be run from the same directory as the server.

The client uses the open() function to get a coid (the server's default resmgr setup takes care of all of this on the server side), and performs a MsgSend() to the server based on this coid, and then waits for the reply. When the reply comes back, the client prints the reply data.

You can give the client the command-line option -n# (where # is the offset from _IO_MAX) to use for the message. If you give anything over 2 as the offset, the MsgSend() will fail, since the server hasn't set up handlers for those messages.

This example is very basic, but it still covers a lot of ground. There are many other things you can do using this same basic framework:

Many of these topics are covered later in this guide.