Fleshing Out the Skeleton

It's time now to start adding some flesh to the basic bones of the resource manager.

This chapter includes:

Message types

As we saw in the Bones of a Resource Manager chapter, your resource manager may need to handle these types of messages:

We'll examine them in the sections and chapters that follow.


Note: The Getting Started with QNX Neutrino guide includes a summary of the handlers for these messages; see Alphabetical listing of connect and I/O functions in its Resource Managers chapter.

Connect messages

A connect message is issued by the client to perform an operation based on a pathname. This may be a message that establishes a longer term relationship between the client and the resource manager (e.g. open()), or it may be a message that is a “one-shot” event (e.g. rename()).

When you call resmgr_attach(), you pass it a pointer to a resmgr_connect_funcs_t structure that defines your connect functions. This structure is defined in <sys/resmgr.h> as follows:

typedef struct _resmgr_connect_funcs {

    unsigned nfuncs;

    int (*open)      (resmgr_context_t *ctp, io_open_t *msg,
                      RESMGR_HANDLE_T *handle, void *extra);

    int (*unlink)    (resmgr_context_t *ctp, io_unlink_t *msg,
                      RESMGR_HANDLE_T *handle, void *reserved);

    int (*rename)    (resmgr_context_t *ctp, io_rename_t *msg,
                      RESMGR_HANDLE_T *handle,
                      io_rename_extra_t *extra);

    int (*mknod)     (resmgr_context_t *ctp, io_mknod_t *msg,
                      RESMGR_HANDLE_T *handle, void *reserved);

    int (*readlink)  (resmgr_context_t *ctp, io_readlink_t *msg,
                      RESMGR_HANDLE_T *handle, void *reserved);

    int (*link)      (resmgr_context_t *ctp, io_link_t *msg,
                      RESMGR_HANDLE_T *handle,
                      io_link_extra_t *extra);

    int (*unblock)   (resmgr_context_t *ctp, io_pulse_t *msg,
                      RESMGR_HANDLE_T *handle, void *reserved);

    int (*mount)     (resmgr_context_t *ctp, io_mount_t *msg,
                      RESMGR_HANDLE_T *handle,
                      io_mount_extra_t *extra);
} resmgr_connect_funcs_t;

To initialize this structure, call iofunc_func_init() to fill it with pointers to the default handlers, and then override any that your resource manager needs to handle specifically.


Note: The resmgr_attach() function copies the pointers to the resmgr_connect_funcs_t and resmgr_io_funcs_t structures, not the structures themselves. You should allocate the structures, declare them to be static, or make them global variables. If your resource manager is for more than one device with different handlers, create separate structures that define the handlers.

The connect messages all have a type of _IO_CONNECT; the subtype further indicates what's happening. The entries are as follows:

nfuncs
The number of functions in the structure. This allows for future expansion.
open
Handles client calls to open(), fopen(), sopen(), and so on. The message subtype is _IO_CONNECT_COMBINE, _IO_CONNECT_COMBINE_CLOSE, or _IO_CONNECT_OPEN.

For more information about the io_open handler, see Ways of adding functionality to the resource manager,” later in this chapter.

unlink
Handles client calls to unlink(). The message subtype is _IO_CONNECT_UNLINK.
rename
Handles client calls to rename(). The message subtype is _IO_CONNECT_RENAME.
mknod
Handles client calls to mkdir(), mkfifo(), and mknod(). The message subtype is _IO_CONNECT_MKNOD.
readlink
Handles client calls to readlink(). The message subtype is _IO_CONNECT_READLINK.
link
Handles client calls to link(). The message subtype is _IO_CONNECT_LINK.
unblock
Handles requests from the kernel to unblock a client during the connect message phase. There's no corresponding message; the call is synthesized by the library.

For more information about the io_unblock handler, see Handling client unblocking due to signals or timeouts in the Signals, Timeouts, and Interrupts chapter.

mount
Handles client calls to mount(). The message subtype is _IO_CONNECT_MOUNT.

For more information about the io_mount handler, see Handling mount() in the Handling Other Messages chapter.

If the message is the _IO_CONNECT message (and variants) corresponding with the open() outcall, then a context needs to be established for further I/O messages that will be processed later. This context is referred to as an OCB (Open Control Block); it holds any information required between the connect message and subsequent I/O messages.

Basically, the OCB is a good place to keep information that needs to be stored on a per-open basis. An example of this would be the current position within a file. Each open file descriptor would have its own file position. The OCB is allocated on a per-open basis. During the open handling, you'd initialize the file position; during read and write handling, you'd advance the file position. For more information, see the section The open control block (OCB) structure in the POSIX-Layer Data Structures chapter of this guide.

I/O messages

An I/O message is one that relies on an existing binding (e.g. OCB) between the client and the resource manager.

As an example, an _IO_READ (from the client's read() function) message depends on the client's having previously established an association (or context) with the resource manager by issuing an open() and getting back a file descriptor. This context, created by the open() call, is then used to process the subsequent I/O messages, like the _IO_READ.

There are good reasons for this design. It would be inefficient to pass the full pathname for each and every read() request, for example. The open() handler can also perform tasks that we want done only once (e.g. permission checks), rather than with each I/O message. Also, when the read() has read 4096 bytes from a disk file, there may be another 20 megabytes still waiting to be read. Therefore, the read() function would need to have some context information telling it the position within the file it's reading from, how much has been read, and so on.

The resmgr_io_funcs_t structure (which you pass to resmgr_attach() along with the connect functions) defines the functions to call for the I/O messages. The resmgr_io_funcs_t structure is defined in <sys/resmgr.h> as follows:

typedef struct _resmgr_io_funcs {
    unsigned    nfuncs;
    int (*read)       (resmgr_context_t *ctp, io_read_t *msg,
                       RESMGR_OCB_T *ocb);
    int (*write)      (resmgr_context_t *ctp, io_write_t *msg,
                       RESMGR_OCB_T *ocb);
    int (*close_ocb)  (resmgr_context_t *ctp, void *reserved,
                       RESMGR_OCB_T *ocb);
    int (*stat)       (resmgr_context_t *ctp, io_stat_t *msg,
                       RESMGR_OCB_T *ocb);
    int (*notify)     (resmgr_context_t *ctp, io_notify_t *msg,
                       RESMGR_OCB_T *ocb);
    int (*devctl)     (resmgr_context_t *ctp, io_devctl_t *msg,
                       RESMGR_OCB_T *ocb);
    int (*unblock)    (resmgr_context_t *ctp, io_pulse_t *msg,
                       RESMGR_OCB_T *ocb);
    int (*pathconf)   (resmgr_context_t *ctp, io_pathconf_t *msg,
                       RESMGR_OCB_T *ocb);
    int (*lseek)      (resmgr_context_t *ctp, io_lseek_t *msg,
                       RESMGR_OCB_T *ocb);
    int (*chmod)      (resmgr_context_t *ctp, io_chmod_t *msg,
                       RESMGR_OCB_T *ocb);
    int (*chown)      (resmgr_context_t *ctp, io_chown_t *msg,
                       RESMGR_OCB_T *ocb);
    int (*utime)      (resmgr_context_t *ctp, io_utime_t *msg,
                       RESMGR_OCB_T *ocb);
    int (*openfd)     (resmgr_context_t *ctp, io_openfd_t *msg,
                       RESMGR_OCB_T *ocb);
    int (*fdinfo)     (resmgr_context_t *ctp, io_fdinfo_t *msg,
                       RESMGR_OCB_T *ocb);
    int (*lock)       (resmgr_context_t *ctp, io_lock_t *msg,
                       RESMGR_OCB_T *ocb);
    int (*space)      (resmgr_context_t *ctp, io_space_t *msg,
                       RESMGR_OCB_T *ocb);
    int (*shutdown)   (resmgr_context_t *ctp, io_shutdown_t *msg,
                       RESMGR_OCB_T *ocb);
    int (*mmap)       (resmgr_context_t *ctp, io_mmap_t *msg,
                       RESMGR_OCB_T *ocb);
    int (*msg)        (resmgr_context_t *ctp, io_msg_t *msg,
                       RESMGR_OCB_T *ocb);
    int (*reserved)   (resmgr_context_t *ctp, void *msg,
                       RESMGR_OCB_T *ocb);
    int (*dup)        (resmgr_context_t *ctp, io_dup_t *msg,
                       RESMGR_OCB_T *ocb);
    int (*close_dup)  (resmgr_context_t *ctp, io_close_t *msg,
                       RESMGR_OCB_T *ocb);
    int (*lock_ocb)   (resmgr_context_t *ctp, void *reserved,
                       RESMGR_OCB_T *ocb);
    int (*unlock_ocb) (resmgr_context_t *ctp, void *reserved,
                       RESMGR_OCB_T *ocb);
    int (*sync)       (resmgr_context_t *ctp, io_sync_t *msg,
                       RESMGR_OCB_T *ocb);
    int (*power)      (resmgr_context_t *ctp, io_power_t *msg,
                       RESMGR_OCB_T *ocb);
} resmgr_io_funcs_t;

You initialize this structure in the same way as the resmgr_connect_funcs_t structure: call iofunc_func_init() to fill it with pointers to the default handlers, and then override any that your resource manager needs to handle specifically. This structure also begins with an nfuncs member that indicates how many functions are in the structure, to allow for future expansion.


Note: The resmgr_attach() function copies the pointers to the resmgr_connect_funcs_t and resmgr_io_funcs_t structures, not the structures themselves. You should allocate the structures, declare them to be static, or make them global variables. If your resource manager is for more than one device with different handlers, create separate structures that define the handlers.

Notice that the I/O functions all have a common parameter list. The first entry is a resource manager context structure, the second is a message (the type of which matches the message being handled and contains parameters sent from the client), and the last is an OCB (containing what we bound when we handled the client's open() function).

You usually have to provide a handler for the following entries:

read
Handles client calls to read() and readdir(). The message type is _IO_READ. For more information about the io_read handler, see Handling the _IO_READ message in the Handling Read and Write Messages chapter.
write
Handles client calls to write(), fwrite(), and so on. The message type is _IO_WRITE. For more information about the io_write handler, see Handling the _IO_WRITE message in the Handling Read and Write Messages chapter.
devctl
Handles client calls to devctl() and ioctl(). The message type is _IO_DEVCTL. For more information about the io_devctl handler, see Handling devctl() messages in the Handling Other Messages chapter.

You typically use the default entry for the following:

close_ocb
Called by the library when the last close() has been received by a particular OCB. You can use this handler to clean up anything associated with the OCB.
stat
Handles client calls to stat(), lstat(), and fstat(). The message type is _IO_STAT. For more information about the io_stat handler, see Handling stat() in the Handling Other Messages chapter.
notify
Handles client calls to select() and ionotify(). The message type is _IO_NOTIFY. For more information about the io_notify handler, see Handling ionotify() and select() in the Handling Other Messages chapter.
unblock
Handles requests from the kernel to unblock the client during the I/O message phase. There's no message associated with this. For more information about the io_unblock handler, see Handling client unblocking due to signals or timeouts in the Signals, Timeouts, and Interrupts chapter.
pathconf
Handles client calls to fpathconf() and pathconf(). The message type is _IO_PATHCONF.
lseek
Handles client calls to lseek(), fseek(), and rewinddir(). The message type is _IO_LSEEK. For more information about the io_lseek handler, see Handling lseek() in the Handling Other Messages chapter.
chmod
Handles client calls to chmod() and fchmod(). The message type is _IO_CHMOD.
chown
Handles client calls to chown() and fchown(). The message type is _IO_CHOWN.
utime
Handles client calls to utime(). The message type is _IO_UTIME.
openfd
Handles client calls to openfd(). The message type is _IO_OPENFD.
fdinfo
Handles client calls to iofdinfo(). The message type is _IO_FDINFO.
lock
Handles client calls to fcntl(), lockf(), and flock(). The message type is _IO_LOCK.
space
Handles client calls to chsize(), fcntl(), ftruncate(), and ltrunc(). The message type is _IO_SPACE.
shutdown
Reserved for future use.
mmap
Handles client calls to mmap(), munmap(), mmap_device_io(), and mmap_device_memory(). The message type is _IO_MMAP.
msg
Handles messages that are manually assembled and sent via MsgSend(). The message type is _IO_MSG. For more information about the io_msg handler, see Handling out-of-band (_IO_MSG) messages in the Handling Other Messages chapter.
reserved
Reserved for future use.
dup
Handles client calls to dup(), dup2(), fcntl(), fork(), spawn*(), and vfork(). The message type is _IO_DUP. For more information about the io_dup handler, see Handling open(), dup(), and close() messages in the Handling Other Messages chapter.
close_dup
Handles client calls to close() and fclose(). The message type is _IO_CLOSE_DUP.

Note: You'll almost never replace the default close_dup handler because the library keeps track of multiple open(), dup(), and close() calls for an OCB. For more information, see open(), dup(), and close(),” below.

lock_ocb
Locks the attributes structure pointed to by the OCB. This is done to ensure that only one thread at a time is operating on both the OCB and the corresponding attributes structure. The lock (and corresponding unlock) functions are synthesized by the resource manager library before and after completion of message handling.
unlock_ocb
Unlocks the attributes structure pointed to by the OCB.
sync
Handles client calls to fsync() and fdatasync(). The message type is _IO_SYNC.
power
Reserved for future use.

Default message handling

Since a large number of the messages received by a resource manager deal with a common set of attributes, the OS provides an iofunc_*() shared library that lets a resource manager handle functions like stat(), chmod(), chown(), lseek(), and so on automatically, without your having to write additional code. As an added benefit, these iofunc_*() default handlers implement the POSIX semantics for the messages, offloading some work from you.

The library contains iofunc_*() default handlers for these client functions:

chmod()
chown()
close()
devctl()
fpathconf()
fseek()
fstat()
lockf()
lseek()
mmap()
open()
pathconf()
stat()
utime()

open(), dup(), and close()

The resource manager shared library automatically handles dup() messages.

Suppose that the client program executed code that eventually ended up performing:

fd = open ("/dev/device", O_RDONLY);
...
fd2 = dup (fd);
...
fd3 = dup (fd);
...
close (fd3);
...
close (fd2);
...
close (fd);

The client generates an open connect message for the first open(), and then two _IO_DUP messages for the two dup() calls. Then, when the client executes the close() calls, it generates three close messages.

Since the dup() functions generate duplicates of the file descriptors, new context information shouldn't be allocated for each one. When the close messages arrive, because no new context has been allocated for each dup(), no release of the memory by each close message should occur either! (If it did, the first close would wipe out the context.)

The resource manager shared library provides default handlers that keep track of the open(), dup(), and close() messages and perform work only for the last close (i.e. the third io_close message in the example above).

Setting resource manager attributes

In addition to the structures that define the connect and I/O functions, you pass a resmgr_attr_t structure to resmgr_attach() to specify the attributes of the resource manager.

The resmgr_attr_t structure is defined as follows:

typedef struct _resmgr_attr {
    unsigned            flags;
    unsigned            nparts_max;
    unsigned            msg_max_size;
    int                 (*other_func)(resmgr_context_t *,
                                      void *msg);
    unsigned            reserved[4];    
} resmgr_attr_t;

The members include:

flags
Lets you change the behavior of the resource manager interface. Set this to 0, or a combination of the following bits (defined in <sys/dispatch.h>):

Note: There are also some _RESMGR_FLAG_* bits (with a leading underscore), but you use them in the flags argument to resmgr_attach().

nparts_max
The number of components that should be allocated to the IOV array.
msg_max_size
The size of the message buffer.

These members will be important when you start writing your own handler functions.

If you specify a value of zero for nparts_max, the resource manager library will bump the value to the minimum usable by the library itself. Why would you want to set the size of the IOV array? As we'll see in the Getting the resource manager library to do the reply section of the Handling Read and Write Messages chapter, you can tell the resource manager library to do our replying for us. We may want to give it an IOV array that points to N buffers containing the reply data. But, since we'll ask the library to do the reply for us, we need to use its IOV array, which of course would need to be big enough to point to our N buffers.

other_func
Lets you specify a routine to call in cases where the resource manager gets an I/O message that it doesn't understand.

Note: In general, we don't recommend that you use this member. For private or custom messages, you should use _IO_DEVCTL or _IO_MSG handlers, as described in the Handling Other Messages chapter. If you want to receive pulses, use pulse_attach().

To attach an other_func, you must set the RESMGR_FLAG_ATTACH_OTHERFUNC bit in the flags member of this structure.

If the resource manager library gets an I/O message that it doesn't know how to handle, it'll call the routine specified by the other_func member, if non-NULL. (If it's NULL, the resource manager library will return an ENOSYS to the client, effectively stating that it doesn't know what this message means.)

You might specify a non-NULL value for other_func in the case where you've specified some form of custom messaging between clients and your resource manager, although the recommended approach for this is the devctl() function call (client) and the _IO_DEVCTL message handler (server) or a MsgSend*() function call (client) and the _IO_MSG message handler (server).

For non-I/O message types, you should use the message_attach() function, which attaches a message range for the dispatch handle. When a message with a type in that range is received, the dispatch_block() function calls a user-supplied function that's responsible for doing any specific work, such as replying to the client.

Ways of adding functionality to the resource manager

You can add functionality to the resource manager you're writing in these fundamental ways:

The first two are almost identical, because the default functions really don't do that much by themselves — they rely on the POSIX helper functions. The third approach has advantages and disadvantages.

Using the default functions

Since the default functions (e.g. iofunc_open_default()) can be installed in the jump table directly, there's no reason you couldn't embed them within your own functions.

Here's an example of how you would do that with your own io_open handler:

main (int argc, char **argv)
{
    …

    /* install all of the default functions */
    iofunc_func_init (_RESMGR_CONNECT_NFUNCS, &connect_funcs,
                      _RESMGR_IO_NFUNCS, &io_funcs);

    /* take over the open function */
    connect_funcs.open = io_open;
    …
}

int
io_open (resmgr_context_t *ctp, io_open_t *msg, 
         RESMGR_HANDLE_T *handle, void *extra)
{
    return (iofunc_open_default (ctp, msg, handle, extra));
}

Obviously, this is just an incremental step that lets you gain control in your io_open handler when the message arrives from the client. You may wish to do something before or after the default function does its thing:

/* example of doing something before */

extern int accepting_opens_now;

int
io_open (resmgr_context_t *ctp, io_open_t *msg,
         RESMGR_HANDLE_T *handle, void *extra)
{
    if (!accepting_opens_now) {
        return (EBUSY);
    }

    /* 
     *  at this point, we're okay to let the open happen,
     *  so let the default function do the "work".
     */

    return (iofunc_open_default (ctp, msg, handle, extra));
}

Or:

/* example of doing something after */

int
io_open (resmgr_context_t *ctp, io_open_t *msg,
         RESMGR_HANDLE_T *handle, void *extra)
{
    int     sts;

    /* 
     * have the default function do the checking 
     * and the work for us
     */

    sts = iofunc_open_default (ctp, msg, handle, extra);

    /* 
     *  if the default function says it's okay to let the open
     *  happen, we want to log the request
     */

    if (sts == EOK) {
        log_open_request (ctp, msg);
    }
    return (sts);
}

It goes without saying that you can do something before and after the standard default POSIX handler.

The principal advantage of this approach is that you can add to the functionality of the standard default POSIX handlers with very little effort.

Using the helper functions

The default functions make use of helper functions — these functions can't be placed directly into the connect or I/O jump tables, but they do perform the bulk of the work.

Here's the source for the two functions iofunc_chmod_default() and iofunc_stat_default():

int
iofunc_chmod_default (resmgr_context_t *ctp, io_chmod_t *msg,
                      iofunc_ocb_t *ocb)
{
    return (iofunc_chmod (ctp, msg, ocb, ocb -> attr));
}

int
iofunc_stat_default (resmgr_context_t *ctp, io_stat_t *msg,
                     iofunc_ocb_t *ocb)
{
    iofunc_time_update (ocb -> attr);
    iofunc_stat (ocb -> attr, &msg -> o);
    return (_RESMGR_PTR (ctp, &msg -> o,
                         sizeof (msg -> o)));
}

Notice how the iofunc_chmod() handler performs all the work for the iofunc_chmod_default() default handler. This is typical for the simple functions.

The more interesting case is the iofunc_stat_default() default handler, which calls two helper routines. First it calls iofunc_time_update() to ensure that all of the time fields (atime, ctime and mtime) are up to date. Then it calls iofunc_stat(), which builds the reply. Finally, the default function builds a pointer in the ctp structure and returns it.

The most complicated handling is done by the iofunc_open_default() handler:

int
iofunc_open_default (resmgr_context_t *ctp, io_open_t *msg,
                     iofunc_attr_t *attr, void *extra)
{
    int     status;

    iofunc_attr_lock (attr);

    if ((status = iofunc_open (ctp, msg, attr, 0, 0)) != EOK) {
        iofunc_attr_unlock (attr);
        return (status);
    }

    if ((status = iofunc_ocb_attach (ctp, msg, 0, attr, 0)) 
        != EOK) {
        iofunc_attr_unlock (attr);
        return (status);
    }

    iofunc_attr_unlock (attr);
    return (EOK);
}

This handler calls four helper functions:

  1. It calls iofunc_attr_lock() to lock the attribute structure so that it has exclusive access to it (it's going to be updating things like the counters, so we need to make sure no one else is doing that at the same time).
  2. It then calls the helper function iofunc_open(), which does the actual verification of the permissions.
  3. Next it calls iofunc_ocb_attach() to bind an OCB to this request, so that it will get automatically passed to all of the I/O functions later.
  4. Finally, it calls iofunc_attr_unlock() to release the lock on the attribute structure.

Writing the entire function yourself

Sometimes a default function will be of no help for your particular resource manager. For example, iofunc_read_default() and iofunc_write_default() functions implement /dev/null — they do all the work of returning 0 bytes (EOF) or swallowing all the message bytes (respectively).

You'll want to do something in those handlers (unless your resource manager doesn't support the _IO_READ or _IO_WRITE messages).

Note that even in such cases, there are still helper functions you can use: iofunc_read_verify() and iofunc_write_verify().