Appendix: Advanced Qnet Topics

This appendix covers some advanced aspects of Transparent Distributed Processing (TDP) or Qnet, including:

Low-level discussion of Qnet principles

The Qnet protocol extends interprocess communication (IPC) transparently over a network of microkernels. This is done by taking advantage of the Neutrino's message-passing paradigm. Message passing is the central theme of Neutrino that manages a group of cooperating processes by routing messages. This enhances the efficiency of all transactions among all processes throughout the system.

As we found out in the How does it work? section of the Transparent Distributed Processing Using Qnet chapter, many POSIX and other function calls are built on this message passing. For example, the write() function is built on the MsgSendv() function. In this section, you'll find several things, e.g. how Qnet works at the message passing level; how node names are resolved to node numbers, and how that number is used to create a connection to a remote node.

In order to understand how message passing works, consider two processes that wish to communicate with each other: a client process and a server process. First we consider a single-node case, where both client and server reside in the same machine. In this case, the client simply creates a connection (via ConnectAttach()) to the server, and then sends a message (perhaps via MsgSend()).

The Qnet protocol extends this message passing over to a network. For example, consider the case of a simple network with two machines: one contains the client process, the other contains the server process. The code required for client-server communication is identical (it uses same API) to the code in the single-node case. The client creates a connection to the server and sends the server a message. The only difference in the network case is that the client specifies a different node descriptor for the ConnectAttach() function call in order to indicate the server's node. See the diagram below to understand how message passing works.

Message passing

Details of Qnet data communication

As mentioned before, Qnet relies on the message passing paradigm of Neutrino. Before any message pass, however, the application (e.g. the client) must establish a connection to the server using the low-level ConnectAttach() function call:

ConnectAttach(nd, pid, chid, index, flags);

In the above call, nd is the node descriptor that specifies which node you're connecting to. The node descriptor tells the kernel whether you're communicating to a local or remote server process. If nd is zero, you're specifying a local server process, and you'll get local message passing from the client to the server, carried out by the local kernel as shown below:

Message passing in the same machine

When you specify a nonzero value for nd, the application transparently passes messages to a server on another machine, and connects to a server on another machine. This way, Qnet not only builds a network of trusted machines, it lets all these machines share their resources with little overhead.

Message passing in two different machines

Node descriptors

Node descriptors refer to nodes (machines) in a QNet network much the same way that file descriptors refer to a particular file. They're dynamically created when needed, last as long as needed, and while in use refer to a particular file. But if two different nodes are referring to the same third node, they may end up with different node descriptors, just as when two different processes open() the same file, they may end up with different file descriptors. A node may even have two different node descriptors to the same remote node, if those references to the remote node have different Quality of Service information.

The <sys/netmgr.h> header file

The <sys/netmgr.h> header defines the ND_LOCAL_NODE macro as zero. You can use it any time that you're dealing with node descriptors to make it obvious that you're talking about the local node.

If you want to see if two node descriptors from the same node refer to the same machine, you can't just arithmetically compare the descriptors for equality; use the ND_NODE_CMP() macro instead:

This is similar to the way that strcmp() and memcmp() work. It's done this way in case you want to do any sorting that's based on node descriptors.

The <sys/netmgr.h> header file also defines the following networking functions:

netmgr_strtond()
Convert a string into a node descriptor.
netmgr_ndtostr()
Convert a node descriptor into a string.
netmgr_remote_nd()
Get a node descriptor that's relative to a remote node.

For more information, see the C Library Reference.