Designing a system using Qnet

In order to explain the design of a system that takes advantage of the power of Qnet by performing distributed processing, consider a multiprocessor hardware configuration that's suitable for a typical telecom box.

This configuration has a generic controller card and several data cards to start with. These cards are interconnected by a high-speed transport (HST) bus. The controller card configures the box by communicating with the data cards, and establishes/enables data transport in and out of the box (i.e., data cards) by routing packets.

The typical challenges to consider for this type of box include:

You need several pieces of software components (along with the hardware) to build your distributed system.

Before going into further details, you might like to review the following sections from Using Qnet for Transparent Distributed Processing chapter in the QNX Neutrino User's Guide:

Configuring the data cards

Power up the data cards to start procnto and qnet in sequence. These data cards need a minimal amount of flash memory to store the QNX Neutrino image.

In the buildfile of the data cards, you should link the directories of the data cards to the controller cards as follows:

[type=link] /bin  = /net/cc0/bin
[type=link] /sbin = /net/cc0/sbin
[type=link] /usr  = /net/cc0/usr

where cc0 is the name of the controller card.

Assuming that the data card has a console and shell prompt, try the following commands:

$ ls /net

You get a list of boards running QNX Neutrino and Qnet:

 cc0  dc0  dc1  dc2  dc3

Or, use the following command on a data card:

$ ls /net/cc0

You get the following output (i.e., the contents of the root of the filesystem for the controller card):

.                  .inodes            mnt0         tmp
..                 .longfilenames     mnt1         usr
.altboot           bin                net          var
.bad_blks          dev                proc         xfer
.bitmap            etc                sbin
.boot              home               scratch

Configuring the controller card

Configure the controller card in order to access different servers running on it—either by the data cards, or by the controller card itself. Make sure that the controller card has a larger amount of flash memory than the data cards do. This flash memory contains all the binaries, data and configuration files that the applications on the data cards access as if they were on a local storage device.

Call the following API to communicate with the mqueue server by any application:

mq_open("/net/cc0/dev/mqueue/app_q", ....)

A simple variation of the above command requires that you run the following command during initialization:

$ ln -s /net/cc0/dev/mqueue /mq

Then all applications, whether they're running on the data cards or on the controller card, can call:

mq_open("/mq/app_q", ....)

Similarly, applications can even utilize the TCP/IP stack running on the controller card.

Enhancing reliability via multiple transport buses

Qnet provides design choices to improve the reliability of a high-speed transport bus, most often a single-point of failure in such type of telecom box.

You can choose between different transport selections to achieve a different Quality of Service (or QoS), such as:
  • loadbalance—no interface specified
  • preferred—specify an interface, but allow failover
  • exclusive—specify an interface, no failover

These selections allow you to control how data will flow via different transports.

In order to do that, first, find out what interfaces are available, by using the ifconfig command at the prompt of any card. For this example, we'll assume that the HST 0 and 1 (hs0 and hs1) interfaces are available.

Select your choice of transport as follows:
Use this command: To select this transport:
ls /net/cc0 Loadbalance, the default choice
ls /net/cc0~preferred:hs0 Preferred. Try HST 0 first; if that fails, then transmit on HST 1.
ls /net/cc0~exclusive:hs0 Exclusive. Try HST 0 first. If that fails, terminate transmission.

You can have another economical variation of the above hardware configuration:

This configuration has asymmetric transport: a High-Speed Transport (HST) and a reliable and economical Low-Speed Transport (LST). You might use the HST for user data, and the LST exclusively for out-of-band control (which can be very helpful for diagnosis and during booting). For example, if you use generic Ethernet as the LST, you could use a DHCP ROM on the data cards to economically boot—no flash would be required on the data cards.

With asymmetric transport, use of the QoS policy as described above likely becomes even more useful. You might want some applications to use the HST link first, but use the LST if the HST fails. You might want applications that transfer large amounts of data to exclusively use the HST, to avoid swamping the LST.

Redundancy and scalability using multiple controller cards

The reliability of such a telecom box also hinges on the controller card, which is a critical component and certainly a potential SPOF (single point of failure). You can increase the reliability of this telecom box by using additional controller cards.

For example, you could add another controller card for redundancy:

Once the (second) controller card is installed, the challenge is in the determination of the primary controller card. This is done by the software running on the controller cards. By default, applications on the data cards access the primary controller card. Assuming cc0 is the primary controller card, Use the following command to access this card in the /cc directory:

ln -s /net/cc0 /cc

The above indirection makes communication between data card and controller card transparent. In fact, the data cards remain unaware of the number of controller cards, or which card is the primary controller card.

Applications on the data cards access the primary controller card. In the event of failure of the primary controller card, the secondary controller card takes over. The applications on the data cards redirect their communications via Qnet to the secondary controller card.

You can also scale your resources to run a particular server application using additional controller cards. For example, if your controller card (either a SMP or non-SMP board) doesn't have the necessary resources (e.g., CPU cycles, memory), you could increase the total processor and box resources by using additional controller cards. Qnet transparently distributes the (load of) application servers across two or more controller cards.