---

Overview

This chapter includes:

• What's new in the networking stack?
• Architecture of io-pkt
• Threading model
• Threading priorities
• Components of core networking
• Getting the source code

What's new in the networking stack?

The QNX Neutrino networking stack is called io-pkt. It replaces the previous generation of the stack, io-net, and provides the following benefits:

• performance improvements. Since io-pkt removes the npkt-to-mbuf translation and mandatory queuing, and also reduces context switching on the packet receive path, the IP receive performance is greatly improved.
• simplified locking of shared resources, resulting in simpler SMP support
• it closely follows the NetBSD code base and architecture, meaning:
  • easier maintenance / upgrade capability of IP stack source
  • existing applications that use BSD standard APIs will port more easily (e.g. tcpdump)
  • enhanced features included with NetBSD stack are also included with io-pkt
  • NetBSD drivers will port in a straightforward manner
• far richer stack feature set, drawing on the latest in improvements from the NetBSD code base
• 802.11 Wi-Fi client and access point capability

The io-pkt manager is intended to be a drop-in replacement for io-net for those people who are dealing with the stack from an outside application point of view. It includes stack variants, associated utilities, protocols, libraries and drivers.

The stack variants are:

io-pkt-v4

IPv4 version of the stack with no encryption or Wi-Fi capability built in. This is a "reduced footprint" version of the stack that doesn't support the following:

• IPv6
• Crypto / IPSec
• 802.11 a/b/g WiFi
• Bridging
• GRE / GRF
• Multicast routing
• Multipoint PPP

io-pkt-v4-hc

IPv4 version of the stack that has full encryption and Wi-Fi capability built in and includes hardware-accelerated cryptography capability (Fast IPsec).

io-pkt-v6-hc

IPv6 version of the stack (includes IPv4 as part of v6) that has full encryption and Wi-Fi capability, also with hardware-accelerated cryptography.

We've designed io-pkt to follow as closely as possible the NetBSD networking stack code base and architecture. This provides an optimal path between the IP protocol and drivers, tightly integrating the IP layer with the rest of the stack.

The io-pkt implementation makes significant changes to the QNX Neutrino stack architecture, including the following:

• io-net is replaced by the stack's link layer
• mbufs are used throughout, including in the drivers
• all buffer management is handled by the stack
• mount and umount capabilities:
  • Only io-net drivers may be both mounted and unmounted. Other drivers may allow you to detach the driver from the stack, by using the ifconfig iface destroy command (if the driver supports it).
  • The IP stack is an integral part of io-pkt; you can't start io-pkt without it. This means that you don't need to specify the -ptcpip option to the stack unless there are additional parameters (e.g. prefix=) that you need to pass to it. If you specify the -ptcpip option without additional parameters, io-pkt accepts it with no effect.
  • Protocols and enhanced stack functionality (e.g. TDP, NAT / IP Filtering) can be mounted, but not unmounted.
• The concepts of producers and consumers no longer exist within io-pkt. Filters still exist, but they use a different API than with io-net. The io-pkt stack does provide other hooks into the stack that you can use to provide a similar level of functionality. These include:

  Berkeley Packet Filter interface

  Lets you read and write, but not modify or discard, both IP and Ethernet packets from your application.

  Packet Filter interface

  pfil hooks, enabled when the PF filter module is loaded, let you read, write, and modify IP and Ethernet packets within the context of the stack process.

• Driver changes:
  • The driver model has changed to provide better integration with the protocol stack. (For example, in io-net, npkts had to be converted into mbufs for use with the stack. In io-pkt, mbufs are used throughout.)
  • The driver API and behavior have been changed to closely match those of the NetBSD stack, allowing NetBSD drivers to be ported to io-pkt.
  • A shim layer, devnp-shim.so, is provided that lets you use an io-net driver as-is.
  • Driver interfaces are no longer sequentially numbered with enx designations; they're named according to driver type (e.g. fxp0 is the interface for an Intel 10/100 driver).
  • Drivers no longer present entries in the name space to be directly opened and accessed with a devctl() command (e.g. open(/dev/io-net/en0)). Instead, a socket file descriptor is opened and queried for interface information. The ioctl() command is then sent to the stack using this device information (see the source to nicinfo for an example of how this works).
• IP Filtering and NAT are now handled through the PF interface with pfctl. This replaces the io-net ipf interface, which is no longer supported.
• SCTP isn't supported in the initial release of io-pkt.
• In io-net, loopback-checksumming was done with ifconfig. In io-pkt this is controlled via sysctl:

  # sysctl -a | grep do_loopback_cksum
  net.inet.ip.do_loopback_cksum = 0
  net.inet.tcp.do_loopback_cksum = 0
  net.inet.udp.do_loopback_cksum = 0
    

• The nicinfo utility operates slightly differently from the way it did under io-net:
  • The default operation with no arguments is to list information on all interfaces. The behavior with io-net was to show stats for /dev/io-net/en0.
  • Under io-net, all Ethernet interface names were in the form enX. Under io-pkt, this name will vary.
  • Ported NetBSD drivers might not support the nicinfo ioctl() command.

Architecture of io-pkt

The io-pkt stack is very similar in architecture to other component subsystems inside of the Neutrino operating system. At the bottom layer are drivers that provide the mechanism for passing data to, and receiving data from, the hardware. The drivers hook into a multi-threaded layer-2 component (that also provides fast forwarding and bridging capability) that ties them together and provides a unified interface into the layer-3 component, which then handles the individual IP and upper-layer protocol-processing components (TCP and UDP).

In Neutrino, a resource manager forms a layer on top of the stack. The resource manager acts as the message-passing intermediary between the stack and user applications. It provides a standardized type of interface involving open(), read(), write(), and ioctl() that uses a message stream to communicate with networking applications. Networking applications written by the user link with the socket library. The socket library converts the message-passing interface exposed by the stack into a standard BSD-style socket layer API, which is the standard for most networking code today.

One of the big differences that you'll see with this stack as compared to io-net is that it isn't currently possible to decouple the layer 2 component from the IP stack. This was a trade-off that we made to allow increased performance at the expense of some reduced versatility. We might look at enabling this at some point in the future if there's enough demand.

In addition to the socket-level API, there are also other, programmatic interfaces into the stack that are provided for other protocols or filtering to occur. These interfaces -- used directly by Transparent Distributed Processing (TDP, also known as Qnet) -- are very different from those provided by io-net, so anyone using similar interfaces to these in the past will have to rewrite them for io-pkt.

---

---

A detailed view of the io-pkt architecture.

At the driver layer, there are interfaces for Ethernet traffic (used by all Ethernet drivers), and an interface into the stack for 802.11 management frames from wireless drivers. The hc variants of the stack also include a separate hardware crypto API that allows the stack to use a crypto offload engine when it's encrypting or decrypting data for secure links. You can load drivers (built as DLLs for dynamic linking and prefixed with devnp-) into the stack using the -d option to io-pkt.

APIs providing connection into the data flow at either the Ethernet or IP layer allow protocols to coexist within the stack process. Protocols (such as Qnet) are also built as DLLs. A protocol links directly into either the IP or Ethernet layer and runs within the stack context. They're prefixed with lsm (loadable shared module) and you load them into the stack using the -p option. The tcpip protocol (-ptcpip) is a special option that the stack recognizes, but doesn't link a protocol module for (since the IP stack is already present). You still use the -ptcpip option to pass additional parameters to the stack that apply to the IP protocol layer (e.g. -ptcpip prefix=/alt to get the IP stack to register /alt/dev/socket as the name of its resource manager).

A protocol requiring interaction from an application sitting outside of the stack process may include its own resource manager infrastructure (this is what Qnet does) to allow communication and configuration to occur.

In addition to drivers and protocols, the stack also includes hooks for packet filtering. The main interfaces supported for filtering are:

Berkeley Packet Filter (BPF) interface

A socket-level interface that lets you read and write, but not modify or block, packets, and that you access by using a socket interface at the application layer (see http://en.wikipedia.org/wiki/Berkeley_Packet_Filter). This is the interface of choice for basic, raw packet interception and transmission and gives applications outside of the stack process domain access to raw data streams.

Packet Filter (PF) interface

A read/write/modify/block interface that gives complete control over which packets are received by or transmitted from the upper layers and is more closely related to the io-net filter API.

For more information, see the Packet Filtering chapter.

Threading model

The default mode of operation is for io-pkt to create one thread per CPU. The io-pkt stack is fully multi-threaded at layer 2. However, only one thread may acquire the "stack context" for upper-layer packet processing. If multiple interrupt sources require servicing at the same time, these may be serviced by multiple threads. Only one thread will service a particular interrupt source at any point in time. Typically an interrupt on a network device indicates that there are packets to be received. The same thread that handles the receive processing may later transmit the received packets out another interface. Examples of this are layer-2 bridging and the "ipflow" fastforwarding of IP packets.

The stack uses a thread pool to service events that are generated from other parts of the system. These events include:

• time-outs
• ISR events
• other things generated by the stack or protocol modules

You can use a command-line option to the driver to control the priority of threads that receive packets. Client connection requests are handled in a floating priority mode (i.e. the thread priority matches that of the client application thread accessing the stack resource manager).

Once a thread receives an event, it examines the event type to see if it's a hardware event, stack event, or "other" event:

• If the event is a hardware event, the hardware is serviced and, for a receive packet, the thread determines whether bridging or fast-forwarding is required. If so, the thread performs the appropriate lookup to determine which interface the packet should be queued for, and then takes care of transmitting it, after which it goes back to check and see if the hardware needs to be serviced again.
• If the packet is meant for the local stack, the thread queues the packet on the stack queue. The thread then goes back and continues checking and servicing hardware events until there are no more events.
• Once a thread has completed servicing the hardware, it checks to see if there's currently a stack thread running to service stack events that may have been generated as a result of its actions. If there's no stack thread running, the thread becomes the stack thread and loops, processing stack events until there are none remaining. It then returns to the "wait for event" state in the thread pool.

This capability of having a thread change directly from being a hardware-servicing thread to being the stack thread eliminates context switching and greatly improves the receive performance for locally terminated IP flows.

Threading priorities

There are a couple of ways that you can change the priority of the threads responsible for receiving packets from the hardware. You can pass the rx_prio_pulse option to the stack to set the default thread priority. For example:

io-pkt-v4 -ptcpip rx_pulse_prio=50

This makes all the receive threads run at priority 50. The current default for these threads is priority 21.

The second mechanism lets you change the priority on a per-interface basis. This is an option passed to the driver and, as such, is supported only if the driver supports it. When the driver registers for its receive interrupt, it can specify a priority for the pulse that is returned from the ISR. This pulse priority is what the thread will use when running. Here's some sample code from the devn-mpc85xx.so Ethernet driver:

if ((rc = interrupt_entry_init(&mpc85xx->inter_rx, 0, NULL,
    cfg->priority)) != EOK) {
        log(LOG_ERR, "%s(): interrupt_entry_init(rx) failed: %d",
           __FUNCTION__, rc);
        mpc85xx_destroy(mpc85xx, 9);
                return rc;
}

For example, to have the stack start up and receive packets on one interface at priority 20 and on a second interface at priority 50 on a single-processor system, you would use the following command-line options:

io-pkt-v4 -t2 -dmpc85xx syspage=1,priority=20,pci=0 \
-dmpc85xx syspage=1,priority=50,pci=1

If you've specified a per-interface priority, and there are more interfaces than threads, the stack sends a warning to slogger. If there are insufficient threads present, the per-interface priority is ignored (but the rx_pulse_prio option is still honored).

The actual options for setting the priority and selecting an individual card depend on the device driver; see the driver documentation for specific option information.

Legacy io-net drivers create their own receive thread, and therefore don't require the -t option to be used if they support the priority option. These drivers use the devnp-shim.so shim driver to allow interoperability with the io-pkt stack.

Components of core networking

The io-pkt manager is the main component; other core components include:

pfctl, lsm-pf-v6.so, lsm-pf-v4.so

IP Filtering and NAT configuration and support.

ifconfig, netstat, sockstat (see the NetBSD documentation), sysctl

Stack configuration and parameter / information display.

pfctl

Priority packet queuing on Tx (QoS).

lsm-autoip.so

Auto-IP interface configuration protocol.

brconfig

Bridging and STP configuration along with other layer-2 capabilities.

pppd, pppoed, pppoectl

PPP support for io-pkt, including PPP, PPPOE (client), and Multilink PPP.

devnp-shim.so

io-net binary-compatibility shim layer.

nicinfo

Driver information display tool (for native and io-net drivers).

libsocket.so

BSD socket application API into the network stack.

libpcap.so, tcpdump

Low-level packet-capture capability that provides an abstraction layer into the Berkeley Packet Filter interface.

lsm-qnet.so

Transparent Distributed Processing protocol for io-pkt.

hostapd, hostapd_cli (see the NetBSD documentation), wpa_supplicant, wpa_cli

Authentication daemons and configuration utilities for wireless access points and clients.

QNX Neutrino Core Networking also includes applications, services, and libraries that interface to the stack through the socket library and are therefore not directly dependent on the Core components. This means that they use the standard BSD socket interfaces (BSD socket API, Routing Socket, PF_KEY, raw socket):

libssl.so, libssl.a

SSL suite ported from the source at http://www.openssl.org.

libnbdrvr.so

BSD porting library. An abstraction layer provided to allow the porting of NetBSD drivers.

libipsec(S).a, setkey

NetBSD IPsec tools.

inetd

Updated Internet daemon.

route

Updated route-configuration utility.

ping, ping6

Updated ping utilities.

ftp, ftpd

Enhanced FTP.

Getting the source code

If you want to get the source code for io-pkt and other components, go to Foundry27, the community portal for QNX developers (http://community.qnx.com/sf/sfmain/do/home).

---