Scaling the network interface driver framework (iflib)

QNX SDP8.0Performance Tuning User GuideUser
You may scale iflib for larger systems with the following method:

Setting the number of PCI interrupts and queues

By default, the iflib framework used by many PCI drivers allocates a PCI MSI-X interrupt and queue for each CPU core to implement receive-side scaling (RSS) on each interface. In a system with many such interfaces, this mechanism can result in large numbers of interrupts allocated and multiple queues running on each core.

Because there are a limited number of MSI-X interrupts available in a system, high core and interface counts can lead to the system running out of interrupts. These errors are reported in slog2info. For example:

Jan 10 23:55:14.235 io_sock.184334 main_buffer 131 ixl2: Failed to enable MSI-X on 3:0:2 :The Requested Number And Type Of Interrupts Are Unavailable Or Could Not Be Reserved With Current Request Type [(PCI_ERR_IRQ_NOT_AVAIL)]

In addition, each queue is implemented as a thread, and multiple queues running on each core can lead to poor performance with excessive context switching between the threads.

You can resolve both these issues by using a sysctl tunable to set the number of queues per interface to an appropriate value rather than letting iflib use the default (the number of CPUs). The tunable uses the format: dev.device_type.device_instance.iflib.override_nrxqs.

For example, in a system with 24 CPUs and eight interfaces, the default total number of interrupts and queue processing threads is 192. To reduce it to 24, first create the configuration file /etc/io-sock.conf with the following contents:

dev.ix.0.iflib.override_nrxqs="3"
dev.ix.1.iflib.override_nrxqs="3"
dev.ix.2.iflib.override_nrxqs="3"
dev.ix.3.iflib.override_nrxqs="3"
dev.ix.4.iflib.override_nrxqs="3"
dev.ix.5.iflib.override_nrxqs="3"
dev.ix.6.iflib.override_nrxqs="3"
dev.ix.7.iflib.override_nrxqs="3"
dev.ix.0.iflib.override_ntxqs="3"
dev.ix.1.iflib.override_ntxqs="3"
dev.ix.2.iflib.override_ntxqs="3"
dev.ix.3.iflib.override_ntxqs="3"
dev.ix.4.iflib.override_ntxqs="3"
dev.ix.5.iflib.override_ntxqs="3"
dev.ix.6.iflib.override_ntxqs="3"
dev.ix.7.iflib.override_ntxqs="3"

Then, pass the file to io-sock when you start it. For example:

io-sock -o config=/etc/iosock.conf -m phy -m pci -d ix
Page updated: