Memory sharing

Guests in a hypervisor system can share memory regions through which they can pass data to each other.

In a QNX Hypervisor system, client applications running in guests create and manage shared memory, and can use shared memory regions to share data. If data needs to be shared between a guest or guests and the hypervisor host domain, applications running in the hypervisor host domain can also create shared memory regions, or attach to regions created by guests.

Assuming that all sharing is between guests, and not with the hypervisor host, to set up memory sharing in your hypervisor system:

Edit the qvm configuration files (*.qvmconf) for the VMs hosting the guests that will share memory to enable shared memory services (see “Configuring the VMs” below).
Create a client application that runs in the guests and uses shared memory services, and run it on multiple guests.

The hypervisor-shmem-examples-march31.tgz archive available in the QNX Hypervisor package includes source code for example memory sharing programs: ghstest.c for a QNX guest, and for the hypervisor host.

The binaries for a QNX Neutrino OS 7.0 guest are included in the hypervisor files. If you want to try this shared memory program with a QNX Neutrino OS 6.6 guest, you can download and unpack the example program binary files (see “Additional files for QNX Neutrino OS 6.6 guests” in the “Assembling a Hypervisor System and Its Components” chapter).

How shared memory works

To use shared memory, a client (an application running in a guest or in the hypervisor host) needs:

a mapping of the shared memory region
a hardware interrupt it can use to signal other users of the shared memory regions that this region has been updated

The hypervisor includes the vdev-shmem vdev, which implements setting up the shared memory mapping and the interrupts you need to use shared memory. This vdev provides additional functionality to simplify using shared memory. These include:

shared memory region names (a hypervisor system may include multiple named shared regions)
selective signaling (e.g., signal Guest 0, but not Guest 3)
signal knowledge (the ability to know which guests have been signaled)

The figure below illustrates memory sharing between two guests. Guest 0 attempts to attach to a 128 MB shared memory area first. Since no such area exists at the specified location, the attempt to attach creates the area and allows the guest to attach to it. Guest 1 can simply attach to the same name to share data memory with Guest 0.

Figure 1. A 128 MB memory allocation shared by Guest 0 and Guest 1

For more information about shared memory regions, see “Factory and control pages” below.

Note:

All the connections to a shared memory region are peers. That is, there is no distinction between the guest that creates a shared memory region and the guest that attaches to it. Simply, the first attempt to attach to a shared memory region creates the region. As far as the guest is concerned, it simply attaches to the region.

This design avoids ordering problems where the system designer would have to make sure that one guest always comes up and creates the shared memory region before another guest tries to attach to it.

Factory and control pages

The QNX Hypervisor shared memory implementation uses factory pages and control pages. These pages are the same size as a hypervisor kernel page (4 KB).

Factory pages

Including vdev-shmem vdevs in the qvm configuration file for the VM hosting a guest causes the qvm process to create a factory page for the VM. Typically, there is only one vdev-shmem per qvm configuration file, hence one factory page per VM.

When an application in the guest attempts to create a shared memory region, the hypervisor uses the factory page to create the shared memory region and its control page. A factory page contains information about the shared memory, including a field with the guest-physical address for the shared memory region's control page. This field changes value as the guest creates or attaches to different shared memory regions.

A factory page may be located anywhere in unallocated memory that the guest can access. For example, assuming no other memory has been allocated yet, if we allocated 192 MB of RAM (ram 192M set in the qvm configuration file), we can place the factory page outside this allocated RAM at 0x10000000 (256 MB). This address is a guest-physical address, not an actual physical address in hardware. The vdev-shmem vdev virtualizes the factory page for the guest. No other device may use this location.

The vdev-shmem vdev can be configured to be seen by the guest as either a PCI device or a memory-mapped I/O (MMIO) device:

MMIO: For an MMIO vdev-shmem, specify the loc and intr properties for the vdev-shmem vdev. The hypervisor puts the factory page at the location you specify, the vdev uses the interrupt you specify, and the guest sees the vdev as an MMIO device.
PCI: For a PCI vdev-shmem, do not specify the loc and intr properties for the vdev-shmem vdev. The hypervisor chooses the location for the factory page and the interrupt the vdev will use, and the guest sees the vdev as a PCI device.

Control page

The first request from a guest for a new shared memory region sets up the requested shared memory region, plus a control page. Every shared memory region has its own control page that prefixes the shared memory region.

After a guest creates a shared memory region or attaches to it, the guest uses a field on the system page to locate the region's control page. When guests need to communicate with other guests, they write to the control page.

Figure 2. A factory page with its pointer to the control page for a shared memory region.

The diagram above illustrates how a VM's factory page points to a control page that is prefixed to a shared memory region.

Configuring the VMs

To include the vdev-shmem vdev in the qvm process for the VM that will host a guest using shared memory services, simply add it to the qvm configuration file. Make sure that it is in the qvm configuration file for every VM with a guest that will need to use the shared memory services. For example:

# qnxcluster.qvmconf example
#Give this guest a name
system cluster
ram 1024M
cpu
cpu
load /emmc/QNX_cluster.ifs
vdev ioapic
        loc 0xf8000000
        intr apic
        name myioapic
vdev ser8250
        intr myioapic:4
vdev timer8254
        intr myioapic:0
vdev mc146818
vdev shmem
vdev pckeyboard
# set up tap device
vdev vdev-virtio-net
        tapdev tap0

The line of particular interest in the qvm configuration information above is:

vdev shmem

This is a PCI device. If you specify its loc and intr properties, the guest will see it as an MMIO device at the specified location. For example:

vdev shmem
	create TEST1, 0xf0000	# name is TEST1 with size of 0xf0000
	loc 0x10000000		# location of factory page
	intr myioapic:10	# hardware interrupt used for signaling

will cause the qvm process to create a shared memory region “TEST1” when it starts up and assembles the VM. The configuration assumes that a vdev ioapic called “myioapic” has already been specified.

Note:

Since the qvm process creates the shared memory region when it starts up (rather than when a guest attempts to attach to it), you can ensure there's enough underlying physical memory on the host system for the region.

For more information about qvm configuration files and how to use them to assemble a VM, see “Assembling and configuring VMs” in the “Configuration” chapter.

Using the allow and deny options

You can use the vdev shmem allow and deny options to establish a list of shared memory regions the guest may and may not access (see vdev shmem).