Guest IPIs

The cost of an IPI between physical CPUs (pCPUs) typically is less than a microsecond when initiated by an operating system running directly on the hardware. While this overhead isn't significant, any excessive use of IPIs may affect system performance.

Just like any OS running running directly on hardware with multiple pCPUs, a guest OS running in a VM with multiple vCPUs may need to issue IPIs. From the perspective of the OS issuing the IPI, the IPI behavior is exactly the same regardless of whether the OS is running directly on hardware or as a guest in a VM.

However, the time overhead cost of an IPI issued by a guest OS in a VM is an order of magnitude greater that the cost of an IPI issued by an OS running directly on hardware. This cost is similar to the cost of a guest exit–entrance cycle, which typically can take ten microseconds, sometimes longer.

Since a guest OS runs in a VM and its CPUs are in fact vCPUs (threads in the hosting qvm process instance), when a guest issues an IPI, the IPI source is a vCPU thread, and each IPI target is another vCPU thread.

The relatively high cost of guest-issued IPIs is due to the work required by the hypervisor to prepare and deliver these IPIs; that is, to the work that must be done by software rather than by hardware to deliver the IPI from its source vCPU thread to its target vCPU thread or threads.

Preparing the IPI for delivery

The hypervisor tasks described below prepare a guest-issued IPI for delivery. They are the same regardless of the board architecture or the state of the target vCPU thread.

When the guest OS issues an IPI, the hypervisor must:

1. Trap the request in the source vCPU thread.
2. Write the hosting VM's interrupt controller registers (virtual ICRs) with the information that will be needed to deliver the guest-issued IPI to all target vCPUs.
3. Compile this information into a list of target vCPUs and interrupt numbers.
4. For each target vCPU, inject the relevant interrupt number. That is, prepare the data and logic that will cause that target vCPU to see that it has an interrupt pending when it resumes.

Delivering the IPI

From this point forward, the work required to deliver a guest-issued IPI to a vCPU is the same as for delivering any interrupt to a vCPU, regardless of the interrupt's source. This work differs according to the state of each target vCPU thread. Also, some boards support posted interrupts, which can reduce overhead.

Target vCPU thread isn't executing

If a target vCPU isn't executing guest code (the guest is stopped on an HLT or WFI), then the host will see the target vCPU thread as stopped on a semaphore. In this case, assuming that the source vCPU thread prepared the IPI for delivery:

• Normal QNX OS scheduling policies apply and the target vCPU thread will begin executing again when its turn comes.
• When the vCPU thread begins executing, it picks up the guest-issued IPI from its VM's virtual ICRs.

Target vCPU thread is executing

If a target vCPU thread is executing guest code:

• The hypervisor must trigger an IPI on the pCPU where the guest code is executing; if posted interrupts aren't supported by the hardware or aren't enabled, the interrupt on the pCPU forces a guest exit and immediate re-entrance.
• When the guest re-enters, the target vCPU thread will pick up the pending guest-issued IPI from its VM's virtual ICRs.

However, note the following:

• If a target vCPU thread is executing vdev code, no forced guest exit is needed. Virtual devices are part of the hypervisor; when execution passes from the virtual device back to the guest, the target vCPU will pick up the pending guest-issued IPI from its VM's virtual ICRs.
• If the hardware supports posted interrupts (as some x86 boards do) and this capability is enabled, then no guest exit is required to deliver a guest-issued IPI to its target vCPU(s). The hypervisor's IPI on the pCPU triggers delivery of the guest-issued IPI, and the guest learns about the interrupt without having to exit.

Reducing the frequency of guest-issued IPIs

The costliest tasks in the preparation and delivery of a guest-issued IPI to its target vCPU(s) are:

• Compiling the list of target vCPUs. There is no mechanism for circumventing this task.
• The guest exit–entrance cycle. This cost can be reduced by enabling posted interrupts, if the hardware supports this capability.

Given the high cost of guest-issued IPIs, even on boards with posted interrupt support, reducing the frequency of guest-issued IPIs can improve both guest and overall system performance. This reduction can often be achieved by managing which vCPUs a guest application runs on; in this case, by binding the relevant processes in the guest to a single vCPU:

• If the threads of a multithreaded process in the guest frequently interact, bind the process to a single vCPU (i.e., a single qvm process vCPU thread).
• If several processes are communicating frequently, bind them to the same vCPU.

With these configurations, the application will behave as though it is running on a single-CPU system and won't issue IPIs that require hypervisor intervention to deliver.

on -C 1 foo

In some cases, if you are running a single-threaded application, it may even prove advantageous to run that application in its own guest OS running in a VM on its own dedicated pCPU.