Introduction
CPU hotplug is a technique or a feature that can be used to dynamically increase or decrease the number of CPUs available in the system. In order for the dynamically added CPUs to become available to the applications, CPU hotplug should be supported appropriately at multiple layers like in the firmware and operating system. This blog post mainly looks at the emerging support for CPU hotplug in KVM virtualization for PowerPC sPAPR virtual machines (pseries guests). In case of virtual machines, CPU hotplug is typically used to vertically scale up or scale down the guest CPUs at runtime based on the requirements. This feature is expected to be useful for supporting vertical scaling of PowerPC guests in KVM Cloud environments. Memory hotplug, which is also part of VM scale up requirement was discussed in my previous post.
CPU hotplug is supported for PowerPC sPAPR by means of device_add/device_del interfaces and not via the legacy cpu-add interface.
Pre requisites
CPU hotplug support for PowerPC sPAPR guests is now part of QEMU upstream and is expected to be available starting from QEMU-2.7 release. This implies that CPU hotplug support is available for pseries machine types starting from pseries-2.7.
In addition to QEMU and guest kernel (which btw has existed for a long time), some changes were done in PowerPC RAS tools also to support CPU hotplug. The minimum version of these packages needed in the guest are listed below:
| Package | Minimum required version |
| powerpc-utils | 1.2.26 |
| ppc64_diag | 2.6.8 |
| librtas | 1.3.9 |
rtas_errd daemon which is provided by ppc64_diag package needs to be running in the guest for CPU hotplug to function correctly.
Basic hotplug operation
This section describes the steps to be followed by a QEMU user for CPU hotplug operation.
* CPU hotplug will be performed in core granularity on PowerPC. For eg. if the core has 8 threads, then one CPU hotplug operation will result in addition of 1 core with 8 CPU threads.
* Start a single thread guest with those command line options required for CPU hotplug.
qemu-system-ppc64 … –smp 2,cores=4,threads=1,maxcpus=4 -cpu POWER8E
-smp 2 will start the guest with 2 CPUs.
maxcpus=4 specifies that this guest can have a maximum of 4 CPUs, which implies 2 more CPUs can be added via CPU hotplug operations.
cores=4,threads=1 specifies the topology of the guest (4 cores each with 1 thread)
-cpu POWER8E specifies the guest CPU model
* Ensure that rtas_errd service is running inside the guest.
# ps aux | grep rtas
root 2518 0.3 0.0 6016 3968 ? Ss 11:52 0:00 rtas_errd[root@localhost ~]# lscpu
CPU(s): 2
On-line CPU(s) list: 0,1
Thread(s) per core: 1
* Connect to QEMU monitor from the host to discover hotpluggable CPUs and issue CPU hotplug command.
query-hotpluggable-cpus is the QMP interface while “info hotpluggable-cpus” is the HMP variant of the same.
(qemu) info hotpluggable-cpus
Hotpluggable CPUs:type: “POWER8E-spapr-cpu-core”
vcpus_count: “1”
CPUInstance Properties:
core-id: “3”type: “POWER8E-spapr-cpu-core”
vcpus_count: “1”
CPUInstance Properties:
core-id: “2”type: “POWER8E-spapr-cpu-core”
vcpus_count: “1”
qom_path: “/machine/unattached/device[2]”
CPUInstance Properties:
core-id: “1”type: “POWER8E-spapr-cpu-core”
vcpus_count: “1”
qom_path: “/machine/unattached/device[1]”
CPUInstance Properties:
core-id: “0”type specifies the CPU core device type to be used with device_add command
vcpus_count specifies the number of threads the core has
core-id specifies the value for the core-id property that needs to be specified during device_add
qom_path if present implies that the CPU core is already plugged in, if absent it implies that the same can be hotplugged
To hot add a CPU,
(qemu) device_add POWER8E-spapr-cpu-core,id=core2,core-id=2
The added CPU core should be visible within the guest as well as in QEMU monitor
[root@localhost ~]# lscpu
CPU(s): 3
On-line CPU(s) list: 0-2
Thread(s) per core: 1qemu) info hotpluggable-cpus
Hotpluggable CPUs:type: “POWER8E-spapr-cpu-core”
vcpus_count: “1”
CPUInstance Properties:
core-id: “3”type: “POWER8E-spapr-cpu-core”
vcpus_count: “1”
qom_path: “/machine/peripheral/core2” <— hot added CPU
CPUInstance Properties:
core-id: “2”type: “POWER8E-spapr-cpu-core”
vcpus_count: “1”
qom_path: “/machine/unattached/device[2]”
CPUInstance Properties:
core-id: “1”type: “POWER8E-spapr-cpu-core”
vcpus_count: “1”
qom_path: “/machine/unattached/device[1]”
CPUInstance Properties:
core-id: “0”
To hot remove a CPU,
(qemu) device_del core2
More options
In this section a few more options and other possibilities with CPU hotplug are explored.
* NUMA guest – The NUMA semantics for CPU hotplug is likely to undergo a change, but this is how it works currently:
qemu-system-ppc64 … –smp 2,cores=4,threads=1,maxcpus=4 -numa node,nodeid=0,cpus=0,cpus=2 -numa node,nodeid=1,cpus=1,cpus=3
This guest will have 4 CPUs with existing CPU=0 and hotpluggable CPU=2 on NUMA node 0 while existing CPU=1 and hotpluggable CPU=3 on NUMA node 1. Currently the NUMA node affinity of the CPUs isn’t shown in the HMP/QMP queries.
The steps to discover, hot-add and hot-remove are same as shown in the previous example, just that the hot-added CPU will automatically end up on the NUMA node that was specified during boot using -numa cmdline.
The CPU added like below
(qemu) device_add POWER8E-spapr-cpu-core,id=core2,core-id=2
will end up in NUMA node 1.
[root@localhost ~]# numactl -H | grep cpus
node 0 cpus: 0 2
node 1 cpus: 1
* CPU compat option
The legacy way to specify CPU compat option is like this:
-cpu host,compat=power7
However with QEMU-2.7, compat option can be specified using -global option also so that the specified compat level is applied uniformly to boot CPUs as well as hotplugged CPUs.
-cpu host -global driver=host-powerpc64-cpu,property=compat,value=power7
The hotplug steps are similar to the previous example.
(qemu) device_add host-spapr-cpu-core,id=core2,core-id=2
Note that compat option will not be specified with every hotplugged CPU as it has already been set as global option which applies to hotplugged CPUs as well.
* Migration
If a guest that has undergone CPU hotplug operations needs to be migrated to another host, like any other device, CPU core device should be specified explicitly on the target side using -device options.
If following hotplug operation is done at the source,
(qemu) device_add POWER8E-spapr-cpu-core,id=core2,core-id=2
then at the target host, the guest should be started with the following options:
qemu-system-ppc64 … -device POWER8E-spapr-cpu-core,id=core2,core-id=2
* Example of SMT4 guest
-smp 4,cores=4,threads=4,maxcpus=16
(qemu) info hotpluggable-cpus
Hotpluggable CPUs:type: “host-spapr-cpu-core”
vcpus_count: “4”
CPUInstance Properties:
core-id: “12”type: “host-spapr-cpu-core”
vcpus_count: “4”
CPUInstance Properties:
core-id: “8”type: “host-spapr-cpu-core”
vcpus_count: “4”
CPUInstance Properties:
core-id: “4”type: “host-spapr-cpu-core”
vcpus_count: “4”
qom_path: “/machine/unattached/device[1]”
CPUInstance Properties:
core-id: “0”(qemu) device_add host-spapr-cpu-core,core-id=12,id=core4
In this example, two things are worth noting:
– The core-ids listed by the HMP query will be 0, 4, 8, … since this is an SMT4 guest.
– Any CPU can be hot added like we added core-id=12 here.
Driving via libvirt
Support for this device based CPU hotplug is still under development in libvirt.
KVM Forum 2016 presentation
The slides for CPU hotplug KVM Forum 2016 presentation can be found here and the presentation video can be seen here.
Posted by Bharata 
Bharata B Rao's Blog 