CPU hotplug support in PowerKVM

ಆಗಷ್ಟ್ 9, 2016
Introduction
Pre requisites
Basic hotplug operation
More options
Driving via libvirt
KVM Forum 2016 presentation

Introduction

CPU hotplug is a technique or a feature that can be used to dynamically increase or decrease the number of CPUs available in the system. In order for the dynamically added CPUs to become available to the applications, CPU hotplug should be supported appropriately at multiple layers like in the firmware and operating system. This blog post mainly looks at the emerging support for CPU hotplug in KVM virtualization for PowerPC sPAPR virtual machines (pseries guests). In case of virtual machines, CPU hotplug is typically used to vertically scale up or scale down the guest CPUs at runtime based on the requirements. This feature is expected to be useful for supporting vertical scaling of PowerPC guests in KVM Cloud environments. Memory hotplug, which is also part of VM scale up requirement was discussed in my previous post.

CPU hotplug is supported for PowerPC sPAPR by means of device_add/device_del interfaces and not via the legacy cpu-add interface.

Pre requisites

CPU hotplug support for PowerPC sPAPR guests is now part of QEMU upstream and is expected to be available starting from QEMU-2.7 release. This implies that CPU hotplug support is available for pseries machine types starting from pseries-2.7.

In addition to QEMU and guest kernel (which btw has existed for a long time), some changes were done in PowerPC RAS tools also to support CPU hotplug. The minimum version of these packages needed in the guest are listed below:

Package Minimum required version
powerpc-utils 1.2.26
ppc64_diag 2.6.8
librtas 1.3.9

rtas_errd daemon which is provided by ppc64_diag package needs to be running in the guest for CPU hotplug to function correctly.

Basic hotplug operation

This section describes the steps to be followed by a QEMU user for CPU hotplug operation.

* CPU hotplug will be performed in core granularity on PowerPC. For eg. if the core has 8 threads, then one CPU hotplug operation will result in addition of 1 core with 8 CPU threads.

* Start a single thread guest with those command line options required for CPU hotplug.

qemu-system-ppc64 … –smp 2,cores=4,threads=1,maxcpus=4 -cpu POWER8E

-smp 2 will start the guest with 2 CPUs.
maxcpus=4 specifies that this guest can have a maximum of 4 CPUs, which implies 2 more CPUs can be added via CPU hotplug operations.
cores=4,threads=1 specifies the topology of the guest (4 cores each with 1 thread)
-cpu POWER8E specifies the guest CPU model

* Ensure that rtas_errd service is running inside the guest.

# ps aux | grep rtas
root      2518  0.3  0.0   6016  3968 ?        Ss   11:52   0:00 rtas_errd

[root@localhost ~]# lscpu
CPU(s):                2
On-line CPU(s) list:   0,1
Thread(s) per core:    1

* Connect to QEMU monitor from the host to discover hotpluggable CPUs and issue CPU hotplug command.

query-hotpluggable-cpus is the QMP interface while “info hotpluggable-cpus” is the HMP variant of the same.

(qemu) info hotpluggable-cpus
Hotpluggable CPUs:

type: “POWER8E-spapr-cpu-core”
vcpus_count: “1”
CPUInstance Properties:
core-id: “3”

type: “POWER8E-spapr-cpu-core”
vcpus_count: “1”
CPUInstance Properties:
core-id: “2”

type: “POWER8E-spapr-cpu-core”
vcpus_count: “1”
qom_path: “/machine/unattached/device[2]”
CPUInstance Properties:
core-id: “1”

type: “POWER8E-spapr-cpu-core”
vcpus_count: “1”
qom_path: “/machine/unattached/device[1]”
CPUInstance Properties:
core-id: “0”

type specifies the CPU core device type to be used with device_add command
vcpus_count specifies the number of threads the core has
core-id specifies the value for the core-id property that needs to be specified during device_add
qom_path if present implies that the CPU core is already plugged in, if absent it implies that the same can be hotplugged

To hot add a CPU,

(qemu) device_add POWER8E-spapr-cpu-core,id=core2,core-id=2

The added CPU core should be visible within the guest as well as in QEMU monitor

[root@localhost ~]# lscpu
CPU(s):                3
On-line CPU(s) list:   0-2
Thread(s) per core:    1

qemu) info hotpluggable-cpus
Hotpluggable CPUs:

type: “POWER8E-spapr-cpu-core”
vcpus_count: “1”
CPUInstance Properties:
core-id: “3”

  type: “POWER8E-spapr-cpu-core”
  vcpus_count: “1”
  qom_path: “/machine/peripheral/core2” <— hot added CPU
  CPUInstance Properties:
    core-id: “2”

type: “POWER8E-spapr-cpu-core”
vcpus_count: “1”
qom_path: “/machine/unattached/device[2]”
CPUInstance Properties:
core-id: “1”

type: “POWER8E-spapr-cpu-core”
vcpus_count: “1”
qom_path: “/machine/unattached/device[1]”
CPUInstance Properties:
core-id: “0”

To hot remove a CPU,

(qemu) device_del core2

More options

In this section a few more options and other possibilities with CPU hotplug are explored.

* NUMA guest – The NUMA semantics for CPU hotplug is likely to undergo a change, but this is how it works currently:

qemu-system-ppc64 … –smp 2,cores=4,threads=1,maxcpus=4 -numa node,nodeid=0,cpus=0,cpus=2 -numa node,nodeid=1,cpus=1,cpus=3

This guest will have 4 CPUs with existing CPU=0 and hotpluggable CPU=2 on NUMA node 0 while existing CPU=1 and hotpluggable CPU=3 on NUMA node 1. Currently the NUMA node affinity of the CPUs isn’t shown in the HMP/QMP queries.

The steps to discover, hot-add and hot-remove are same as shown in the previous example, just that the hot-added CPU will automatically end up on the NUMA node that was specified during boot using -numa cmdline.

The CPU added like below

(qemu) device_add POWER8E-spapr-cpu-core,id=core2,core-id=2

will end up in NUMA node 1.

[root@localhost ~]# numactl -H | grep cpus
node 0 cpus: 0 2
node 1 cpus: 1

* CPU compat option

The legacy way to specify CPU compat option is like this:

-cpu host,compat=power7

However with QEMU-2.7, compat option can be specified using -global option also so that the specified compat level is applied uniformly to boot CPUs as well as hotplugged CPUs.

-cpu host -global driver=host-powerpc64-cpu,property=compat,value=power7

The hotplug steps are similar to the previous example.

(qemu) device_add host-spapr-cpu-core,id=core2,core-id=2

Note that compat option will not be specified with every hotplugged CPU as it has already been set as global option which applies to hotplugged CPUs as well.

* Migration
If a guest that has undergone CPU hotplug operations needs to be migrated to another host, like any other device, CPU core device should be specified explicitly on the target side using  -device options.

If following hotplug operation is done at the source,

(qemu) device_add POWER8E-spapr-cpu-core,id=core2,core-id=2

then at the target host, the guest should be started with the following options:
qemu-system-ppc64 … -device POWER8E-spapr-cpu-core,id=core2,core-id=2

* Example of SMT4 guest

-smp 4,cores=4,threads=4,maxcpus=16

(qemu) info hotpluggable-cpus
Hotpluggable CPUs:

type: “host-spapr-cpu-core”
vcpus_count: “4”
CPUInstance Properties:
core-id: “12”

type: “host-spapr-cpu-core”
vcpus_count: “4”
CPUInstance Properties:
core-id: “8”

type: “host-spapr-cpu-core”
vcpus_count: “4”
CPUInstance Properties:
core-id: “4”

type: “host-spapr-cpu-core”
vcpus_count: “4”
qom_path: “/machine/unattached/device[1]”
CPUInstance Properties:
core-id: “0”

(qemu) device_add host-spapr-cpu-core,core-id=12,id=core4
In this example, two things are worth noting:

– The core-ids listed by the HMP query will be 0, 4, 8, … since this is an SMT4 guest.
– Any CPU can be hot added like we added core-id=12 here.

Driving via libvirt

Support for this device based CPU hotplug is still under development in libvirt.

KVM Forum 2016 presentation

The slides for CPU hotplug KVM Forum 2016 presentation can be found here and the presentation video can be seen here.


2015 in review

ಡಿಸೆಂಬರ್ 30, 2015

The WordPress.com stats helper monkeys prepared a 2015 annual report for this blog.

Here’s an excerpt:

The concert hall at the Sydney Opera House holds 2,700 people. This blog was viewed about 32,000 times in 2015. If it were a concert at Sydney Opera House, it would take about 12 sold-out performances for that many people to see it.

Click here to see the complete report.


2014 in review

ಡಿಸೆಂಬರ್ 30, 2014

The WordPress.com stats helper monkeys prepared a 2014 annual report for this blog.

Here’s an excerpt:

The concert hall at the Sydney Opera House holds 2,700 people. This blog was viewed about 31,000 times in 2014. If it were a concert at Sydney Opera House, it would take about 11 sold-out performances for that many people to see it.

Click here to see the complete report.


2013 in review

ಡಿಸೆಂಬರ್ 31, 2013

The WordPress.com stats helper monkeys prepared a 2013 annual report for this blog.

Here’s an excerpt:

The concert hall at the Sydney Opera House holds 2,700 people. This blog was viewed about 22,000 times in 2013. If it were a concert at Sydney Opera House, it would take about 8 sold-out performances for that many people to see it.

Click here to see the complete report.


English and dignity

ಡಿಸೆಂಬರ್ 11, 2011

An extract from Dr. Rajendra Prasad, India’s first President’s autobiography (written between 1942 and 45):

It was not that I was blind to the drawbacks of education as imparted in government institutions It was apparent to me that it was rotten. Because of the medium of instruction being a foreign language, it involved colossal waste of energy and time. It could not lead to that natural development of personality which can be so easily attained through proper education given in one’s own language. Obviously our students cannot have much time for thinking and meditating when all the while they are busy memorising the meaning of foreign words. For this reason alone the system is wasteful. It is not wrong to learn a foreign language. It is good to learn one, particularly today when knowledge of at least one European language has become almost obligatory. But to study a language in order to acquire a working knowledge of it is one thing and to learn every subject through its medium is another. So I have always expressed my opposition to making a foreign tongue the medium of instruction but not to learning it for its own sake. The British had their reasons for making Indians learn their language. They wanted a class of people who could work for and co-operate with them; they wanted to create a privileged class of Indians who would be like Englishmen in their thinking and outlook. They wanted another cheaper class which could do all the work in the offices in English. They thought that if they could get English-knowing men to assist them they would be saved the bother of learning the Indian languages themselves. It was this policy that formed the foundation of our system of education. Of course, the system did produce men of independent thought and action. But they formed an exception and the rule was clerks and office-workers who thought they had got the reward for their education.

More from same autobiography:

At one of the meetings in Orissa, Gandhiji was asked why he was so much against English education when men like Raja Ram Mohan Roy, Lokamanya Tilak and he himself, who were the products of that very education, had risen to great heights. Gandhiji, in his characteristic manner, replied: “I am a mere nonentity. But as for Lokamanya, he would have been a much greater man had he not been forced to study through the medium of English, a foreign tongue. Where do Raja Ram Mohan Roy and Lokamanya stand in comparison with Shankaracharya, Guru Nanak, Guru Gobind Singh and Kabir ? They had in their days none of the communication facilities that we enjoy today. Even then they were able to revolutionize the world of thought.”

Now to current generation: (From Deccan Herald of 11 Dec 2011)

“I decided to learn English at India International Centre in New Delhi, where a function was organized on November 14 to felicitate me. It was a happy occasion where Union Rural Development Minister Jairam Ramesh honoured me. But at the same time, I felt lonely and isolated at the function because, barring me, everyone else was conversing in English. I felt small,” rued Sushil. “It was then and there that I decided that if one has to earn dignity, one should have a good knowledge of English,” added Sushil, who has recently appointed the brand ambassador of Mahatma Gandhi National Rural Employment Guarantee Scheme (MGNREGS).

If you remember Sushil Kumar rose into fame recently after he won 1 crore rupees in a TV show.


ಸುಬ್ರಹ್ಮಣ್ಯದಲ್ಲಿ ಉರುಳು ಸೇವೆ ಮಾಡುವವರ ಬವಣೆ

ನವೆಂಬರ್ 25, 2009

ಶಿರಾಡಿ ಘಾಟಿಯಲ್ಲಿ ಚಾರಣ ಮುಗಿಸಿ ಗುಂಡ್ಯದಿಂದ ಕುಕ್ಕೆ ಸುಬ್ರಹ್ಮಣ್ಯಕ್ಕೆ ಷಷ್ಠಿಯ ಹಿಂದಿನ ದಿನ ಬರುತ್ತಿದೆವು. ಸುಬ್ರಹ್ಮಣ್ಯಕ್ಕೆ 2 km ಇರುವಾಗ ವಾಹನ ದಟ್ಟಣೆ ಹೆಚ್ಚಾಗಿ bus ನಿಂತೇ ಹೋಯಿತು. ಎಷ್ಟು ಹೊತ್ತಾದರೂ traffic clear ಆಗದಿದ್ದಾಗ, ಈ ಸಾದಾ ರಸ್ತೆಯಲ್ಲೂ ಒಂದು ಚಾರಣವಾಗಿಬಿಡಲಿ ಎಂದು ಬಸ್ ನಿಂದ ಇಳಿದು ನಡೆಯುತ್ತಾ ಸಾಗಿದಾಗ jam ಏಕಾಗಿದೆ ಎಂದು ಸ್ಪಷ್ಟವಾಯಿತು. ಕುಮಾರಧಾರದಲ್ಲಿ ಮಿಂದ ಭಕ್ತರು ದೇವಸ್ಥಾನದ ಕಡೆಗೆ ಉರುಳು ಸೇವೆ ಮಾಡುತ್ತಿದ್ದರು. ಇದೊಂದು ವಾರ್ಷಿಕ ಕಾರ್ಯಕ್ರಮವಾದರೂ, ದೇವಸ್ಥಾನದವರಾಗಲೀ, ಪೋಲಿಸ್ ನವರಾಗಲಿ ಹೆಚ್ಚೇನು ವ್ಯವಸ್ಥೆ ಮಾಡಿಕೊಂಡಂತೆ ಕಾಣಲ್ಲಿಲ. ಇದ್ದ ಒಂದೇ ವ್ಯವಸ್ಥೆ  ಎಂದರೆ,  ಭಕ್ತರು ಯಾವ ಬದಿಯಲ್ಲಿ ಉರುಳು ಹಾಕುತ್ತಿದ್ದರೋ ಆ ಬದಿಯ ವಾಹನ ಸಂಚಾರವನ್ನು ನಿಲ್ಲಿಸಲಾಗುತ್ತಿತ್ತು. ಯಾರ ನಿರ್ವಹಣೆಯೂ  ಇಲ್ಲದ ಕಾರಣ ಭಕ್ತರು ಎರಡು ಬದಿಯಲ್ಲೂ  ಉರುಳು ಹಾಕುತ್ತಿದ್ದರಿಂದ, ಅಲ್ಲಲ್ಲಿ ಎರಡು ಬದಿಯೂ ವಾಹನವನ್ನು ನಿಲ್ಲಿಸಲಾಗಿತ್ತು. ಇದೇನಿದು, ಇವರಿಂದ ಅನ್ಯರಿಗೆಲ್ಲಾ ಎಷ್ಟೊಂದು ತೊಂದರೆ ಎಂದೊಮ್ಮೆ  ಅನಿಸಿದರೂ, ನಂತರ ಇವರ ಕಷ್ಟ ಯಾರಿಗೂ ಬೇಡ ಅನಿಸಿತು.

  • ನದಿಯಿಂದ ದೇವಸ್ಥಾನ ಸಾಕಷ್ಟು ದೂರದಲ್ಲಿದೆ (2 km ಇರಬಹುದು)
  • ಉರುಳು ಹಾಕುವವರಿಗೆ ದಾರಿಯಲ್ಲಿ ಕಾದಿರಿಸಿದ ಜಾಗದ ವ್ಯವಸ್ಥೆ ಯಿಲ್ಲ. ಅಲ್ಲದೆ ದಾರಿಯೂ ಅಷ್ಟೇನು ವಿಸ್ತಾರವಿಲ್ಲ.
  • ವಾಹನಗಳೂ ಅಲ್ಲೇ ಸಂಚರಿಸುವುದರಿಂದ, ಉರುಳುವವರ ತಲೆ ಕೈ ಕಾಲು ವಾಹನದಡಿ ಬಂದರೂ ಆಶ್ಚರ್ಯವಿಲ್ಲ.
  • ಕಂಡಕಂಡಲ್ಲಿ ಉಗುಳಿರುವುದರಿಂದ, ಉರುಳು ಹಾಕುವವರು ಉಗುಳ ಮೇಲೆ ಉರುಳದೆ ವಿಧಿಯಿಲ್ಲ.
  • ಉರುಳುವಾಗ ಅನೇಕರು ವಾಂತಿ ಮಾಡುತ್ತಿದ್ದರು. ಅದನ್ನು ಅಲ್ಲೇ ಸ್ವಚ್ಛ ಮಾಡುವವರಿಲ್ಲದೆ, ಭಕ್ತರಿಗೆ ಅತ್ಯಂತ ಕಷ್ಟ ಆಗುತ್ತಿತ್ತು.

ದೇವಸ್ಥಾನ ತಲುಪಿದಾಗ, ಸ್ಥಳದ greatness ಬಗ್ಗೆ ಧ್ವನಿ ವರ್ಧಕದಲ್ಲಿ ಹೀಗೆ ಪ್ರಸಾರವಾಗುತ್ತಿತ್ತು – “… ದಿನಕ್ಕೆ ಸಾವಿರಾರು ಭಕ್ತರು ಬರುತ್ತಾರೆ… ಸಚಿನ್ ತೆಂಡೂಲ್ಕರ್ ಬಂದಿದ್ದರು… ಈ ಮಂತ್ರಿ ಮಹೋದಯರೂ ಬಂದಿದ್ದರು….”. ಇಷ್ಟೆಲ್ಲಾ ದೊಡ್ಡ ದೇವಸ್ಥಾನದಲ್ಲಿ, ಉರುಳು ಸೇವೆ ಮಾಡುವ ಭಕ್ತರಿಗೆ, ಇಷ್ಟೊಂದು ಕನಿಷ್ಠ ವ್ಯವಸ್ಥೆ ಯೇ ಎಂದು ಆಶ್ಚರ್ಯ ಆಗದೆ ಇರಲ್ಲಿಲ್ಲ. ಸಾಮಾನ್ಯವಾಗಿ ಜೀವನದಲ್ಲಿ ತೊಂದರೆಗಳಿರುವವರು ಉರುಳು ಸೇವೆ ಮಾಡುತ್ತಾರೆ, ಅಷ್ಟು  ಕಷ್ಟ ಸಹಿಸಿದವರಿಗೆ ಇದೇನು ಮಹಾ ಕಷ್ಟ ಏಂಬ ಧೋರಣೆಯೇ ?


ಷ ಕಾರದ ಉಚ್ಚಾರ ಅಸ್ಟು ಕಸ್ಟವೇ ?

ನವೆಂಬರ್ 7, 2009

ಈ ನಡುವೆ FM ರೇಡಿಯೋದಲ್ಲಿ ಕನ್ನಡ ಪದಗಳ ತಪ್ಪು  ಉಚ್ಚಾರಣೆ ಸಾಮಾನ್ಯವಾಗುತ್ತಾ ಬರುತ್ತಿದೆ. ಅಲ್ಪ ಪ್ರಾಣ ಮತ್ತು ಮಹಾ ಪ್ರಾಣಗಳನ್ನು ಸರಿಯಾಗಿ ಬಳಸುವುದು ಸ್ವಲ್ಪ  ಪ್ರಯಾಸಕರವೇ ಎನ್ನಬಹುದು (ಇಂಥ vs ಇಂತ ಇತ್ಯಾದಿ). ಆದರೆ ಕಷ್ಟ, ಇಷ್ಟ, ಎಷ್ಟು , ಅಷ್ಟು, ಮುಂತಾದ ಪದಗಳನ್ನು ಕಸ್ಟ, ಇಸ್ಟ, ಎಸ್ಟು, ಅಸ್ಟು ಎಂದು ಉಚ್ಚರಿಸುವುದು  ಸರ್ವೇಸಾಮಾನ್ಯವಾಗಿ ಹೋಗಿದೆ.  ಇತರೆ channel ಗಳಲ್ಲಿ ಹೋಗಲಿ ಆದರೆ ಕನ್ನಡವನ್ನು ಮುತುವರ್ಜಿ ಇಂದ ಬಳಸುವ channel ಗಳಲ್ಲೂ ಇದನ್ನು ಕೇಳಿದಾಗ ಷ ಕಾರವನ್ನು  ಉಚ್ಚರಿಸುವುದು ಅಷ್ಟೊಂದು ಕಷ್ಟವೇ ಅನಿಸುತ್ತದೆ. ಅಥವಾ ಇಲ್ಲಿ ಷ ಕಾರವಿದೆಯೆಂದೇ ನಮ್ಮ ಕೆಲ FM ಬಂಧುಗಳಿಗೆ ತಿಳಿಯದೆ ? ಸಾಮಾನ್ಯವಾಗಿ ಭಾಷೆಯು ಬದಲಾವಣೆಗಳನ್ನು ಕಾಣುತ್ತಾ ಮಾರ್ಪಾಡಾಗುತ್ತ  ಬೆಳೆಯುತ್ತದೆಂದು ಕೇಳಿದ್ದೇನೆ. ಇದನ್ನು ಬೆಳವಣಿಗೆ ಎನ್ನಬಹುದೇ ?