In our previous blog post, we discussed how to build and execute xen hypervisor on Orange Pi PC 2. We used a pre-built stock Linux image as dom0 kernel.
In this post, we shall demonstrate how to properly configure dom0 kernel for xen.
If you remember from our previous post, we saw output from Xen hypervisor suggesting that it had loaded and executed dom0 kernel but never saw any output from Linux kernel as seen in a native boot. The problem is that the serial terminal that we are monitoring is being used by Xen therefore Linux cannot use this UART interface.
How to we get around this problem? It is one of the resource partitioning problems encountered in setting up virtualized systems.
One platforms where multiple UART connections are available it might be possible to let hypervisor use one console while guests use the other in passthrough mode.
Such a passthrough is not possible on Orange Pi PC2 since we only have one UART connection on this platform.
Even if there were multiple terminals, Xen blacklists all serial devices from guests on Allwinner chips due to a security concern. The memory space for all UART controllers share the same page. So if one UART memory is mapped to a guest, it would be possible for it to access the memory assigned to hypervisor. Not a good idea!
The solution lies in using Hypervisor Virtual Console (HVC). Hypervisor controls the physical console and provides a virtual console interface for guests. Guests can dump their output into this console. Users can switch between xen and guest output through simple key combinations.
Using HVC requires a driver within the dom0 guest kernel. Linux contains this driver and we actually instructed it to use this console through command line arguments. However, HVC was not enabled in the prebuilt Linux image that we used earlier. So we’ll need to rebuild the Linux kernel with HVC console.
We need to enable hvc0 and certain other xen related options within Linux dom0 kernel to get it working properly. It would be a good idea to glance through a relevant Xen Wiki article:
https://wiki.xen.org/wiki/Mainline_Linux_Kernel_Configs
Basically for ARM kernel, the following configuration options are relevant
CONFIG_XEN CONFIG_PARAVIRT
Enabling these two options from menuconfig should be sufficient to enable other relevant domU options below:
CONFIG_HVC_DRIVER CONFIG_HVC_IRQ CONFIG_HVC_XEN CONFIG_XEN_BLKDEV_FRONTEND CONFIG_HVC_XEN_FRONTEND CONFIG_XEN_FBDEV_FRONTEND CONFIG_XEN_NETDEV_FRONTEND CONFIG_INPUT_XEN_KBDDEV_FRONTEND CONFIG_XEN_XENBUS_FRONTEND CONFIG_XEN_GRANT_DEV_ALLOC
For ARM kernel, the dom0 config options are not visible from menuconfig and you would be required to manually add these options to .config file.
CONFIG_XEN_DOM0 CONFIG_XEN_BLKDEV_BACKEND CONFIG_XEN_NETDEV_BACKEND CONFIG_XEN_BALLOON CONFIG_XEN_SCRUB_PAGES CONFIG_XEN_DEV_EVTCHN CONFIG_XEN_BACKEND CONFIG_XENFS CONFIG_XEN_COMPAT_XENFS CONFIG_XEN_SYS_HYPERVISOR CONFIG_XEN_GNTDEV
There are multiple options to setup a Linux kernel build.
1. You can setup an Armbian build system by checking out:
https://github.com/armbian/build
This will allow you to recompile the same kernel version that you used previously from armbian pre-built image.
2. Use the cutting-edge latest kernel source from Linus’ tree. We’ll demonstrate how to build this version of the kernel.
You can checkout the Linus tree as:
git clone https://github.com/torvalds/linux.git
Now reconfigure and build the tree as
cd linux make ARCH=arm64 defconfig cp <path to config file>/linux-sun50i-dev.config .config yes "" | make oldconfig make Image ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-
Once build completes, you should have a kernel binary at arch/arm64/boot/Image
Assuming you already have a functional SD card with xen as described in previous blog post.
Plug the SD card into your machine.
Replace the Image file in sdcard /boot directory with the one you just generated from Linus tree.
Plug the card back into Orange Pi PC2. Power on and this time you should see xen as well as dom0 Linux output on your console:
Starting kernel ... - UART enabled - - CPU 00000000 booting - - Current EL 00000008 - - Xen starting at EL2 - - Zero BSS - - Setting up control registers - - Turning on paging - - Ready - ... (XEN) Xen version 4.10-unstable (awais@) (aarch64-linux-gnu-gcc (Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609) debug=y Wed Sep 20 13:44:38 PKT 2017 ... (XEN) *** LOADING DOMAIN 0 *** (XEN) Loading kernel from boot module @ 0000000050000000 (XEN) Loading ramdisk from boot module @ 000000004e000000 (XEN) Allocating 1:1 mappings totalling 256MB for dom0: (XEN) BANK[0] 0x00000060000000-0x00000070000000 (256MB) (XEN) Grant table range: 0x0000007f800000-0x0000007f867000 (XEN) handle / .... [1.325960] sun4i-usb-phy 1c19400.phy: Couldn't request ID GPIO [1.327770] sun8i-h3-r-pinctrl 1f02c00.pinctrl: initialized sunXi PIO driver [1.330891] sun50i-h5-pinctrl 1c20800.pinctrl: initialized sunXi PIO driver [1.334072] xen:xen_evtchn: Event-channel device installed ... [1.738743] EXT4-fs (mmcblk0p1): mounted filesystem with writeback data mode. Opts: (null) [1.738843] VFS: Mounted root (ext4 filesystem) readonly on device 179:1. ... [2.182869] systemd[1]: Detected virtualization xen. [2.182915] systemd[1]: Detected architecture arm64. ... Linux hvc0 login :
You can now log into the dom0 Linux and should be able to do all the usual stuff at the terminal.
We have completed the second step of setting up a Xen system. In our next post, we shall explain setup of Xen userland which will subsequently allow us to run and manage domU guests.
Awais Masood is an Associate Software Architect at Vadion. He has been working on embedded systems for more than 10 years and has an extensive experience in porting device drivers and OS kernels on ARM based hardware platforms.