Back to Blog Back to Career

Xen Hypervisor : Setting up dom0 Linux kernel on OrangePi PC 2 (Part 2)

Categories :
Author : Blogger
Date : 30-10-2017
, , , ,
No Comments


In our previous blog post, we discussed how to build and execute xen hypervisor on Orange Pi PC 2. We used a pre-built stock Linux image as dom0 kernel.

In this post, we shall demonstrate how to properly configure dom0 kernel for xen.

Console Output

If you remember from our previous post, we saw output from Xen hypervisor suggesting that it had loaded and executed dom0 kernel but never saw any output from Linux kernel as seen in a native boot. The problem is that the serial terminal that we are monitoring is being used by Xen therefore Linux cannot use this UART interface.

How to we get around this problem? It is one of the resource partitioning problems encountered in setting up virtualized systems.  

One platforms where multiple UART connections are available it might be possible to let hypervisor use one console while guests use the other in passthrough mode.

Such a passthrough is not possible on Orange Pi PC2 since we only have one UART connection on this platform.

Even if there were multiple terminals, Xen blacklists all serial devices from guests on Allwinner chips due to a security concern. The memory space for all UART controllers share the same page. So if one UART memory is mapped to a guest, it would be possible for it to access the memory assigned to hypervisor. Not a good idea!

 

Hypervisor Virtual Console

The solution lies in using Hypervisor Virtual Console (HVC). Hypervisor controls the physical console and provides a virtual console interface for guests. Guests can dump their output into this console. Users can switch between xen and guest output through simple key combinations.

Using HVC requires a driver within the dom0 guest kernel. Linux contains this driver and we actually instructed it to use this console through command line arguments. However, HVC was not enabled in the prebuilt Linux image that we used earlier. So we’ll need to rebuild the Linux kernel with HVC console.

 

Linux Dom0 Configuration

We need to enable hvc0 and certain other xen related options within Linux dom0 kernel to get it working properly. It would be a good idea to glance through a relevant Xen Wiki article:

https://wiki.xen.org/wiki/Mainline_Linux_Kernel_Configs

Basically for ARM kernel, the following configuration options are relevant

For domU

CONFIG_XEN
CONFIG_PARAVIRT

Enabling these two options from menuconfig should be sufficient to enable other relevant domU options below:

CONFIG_HVC_DRIVER
CONFIG_HVC_IRQ
CONFIG_HVC_XEN
CONFIG_XEN_BLKDEV_FRONTEND
CONFIG_HVC_XEN_FRONTEND
CONFIG_XEN_FBDEV_FRONTEND
CONFIG_XEN_NETDEV_FRONTEND
CONFIG_INPUT_XEN_KBDDEV_FRONTEND
CONFIG_XEN_XENBUS_FRONTEND
CONFIG_XEN_GRANT_DEV_ALLOC

For dom0

For ARM kernel, the dom0 config options are not visible from menuconfig and you would be required to manually add these options to .config file.

CONFIG_XEN_DOM0
CONFIG_XEN_BLKDEV_BACKEND
CONFIG_XEN_NETDEV_BACKEND
CONFIG_XEN_BALLOON
CONFIG_XEN_SCRUB_PAGES
CONFIG_XEN_DEV_EVTCHN
CONFIG_XEN_BACKEND
CONFIG_XENFS
CONFIG_XEN_COMPAT_XENFS
CONFIG_XEN_SYS_HYPERVISOR
CONFIG_XEN_GNTDEV

Building Linux Kernel

There are multiple options to setup a Linux kernel build.

1. You can setup an Armbian build system by checking out:

https://github.com/armbian/build

This will allow you to recompile the same kernel version that you used previously from armbian pre-built image.

2. Use the cutting-edge latest kernel source from Linus’ tree. We’ll demonstrate how to build this version of the kernel.

You can checkout the Linus tree as:

git clone  https://github.com/torvalds/linux.git

Now reconfigure and build the tree as

cd linux
make ARCH=arm64 defconfig
cp <path to config file>/linux-sun50i-dev.config .config

yes "" | make oldconfig

make Image ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-

Once build completes, you should have a kernel binary at arch/arm64/boot/Image

Running dom0 Kernel

Assuming you already have a functional SD card with xen as described in previous blog post.

Plug the SD card into your machine.

Replace the Image file in sdcard /boot directory with the one you just generated from Linus tree.

Plug the card back into Orange Pi PC2. Power on and this time you should see xen as well as dom0 Linux output on your console:

Starting kernel ...
- UART enabled -
- CPU 00000000 booting -
- Current EL 00000008 -
- Xen starting at EL2 -
- Zero BSS -
- Setting up control registers -
- Turning on paging -
- Ready -
...
(XEN) Xen version 4.10-unstable (awais@) (aarch64-linux-gnu-gcc (Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609) debug=y  Wed Sep 20 13:44:38 PKT 2017
...
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Loading kernel from boot module @ 0000000050000000
(XEN) Loading ramdisk from boot module @ 000000004e000000
(XEN) Allocating 1:1 mappings totalling 256MB for dom0:
(XEN) BANK[0] 0x00000060000000-0x00000070000000 (256MB)
(XEN) Grant table range: 0x0000007f800000-0x0000007f867000
(XEN) handle /
....

[1.325960] sun4i-usb-phy 1c19400.phy: Couldn't request ID GPIO
[1.327770] sun8i-h3-r-pinctrl 1f02c00.pinctrl: initialized sunXi PIO driver
[1.330891] sun50i-h5-pinctrl 1c20800.pinctrl: initialized sunXi PIO driver
[1.334072] xen:xen_evtchn: Event-channel device installed
...
[1.738743] EXT4-fs (mmcblk0p1): mounted filesystem with writeback data mode. Opts: (null)
[1.738843] VFS: Mounted root (ext4 filesystem) readonly on device 179:1.
...
[2.182869] systemd[1]: Detected virtualization xen.
[2.182915] systemd[1]: Detected architecture arm64.
...

Linux hvc0

login :

 

You can now log into the dom0 Linux and should be able to do all the usual stuff at the terminal.

We have completed the second step of setting up a Xen system. In our next post, we shall explain setup of Xen userland which will subsequently allow us to run and manage domU guests.

 

About Author

Awais Masood is an Associate Software Architect at Vadion. He has been working on embedded systems for more than 10 years and has an extensive experience in porting device drivers and OS kernels on ARM based hardware platforms.