Xen is one of the most actively developed and deployed open-source hypervisors. It has a significant market share in server virtualization market and actively developed for x86/amd64 platforms and sponsored by Intel. It has also found traction for ARM architecture and hardware based virtualization is well supported for both 32 and 64 bit ARM processors.
In this series of blog posts, we shall demonstrate porting and deployment of Xen hypervisor on an inexpensive ARM64 based Single Board Computer (SBC).
The complete bringup of Xen hypervisor on Orange Pi PC2 was performed in-house by Vadion Embedded Business Unit.
Xen Architecture
Xen employs a micro-kernel architecture which means that the core hypervisor kernel is minimal and mostly consists of kernel/scheduler, architecture specific code and some very essential drivers. Unlike monolithic kernels, most of the device drivers do not reside within the hypervisor.
Xen supports large number of platforms by employing a domain 0 (dom0) kernel that runs at a higher privileged level than guests. A dom0 kernel (Linux, BSD,…) provides most of the hardware device drivers as well as backend drivers for para-virtualization interfaces.
You can visit the Xen Project Wiki for more in-depth understanding of Xen.
About the Hardware Platform
We selected Orange Pi PC2 (http://www.orangepi.org/orangepipc2) for the following reasons.
- It is inexpensive (~$20) and can be easily ordered almost anywhere across the globe.
- It has a very small form factor. You can even carry it in your pocket.
- It contains an Allwinner H5 SoC (Quad Cortex-A53) i.e. an ARMv8 (64-bit) core with hardware virtualization extensions.
- This board contains an ethernet jack as well as two display interfaces (HDMI and Analog) which will allow us to demonstrate some interesting virtualization use-cases.
(Source: http://www.orangepi.org/orangepipc2/)
Setting up Hardware
Setting up Orange Pi P 2 is easy. You will need:
- A power adapter (better order with the board)
- A microSD card (16 GB should suffice)
- A 3.3V USB-Serial converter
You will need to connect a 3.3V USB-Serial converter to Debug TTL UART pins located between the power jack and HDMI connector. This will allow you to see console output from Xen, Linux guests.
There is already a good online tutorial regarding setting up UART on Allwinner boards and you can refer to it for further details:
Setting up Native Boot
Xen hypervisor, like Linux or any other kernel, does require a bootloader. We also need a functional Linux kernel for dom0.
Before moving ahead with Xen, it is essential to have a bootloader and native Linux kernel working on this platform.
We highly recommend using the Raspbian prebuilt images available at:
https://www.armbian.com/orange-pi-pc2/
Once you have downloaded one of the images, you can write the image on a micro SD card, place the card into the slot on board, do a power reset and see Linux booting on the console terminal.
In case you are wondering how to write the image on SD card, ‘dd’ is a very simple (and dangerous) utility for this purpose.
sudo dd if=<imagefilename>.img of=/dev/mmcblk0p1 bs=4M
Note: of= option should be set with the correct path of the SD card inserted into your built-in or external card-reader.
Note: An incorrect argument (e.g. your hard disk) can destroy your data and OS installation. So be careful!
Build Environment
All of the work was performed on Linux Mint 18 Sonya which is an Ubuntu (16.04) based distribution.
Install necessary packages such as build-essentials and aarch64 cross compiler before you start building Xen hypervisor code
sudo apt-get install build-essential gcc-5-aarch64-linux-gnu sudo apt-get build-dep xen
Getting Xen Code
Xen source code is hosted at xenbits git repositories. You can clone it using git
git clone http://xenbits.xen.org/gitweb/?p=xen.git
Patching Xen Code
Currently H5 requires patching Xen source code. Since we at Vadion were the first to run Xen on this platform, we made all necessary changes to port Xen on Allwinner H5.
We have already submitted these patches to the community and once accepted, users will be able to run the mainline xen source on this platform.
<Attachment: Patch 0001>
<Attachment: Patch 0002>
You can use ‘git apply’ command to apply these patches into your cloned Xen repository.
Building Xen Hypervisor
Change into the cloned Xen repository and execute build command:
cd xen ./configure make dist-xen XEN_TARGET_ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- CONFIG_EARLY_PRINTK=sun7i
Note that if you are still missing and dependencies, the ‘configure’ operation will fail throwing errors. You should be able to look at error messages and fix/install any required packages.
If everything goes right with build, the xen binary should be available in dist/install/boot directory.
Setting up Xen Boot
Now that we have xen binary, we can start setting up u-boot to boot xen instead of Linux which will then boot existing Linux kernel image as a dom0 guest.
The first thing that is needed is to update the boot.scr script. The updates will:
1- Load xen, dom0 kernel and init ramdisk image into memory
2- Take care of the initial memory map by loading the binaries to right places
Linux kernel 0x50000000 device tree 0x46000000 xen 0x48000000 ramdisk image 0x4E000000
3- Update device tree to add xen and dom0 boot arguments and to pass ramdisk image to Linux kernel
fdt chosen fdt set /chosen \#address-cells <1> fdt set /chosen \#size-cells <1> fdt set /chosen xen,xen-bootargs \"${xen_bootargs}\" fdt set /chosen xen,dom0-bootargs \"${dom0_bootargs}\" fdt set /chosen linux,initrd-start <${ramdisk_address}> fdt set /chosen linux,initrd-end <${ramdisk_end}> fdt mknod /chosen module@0 fdt set /chosen/module@0 compatible "xen,linux-zimage" "xen,multiboot-module" fdt set /chosen/module@0 reg <${kernel_addr_r} ${kernel_size}>
You can use the attached boot.cmd file which contains all of the necessary stuff
<Attachment: boot.cmd>
Generate boot.scr file from this script as:
mkimage -C none -A arm64 -T script -d boot.cmd boot.scr
Booting Xen
You will need to replace the boot.scr file in sd card /boot directory with the one created above.
Copy the xen binary file you built earlier to /boot directory. The boot.scr script assumes that hypervisor binary is named xen. So please rename from xen-4.*-unstable to xen.
Now place the SD card into the board, do a power reset and have a look at serial console terminal. You will see something like this:
Starting kernel ... - UART enabled - - CPU 00000000 booting - - Current EL 00000008 - - Xen starting at EL2 - - Zero BSS - - Setting up control registers - - Turning on paging - - Ready - ... (XEN) Xen version 4.10-unstable (awais@) (aarch64-linux-gnu-gcc (Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609) debug=y Wed Sep 20 13:44:38 PKT 2017 ... (XEN) Brought up 4 CPUs ... (XEN) *** LOADING DOMAIN 0 *** (XEN) Loading kernel from boot module @ 0000000050000000 (XEN) Loading ramdisk from boot module @ 000000004e000000 (XEN) Allocating 1:1 mappings totalling 256MB for dom0: (XEN) BANK[0] 0x00000060000000-0x00000070000000 (256MB) (XEN) Grant table range: 0x0000007f800000-0x0000007f867000 (XEN) handle / .... (XEN) Loading zImage from 0000000050000000 to 0000000060080000-0000000060b71808 (XEN) Loading dom0 initrd from 000000004e000000 to 0x0000000068200000-0x0000000068700000 (XEN) Allocating PPI 16 for event channel interrupt (XEN) Loading dom0 DTB to 0x0000000068000000-0x0000000068004440 ... (XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen) (XEN) Freed 292kB init memory. (XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER4 (XEN) d0v0: vGICD: unhandled word write 0xffffffff to ICACTIVER8
Congratulations! You have successfully booted xen on Orange Pi PC2.
You cannot see any output from dom0 Linux kernel because you are using a stock build that did not enable some important xen options.
In our next blog post, we shall demonstrate how to enable xen features in dom0 Linux kernel.
About Author
Awais Masood is an Associate Software Architect at Vadion. He has been working on embedded systems for more than 10 years and has an extensive experience in porting device drivers and OS kernels on ARM based hardware platforms.