Running ARM Grub on U

At the Mini-debconfin Cambridge back in November there was an ARM Sprint (which Hectorwrote up as a Bits from ARMportersmail). During this there a brief discussion about usingGRUB as a standard bootloader,particularly for ARM server devices. This has the advantage ofproviding a more "normal" (which in practice means "x86 server-like")as well as flexible solution compared with the existing flash-kernel tool which is often used on ARM.

On ARMv7 devices this will more than likely involve chain loading fromthe U-Boot supplied by themanufacturer. For test and development it would be useful to be ableto set up a similar configuration usingQemu.

Cross-compilers

Although this can be built and run on an ARM system I am using a crosscompiler here. I'm usinggcc-linaro-arm-linux-gnueabihf-4.8-2013.08_linux fromLinaro, which can be downloaded from thelinaro-toolchain-binariespage on Launchpad. (It looks like 2013.10 is the latest availableright now, I can't see any reason why that wouldn't be fine).

Once the cross-compiler has been downloaded unpack it somewhere, Iwill refer to the resultinggcc-linaro-arm-linux-gnueabihf-4.8-2013.08_linux directory as$CROSSROOT.

Make sure $CROSSROOT/bin (which contains arm-linux-gnueabihf-gccetc) is in your $PATH.

Qemu

I'm using the version packaged in Jessie, which is 1.7.0+dfsg-2. Weneed both qemu-system-arm for running the final system andqemu-user to run some of the tools. I'd previously triedan older version of qemu (1.6.x?) and had some troubles, although theymay have been of my own making...

Das U-boot for Qemu

First thing to do is to build a suitable u-boot for use in the qemuemulated environment. Since we need to make some configuration changeswe need to build from scratch.

Start by cloning the upstream git tree:

$ git clone git://git.denx.de/u-boot.git$ cd u-boot

I am working on top of e03c76c30342 "powerpc/mpc85xx: UpdateCONFIG_SYS_FSL_TBCLK_DIV for T1040" dated Wed Dec 11 12:49:13 2013+0530.

We are going to use the Versatile Express Cortex-A9 u-boot but first weneed to enable some additional configuration options:

• CONFIG_API -- This enables the u-boot API which Grub uses toaccess the lowlevel services provided by u-boot. This means thatgrub doesn't need to contains dozens of platform specific flash,mmc, nand, network, console drivers etc and can be completelyplatform agnostic.
• CONFIG_SYS_MMC_MAX_DEVICE -- Setting CONFIG_API needs this.
• CONFIG_CMD_EXT2 -- Useful for accessing EXT2 formattedfilesystems. In this example I use a VFAT /boot for conveniencebut in a real system we would want to use EXT2 (or even somethingmore modern)).
• CONFIG_CMD_ECHO -- Just useful.

You can add all these to include/configs/vexpress_common.h:

#define CONFIG_API#define CONFIG_SYS_MMC_MAX_DEVICE   1#define CONFIG_CMD_EXT2#define CONFIG_CMD_ECHO

Or you can apply the patch which I sent upstream:

$ wget -O - http://patchwork.ozlabs.org/patch/304786/raw | git apply --index$ git commit -m "Additional options for grub-on-uboot"

Finally we can build u-boot:

$ make CROSS_COMPILE=arm-linux-gnueabihf- vexpress_ca9x4_config$ make CROSS_COMPILE=arm-linux-gnueabihf-

The result is a u-boot binary which we can load with qemu.

GRUB for ARM

Next we can build grub. Start by cloning the upstream git tree:

$ git clone git://git.sv.gnu.org/grub.git$ cd grub

By default grub is built for systems which have RAM at address0x00000000. However the Versatile Express platform which we aretargeting has RAM starting from 0x60000000 so we need to make acouple of modifications. First in grub-core/Makefile.core.def weneed to change arm_uboot_ldflags, from:

-Wl,-Ttext=0x08000000

to

-Wl,-Ttext=0x68000000

and second we need make a similar change to include/grub/offsets.h changingGRUB_KERNEL_ARM_UBOOT_LINK_ADDR from 0x08000000 to 0x68000000.

Now we are ready to build grub:

$ ./autogen.sh$ ./configure --host arm-linux-gnueabihf$ make

Now we need to build the final grub "kernel" image, normally thiswould be taken care of by grub-install but because we are crossbuilding grub we cannot use this and have to use grub-mkimagedirectly. However the version we have just built is for the ARM targetand not for host we are building things on. I've not yet figured outhow to build grub for ARM while building the tools for the host system(I'm sure it is possible somehow...). Luckily we can use qemu to runthe ARM binary:

$ cat load.cfgset prefix=(hd0)$ qemu-arm -r 3.11 -L $CROSSROOT/arm-linux-gnueabihf/libc     ./grub-mkimage -c load.cfg -O arm-uboot -o core.img -d grub-core/     fat ext2 probe terminal scsi ls linux elf msdospart normal help echo

Here we create load.cfg which is the setup script which will bebuilt into the grub kernel, our version just sets the root device sothat grub can find the rest of its configuration.

Then we use qemu-arm-static to invoke grub-mkimage. The "-r 3.11"option tells qemu to pretend to be a 3.11 kernel (which is required bythe libc used by our cross compiler, without this you will get afatal: kernel too old message) and "-L $CROSSROOT/..." tells itwhere to find the basic libraries, such as the dynamic linker (luckilygrub-mkimage doesn't need much in the way of libraries so we don'tneed a full cross library environment.

The grub-mkimage command passes in the load.cfg and requests anoutput kernel targeting arm-uboot, core.img is the output fileand the modules are in grub-core (because we didn't actually installgrub in the target system, normally these would be found in/boot/grub). Lastly we pass in a list of default modules to build intothe kernel, including filesystem drivers (fat, ext2), disk drivers(scsi), partition handling (msdos), loaders (linux, elf), themenu system (normal) and various other bits and bobs.

So after all the we now have our grub kernel in core.img.

Putting it all together

Before we can launch qemu we need to create various disk images.

Firstly we need some images for the 2 64M flash devices:

$ dd if=/dev/zero of=pflash0.img bs=1M count=64$ dd if=/dev/zero of=pflash1.img bs=1M count=64

We will initialise these later from the u-boot command line.

Secondly we need an image for the root filesystem on an MMCdevice. I'm using a FAT formatted image here simply for theconvenience of using mtools to update the images duringdevelopment.

$ dd if=/dev/zero of=mmc.img bs=1M count=16$ /sbin/mkfs.vfat mmc.img

Thirdly we need a kernel, device tree and grub configuration on ourroot filesystem. For the first two I extracted them from the standardarmmp kernel flavour package. I used thebackports.org version3.11-0.bpo.2-armmpversion and extracted /boot/vmlinuz-3.11-0.bpo.2-armmp as vmlinuzand /usr/lib/linux-image-3.11-0.bpo.2-armmp/vexpress-v2p-ca9.dtb asdtb. Then I hand coded a simple grub.cfg:

menuentry 'Linux' {        echo "Loading vmlinuz"        set root='hd0'        linux /vmlinuz console=ttyAMA0 ro debug        devicetree /dtb}

In a real system the kernel and dtb would be provided by the kernelpackages and grub.cfg would be generated by update-grub.

Now that we have all the bits we need copy them into the root ofmmc.img. Since we are using a FAT formatted image we can usemcopy from the mtools package.

$ mcopy -v -o -n -i mmc.img core.img dtb vmlinuz grub.cfg ::

Finally after all that we can run qemu passing it our u-boot binaryand the mmc and flash images and requesting a Cortex-A9 basedVersatile Express system with 1GB of RAM:

$ qemu-system-arm -M vexpress-a9 -kernel u-boot -m 1024m -sd mmc.img     -nographic -pflash pflash0.img -pflash pflash1.img

Then at the VExpress# prompt we can configure the default bootcmdto load grub and save the environment to the flash images. Thebackslash escapes (\$ and \;) should be included as written hereso that e.g. the variables are only evaluated when bootcmd isevaluated and not immediately when setting bootcmd and the bootmis set as part of bootcmd instead of executed immediately:

VExpress# setenv bootcmd fatload mmc 0:0 \${loadaddr} core.img \; bootm \${loadaddr}VExpress# saveenv

Now whenever we boot the system it will automatically load boot grubfrom the mmc and launch it. Grub in turn will load the Linux binaryand DTB and launch those. I haven't actually configure Linux with aroot filesystem here so it will eventually panic after failing tofind root.

Future work

The most pressing issue is the hard coded load address built in to thegrub kernel image. This is something which needs to be discussed withthe upstream grub maintainers as well as the Debian packagemaintainers.

Now that the ARM packages have hit Debian (in experimental in the2.02~beta2-1 package) I also plan to start looking at debian-installer integration as well as updating flash-kernel to setup the chain load of grub instead of loading akernel directly.