IOMMU support in upstream qemu-system-sparc64

I didn't write about SPARC emulation lately because I didn't work on it. But Mark Cave-Ayland is still working hard, so there is a lot of new stuff. The sun4m emulation has a new graphic adapter: cg3. Also the old adapter (tcx) supports acceleration.

And now, something really cool for the sun4u emulation. As of today git master has a working IOMMU support in qemu-system-sparc64! This means that you don't need to follow all the steps in
Debian/sparc64 Wheezy under QEMU How-To. Just insert your image into -cdrom and voilà!
Well done, Mark!

Now, while using an IDE drive simplifies the installation process a lot, using virtio still makes sense. Quick performance test:

root@debian-wheezy:~#  dd if=/dev/vdb of=/dev/null bs=1024k count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 1.56761 s, 66.9 MB/s

root@debian-wheezy:~#  dd if=/dev/sda of=/dev/null bs=1024k count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.59107 s, 10.9 MB/s


Yes, IDE is slower, as the emulated kernel has to do more work, but still, the performance is pretty much ok.


Also, not every OS running on sun4u machines has support for virtio. From an undisclosed source I know that Mark is working on booting NetBSD/sparc64 under vanilla QEMU. Stay tuned. ;-)

Debian/sparc64 Wheezy under QEMU How-To

The steps to install Debian Wheezy RC1 / SPARC64 under QEMU.

Since the installation process is not obvious for the current QEMU and Debian versions, I gathered this How-To. Feel free to send any feedback.

Virtio performance and filesystems

Some time ago there was a comment to my post about booting Linux/sparc64 under QEMU, saying that virtio performance sucks.

At first I couldn't reproduce the problem. Tried a few caching strategies, but the I/O performance was not drastically affected by them. And then it came to my mind that the reason might be the guest file system.

Let's take a look:

$ dd if=/dev/zero of=disk-tmp bs=1024k count=300
300+0 records in
300+0 records out
314572800 bytes (315 MB) copied, 0.430909 s, 730 MB/s

  # so, the theoretical maximal raw performance of this partition on my disk is ~ 730 MB/s.

$ sparc64-softmmu/qemu-system-sparc64 -drive file=../boot-disk.qcow2,if=virtio,index=0 -drive file=disk-tmp,if=virtio,index=1
[...]
root@debian-wheezy:~#  mkfs.ext4dev /dev/vdb1
[...]
root@debian-wheezy:~#  mount /dev/vdb1  /mnt/
root@debian-wheezy:~#  dd if=/dev/zero of=/mnt/100M-file bs=1024k count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 3.2384 s, 32.4 MB/s

# while it seems to be 20 times slower than the host speed it is still a lot (the number 730MB/s is the theoretical limit, without performing regular syncs. QEMU does execute syncs to ensure the data integrity for the unlikely case of the crash.

Let's take a look on the read performance:

root@debian-wheezy:~#  umount /mnt/
root@debian-wheezy:~#  mount /dev/vdb1  /mnt/
[  582.673975] EXT4-fs (vdb1): mounted filesystem with ordered data mode. Opts: (null)
root@debian-wheezy:~#  dd if=/mnt/100M-file of=/dev/null bs=1024k
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 1.08423 s, 96.7 MB/s
root@debian-wheezy:~#

This looks even better. But what happens with the usual ext4?

root@debian-wheezy:~#  mkfs.ext4 /dev/vdb1
[...]
root@debian-wheezy:~#  mount /dev/vdb1  /mnt/
root@debian-wheezy:~#  dd if=/dev/zero of=/mnt/100M-file bs=1024k count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 3.54709 s, 29.6 MB/s

Ok, that's a bit less than the newer version. And what about ext3?

root@debian-wheezy:~#  mkfs.ext3 /dev/vdb1
root@debian-wheezy:~#  dd if=/dev/zero of=/mnt/100M-file bs=1024k count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 13.8049 s, 7.6 MB/s

Wow. That's slow. The ext3 partition access with the default options is more than 4 times slower than the ext4 partition access with the default options.

The short conclusion: if you need performance, don't use the ext3 file system with the default settings for your virtual machines.

/ happy hacking

Networking in Linux/sparc64 under qemu...

... is also possible. The bulit-in ne2k-pci network card doesn't work, but hey there is an even faster virtio-net alternative. At the OpenBIOS "ok" prompt, before the "boot" command in the previous post, type

cd /pci@1fe,0/pci1af4,1000
0 encode-int " interrupts" property
device-end

Will send this patch upstream as soon as we clarify whether "0" is allowed for the "interrupts" property.

Booting Linux/sparc64 on todays OpenBIOS

Historical day for everyone interested in the vanilla qemu-system-sparc64 emulator. For the first time it can boot Linux!

Although the patches I sent upstream are  fixing CPU and IOMMU, there is still one missing piece: cmd646 IDE. Luckily it's not a showstopper at all: qemu-sparc64 is a PCI machine, which means one could use the fast virtio instead! Now, this is the command line:

$  sparc64-softmmu/qemu-system-sparc64 -m 256 -nographic -prom-env 'auto-boot?=false' -kernel /path/to/kernel/image -drive file=/path/to/debian-disk,if=virtio,index=0 -append 'root=/dev/vda1 init=/bin/sh'

Some time ago, I used Forth to workaround missing qemu features  to get OBP working.
Now it's pretty similar: OpenBIOS doesn't have all the properties necessary for Linux to get the interrupt mapping.

So at the ok command prompt where you'd get after the command above, type:
 
cd /
1 encode-int " #interrupt-cells" property
cd /pci@1fe,0/ebus
000001fe encode-int 020003f8 encode-int encode+ 1 encode-int encode+
ffe29140 encode-int encode+ 2b encode-int encode+ " interrupt-map" property
000001ff encode-int ffffffff encode-int encode+ 00000003 encode-int encode+
" interrupt-map-mask" property

cd /pci@1fe,0/ebus@3/su
1 encode-int " interrupts" property
cd /pci@1fe,0/pci-ata@5
0 encode-int " interrupts" property

cd /pci@1fe,0/pci1af4,1001
0 encode-int " interrupts" property
device-end
boot


The only magical constant above is actually "ffe29140" - the address of the root pci node in the OpenBIOS device hierarchy. Although it the same in all the recent builds, in theory it could be moved somewhere else one day. But I guess till that day OpenBIOS will have the missing properties... ;-)

Update:

Oh, and a few words to /path/to/kernel/image and /path/to/debian-disk:

I found no Linux/sparc64 distribution which has a built in virtio driver. This makes installing from a  CD/DVD image not possible. The solution is build your own kernel with the virtio driver compiled in and put it at /path/to/kernel/image.

As for the disk, the user space utilities from the sparc32 world can be used in the sparc64 world as is. So, you can install Debian (or your favorite sparc32 distribution) in the /path/to/debian-disk, using qemu-system-sparc (no 64 at the end), and then use it with qemu-system-sparc64 as described above.

If you know a Linux/sparc64 distribution with the virtio support, please let me know.

/Happy hacking