Saturday, April 16, 2016

Bad, bad cafe! (0xbaddcafe)

Debugging Solaris 10 boot I saw something interesting in an exception trace:

143368: Unaligned Memory Access (v=0034)
pc: 00000000f02421f8  npc: 00000000f02421fc
%g0-3: 0000000000000000 0000000000000001 0000000000000000 00000000edd00620
%g4-7: baddcafebaddcafe 0000000000002e7f 0000000000000000 00000000f0243de8 
%o0-3: 00000000018d46e0 0000000000000001 00000000ede8e7e1 0000000001213010

And indeed, this is not a random pattern. It's a helping hand from the great, wise Solaris engineers who cared to help the ancestors in finding problems with hardware and kernel modules:

#define  KMEM_UNINITIALIZED_PATTERN      0xbaddcafebaddcafeULL

Looking at the OpenSolaris sources and Solaris documentation, there are more such helping patterns:

Uninitialized Data: 0xbaddcafe
Redzone: 0xfeedface
Freed Buffer Checking: 0xdeadbeef

They are described in the "Detecting Memory Corruption" chapter of Solaris Modular Debugger Guide, but did actually appear long before mdb.

Sunday, April 10, 2016

FreeBSD-10.3/sparc64 under QEMU

I made a wrong statement on the debian-sparc mailing list, saying that the upstream qemu-system-sparc64 can already boot FreeBSD. As it turned out I spent too little time with the upstream QEMU. This made me feel obliged to fix it. This is how it's going to look in the QEMU 2.6.0, if my patches get accepted:

$ qemu-system-sparc64 -nographic -m 1024 -boot d -cdrom FreeBSD-10.3-RELEASE-sparc64-bootonly.iso
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.3-RELEASE #0 r297264: Fri Mar 25 06:26:08 UTC 2016 sparc64
gcc version 4.2.1 20070831 patched [FreeBSD]

Console type [vt100]: xterm

When finished, type 'exit' to return to the installer.
# uname -a
FreeBSD  10.3-RELEASE FreeBSD 10.3-RELEASE #0 r297264: Fri Mar 25 06:26:08 UTC 2016  sparc64
# ls
.cshrc          HARDWARE.HTM    bin             libexec         sbin
.profile        HARDWARE.TXT    boot            media           sys
.rr_moved       README.HTM      dev             mnt             tmp
COPYRIGHT       README.TXT      docbook.css     proc            usr
ERRATA.HTM      RELNOTES.HTM    etc             rescue          var
ERRATA.TXT      RELNOTES.TXT    lib             root
So, after all my statement should be correct. :-)

Saturday, February 6, 2016

Yo dawg, I heard you like debugging

Here is the story: my sun4v can boot OBP, but booting Solaris 10 hangs with no error messages. Ok, being there, done that. Let’s start the Solaris kernel with a debugger. I really liked kadb for debugging early boot stuff, but the Solaris 10 image supplied with the OpenSPARC project has only its successor - kmdb.  Well, kmdb is indeed more advanced, but it’s also quite bigger than its predecessor.  Which might be (or might be not) the reason for it failing to boot:

Sun Fire T2000, No Keyboard
Copyright 2005 Sun Microsystems, Inc.  All rights reserved.
OpenBoot 4.20.0, 256 MB memory available, Serial #1122867.
[mo23723 obp4.20.0 #0]
Ethernet address 0:80:3:de:ad:3, Host ID: 80112233.
ok boot -kdv
Boot device: /virtual-devices/disk@0  File and args: -kdv
Loading ufs-file-system package 1.4 04 Aug 1995 13:02:54.
FCode UFS Reader 1.12 00/07/17 15:48:16.
Loading: /platform/SUNW,Sun-Fire-T2000/ufsboot
Loading: /platform/sun4v/ufsboot
The boot filesystem is logging.
The ufs log is empty and will not be used.
Size: 0x76e40+0x1c872+0x3123a Bytes
module /platform/sun4v/kernel/sparcv9/unix: text at [0x1000000, 0x1076e3f] data at 0x1800000
module misc/sparcv9/krtld: text at [0x1076e40, 0x108f737] data at 0x184dab0
module /platform/sun4v/kernel/sparcv9/genunix: text at [0x108f738, 0x11dd437] data at 0x18531c0
module /platform/sun4v/kernel/misc/sparcv9/platmod: text at [0x11dd438, 0x11dd43f] data at 0x18a4be0
module /platform/sun4v/kernel/cpu/sparcv9/SUNW,UltraSPARC-T1: text at [0x11dd440, 0x11e06ff] data at 0x18a5300
Loading kmdb...
module /platform/sun4v/kernel/misc/sparcv9/kmdbmod: text at [0x11e0700, 0x124b2bf] data at 0x18b4da0
module /kernel/misc/sparcv9/ctf: text at [0x124b2c0, 0x1252d97] data at 0x18d6ed0
module /kernel/misc/sparcv9/zmod: text at [0x1252d98, 0x1257a67] data at 0x18d7af8
failed to decompress CTF data for unix: File data structure corruption detected
failed to decompress CTF data for genunix: String name offset is corrupt
failed to decompress CTF data for ctf: File data structure corruption detected
failed to decompress CTF data for zmod: File data structure corruption detected

What is the solution? Connect another debugger (gdb) to QEMU and debug the Solaris debugger (kmdb). Sounds reasonable, right?  In the next step I found a place where memory is already corrupted. This has been easy: as you see, the Solaris engineers put some sanity checks in the CTF code. Well done, Sun guys!
Finding the place where it gets corrupted is a bit harder: gdb has no watch-points on the physical memory, supporting only virtual memory watch-points. The solution is indeed starting the QEMU process itself in a debugger. At this point it gets slightly insane:

I put a debugger (kmdb) in a debugger (gdb x86-64) and connected it to a debugger (gdb sparc-v9) so I can debug while I’m debugging a debugger.

Saturday, January 30, 2016

sun4v in QEMU

Back in 2012 I played with sun4v emulation in QEMU, using it mostly instead of pain killers to get some distraction from a broken leg. The project was considered to be a toy, since I hadn’t expected to get it far enough to be useful for anything. I got it up to the OBP ok prompt, so it’s been sort of already useful at least for playing with post-sun4u OpenBoot and Forth.

Now I’m considering tidying up the code and submitting it upstream.  Tell you what.  Cleaning up the old code is pain. The usual problem with the quick and dirty code that you write once intending to throw it away immediately is that for whatever reasons this code is not thrown away in the 99% of cases. Instead it finds its way into production systems where it lives years and years.

So, a note to myself and the two other guys reading this blog: use a version control system (preferably git :-) ) for any project lasting more than 8 hours. Do it regardless whether you think you never going to need it.  I used to think a week is a good threshold, but even one week is way too much (and if you worked that week something like 16 hours a day, sorting out the mess you created would require some weeks).

Anyway, I’m back to my sun4v experiments.  How many weekends it’ll take to get it into a good shape? Let’s see.

Stay tuned.

Saturday, October 10, 2015

Oracle strikes back!

The Oracle Enterprise Linux for sun4v has just been released!

Based on RHEL 6 and kernel 4.1.

How cool is that?!?

Saturday, June 27, 2015

The number of Debian/SPARC64 users doubled overnight

That's right, I've upgraded my virtual wheezy image to Debian/sparc64 Sid. So I'm the second user who has installed and run Debian popularity contest:

Don't know who is the first user, and don't know why the submission on ~ 29.04.2014 on the graph above hasn't doubled the number of users back then.

Update: Debian/SPARC64 is mostly working, but there are some caveats:
Link time optimization with "gold" linker produces broken binaries. Unfortunately this hits systemd and udev, so those have to be re-built manually. I've submitted a bug for the upstream binutils.

In the mean time if anyone needs a working systemd-udevd, you can ask me. There is also a report that the qt build is broken, but I don't use it - only need a working console.

Saturday, November 22, 2014

Solaris 9 under OpenBIOS

Together with Mark Cave-Ayland fixed a bug in OpenBIOS preventing Solaris 9 boot in a single user mode on sun4m. As it turned out, Sun relied on some undocumented behavior while walking though a device tree: calling the "child" function with a null argument is expected to produce the same result as calling it with a pointer to the root node.

So, in the next OpenBIOS release, it should be possible to boot Solaris 9 just like the versions 5.7 and 8 with just '-s' switch instead of '-b'.
Unfortunately my schedule rarely fits the QEMU-Release schedule, so this fix probably won't make it into the upcoming QEMU 2.2. release because of the code freeze. But for those, who are starting their day with 'git pull' the release schedule doesn't matter. ;-)

Stay tuned...

Wednesday, September 3, 2014

Open Firmware for qemu-system-ppc -M prep

I'm glad to announce the result of some weekends of work: a new firmware which is not just open, but is also powerful (it runs on Power CPUs), free (the individual files are licensed under MIT and BSD licenses) and of course legendary - it's based on the original PReP firmware from Mitch Bradley's FirmWorks company.

That's how it happened: Mitch licensed the files from the original PReP firmware under the MIT license, I adjusted them to the current OFW tree and added some build files. Then I gathered a small firmware for QEMU PReP machine which used a Cirrus Logic graphics adapter (need a one line change in QEMU code to enable it). Then I added a driver for QEMU "standard" VGA Adapter (also known as Bochs Graphic Adapter) and this night Mitch has committed it together with my atapi fixes as svn.3738 here:

What's in the tree? The original 1994-2000 stuff landed in the /cpu/ppc/prep directory. The build files for generic PReP firmware on top of the generic PPC firmware are in the /cpu/ppc/prep/build folder. And QEMU-specific stuff is in /cpu/ppc/prep/qemu folder. For those who don't want to compile it, but would like to try it straight away I've built two binaries: qprepofw-vga-svn-3738.rom which uses QEMU in graphics mode, and qprepofw-serial-svn-3738.rom for serial console (in the current QEMU version I found no way for guest to see if qemu-system-ppc -M prep is launched with an additional -nographic option).

How to use it:

qemu-system-ppc -M prep  -bios qprepofw-vga-svn-3738.rom  -cpu 603 -hda /path/hdd-image -fda /path/floppy -cdrom path/cd


qemu-system-ppc -M prep -bios qprepofw-serial-svn-3738.rom  -cpu 603 -hda /path/hdd-image -fda /path/floppy -cdrom path/cd -nographic

And then

boot disk
boot cdrom
boot floppy

The advantages over OpenHackWare (default QEMU firmware for PReP target) and OpenBIOS:
  • can boot from floppies and cdrom images. It's possible to boot for instance NetBSD-6.1.3-prep.iso. Of Course, It Runs NetBSD!™
  • supports QEMU's -vga std, -vga cirrus graphic adapters and serial console
  • is written in Forth (OpenHackWare is not), has command history, Forth syntax highlighting and code completion.
  • uses the original layout for NVRAM
The drawbacks:
  • doesn't detect the amount of RAM present and the screen resolution. For now I've just hard-coded 128MiB and 1024x768. Is a trivial thing to support once QEMU chooses the way this information is passed to the firmware (OHW and OpenBIOS use different mechanisms, there are patches which are removing the former one and adding the later, but they are not in upstream yet)
  • can't boot from Apple Partition Boot Code. The original RS/6000 firmware was probably not able to do it either, but OpenBIOS can do it and maybe it's a nice feature for testing (not sure if it is important yet: on one hand there are more PPC distributions for Macs, on the other hand, Macs had slightly different hardware and firmware than PReP machines)
So, digital archaeologists,  have fun and please report if you could do with OHW something which OFW can't.

Stay tuned...

Saturday, August 30, 2014

Playing with Open Firmware

Made a couple of small improvements in Open Firmware and sent the patches to Mitch Bradley. One fix is probably not relevant for the most of users: it has to do with handling fdisk/MBR partitions on CDs. Another one may be more interesting for those who use CD/DVD drives on OLPC.
I was hoping to do something useful, but it turned out OLPC don't have a built-in cdrom drive. Still there can be one or two users who have external drives and would like to boot from them.

I didn't realize till recent times that as a part of his work on the OLPC Project, Mitch released the original Open Firmware base for ARM, MIPS, PPC and x86 machines. It has some advantages over OpenBIOS: Forth syntax highlighting,  word completion, more comfortable debugging and an emulator of Power CPUs to simplify cross-platform building.

It is distributed  under MIT Open-Source License, which may be more suitable than GPL for some projects.

May the Forth be with you. :-)

Saturday, August 9, 2014

Upstream QEMU can run NetBSD/sparc64

Mark did a good job fixing OpenBIOS properties and cmd646 emulation in QEMU, so NetBSD started to get up to the user space. Then I sent my patch for emulation of short load instructions upstream, so with all this patches applied qemu-system-sparc64 can successfully boot NetBSD 6.1.4 (and likely all the other sparc64 versions):
Updating motd.
Starting sshd.
postfix: rebuilding /etc/mail/aliases (missing /etc/mail/aliases.db)
Starting inetd.
Starting cron.
Sat Aug  9 11 11:18:59 CEST 2014

NetBSD/sparc64 (netbsd614) (console)

login: root
Aug  9 11:19:27 netbsd614 login: ROOT LOGIN (root) on tty console
Last login: Fri Jan  1 13:21:40 2010 on console
Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
    2006, 2007, 2008, 2009, 2010, 2011, 2012
    The NetBSD Foundation, Inc.  All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
    The Regents of the University of California.  All rights reserved.

NetBSD 6.1.4 (GENERIC)

Welcome to NetBSD!

Terminal type is wsvt25.
We recommend that you create a non-root account and use su(1) for root access.
netbsd614#  ping
PING ( 48 data bytes
64 bytes from icmp_seq=0 ttl=255 time=2.463 ms
64 bytes from icmp_seq=1 ttl=255 time=1.994 ms

Have fun and please report the bugs to the qemu-devel@ mailing list.

Sunday, June 8, 2014

IOMMU support in upstream qemu-system-sparc64

I didn't write about SPARC emulation lately because I didn't work on it. But Mark Cave-Ayland is still working hard, so there is a lot of new stuff. The sun4m emulation has a new graphic adapter: cg3. Also the old adapter (tcx) supports acceleration.

And now, something really cool for the sun4u emulation. As of today git master has a working IOMMU support in qemu-system-sparc64! This means that you don't need to follow all the steps in
Debian/sparc64 Wheezy under QEMU How-To. Just insert your image into -cdrom and voilà!
Well done, Mark!

Now, while using an IDE drive simplifies the installation process a lot, using virtio still makes sense. Quick performance test:

root@debian-wheezy:~#  dd if=/dev/vdb of=/dev/null bs=1024k count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 1.56761 s, 66.9 MB/s

root@debian-wheezy:~#  dd if=/dev/sda of=/dev/null bs=1024k count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 9.59107 s, 10.9 MB/s

Yes, IDE is slower, as the emulated kernel has to do more work, but still, the performance is pretty much ok.

Also, not every OS running on sun4u machines has support for virtio. From an undisclosed source I know that Mark is working on booting NetBSD/sparc64 under vanilla QEMU. Stay tuned. ;-)

Monday, May 19, 2014

800km (500 mi) from San Francisco to Los Angeles

Took my bicycle overseas to ride from San Francisco to Los Angeles. We did it in 7 days. Was a great trip! Not writing the whole story here, just a couple of pictures.
The bike arrived

Saturday, April 20, 2013

Using QEMU to navigate a 3000 tons ship

Some time ago I was approached by a passionate engineer Jean Michel Schramm, who asked whether QEMU would help saving public finances and a navigation system of a 3000 tons ship.

The project looked interesting, so we gave it a try and succeeded!

More details for those who plan to use QEMU for replacement of
hardware becoming extinct.

Saturday, April 6, 2013


Had a really great time at Moscow Center of SPARC Technologies. They invited me to talk about QEMU. The MCST people are very bright and motivated. And their SPARC CPUs are much faster than QEMU can currently emulate. :-) The machines are not Sun clones. For instance it was interesting to see a SPARC V8 machine with a PCI bus.

Despite the company name, they produce not only SPARC systems, but their own VLIW chips, compilers and binary translators as well. Russian readers can find out more on MCST web-site.

Sunday, March 31, 2013


Sunday, March 24, 2013

Debian/sparc64 Wheezy under QEMU How-To

The steps to install Debian Wheezy RC1 / SPARC64 under QEMU.

Since the installation process is not obvious for the current QEMU and Debian versions, I gathered this How-To. Feel free to send any feedback.

Saturday, March 23, 2013

Virtio performance and filesystems

Some time ago there was a comment to my post about booting Linux/sparc64 under QEMU, saying that virtio performance sucks.

At first I couldn't reproduce the problem. Tried a few caching strategies, but the I/O performance was not drastically affected by them. And then it came to my mind that the reason might be the guest file system.

Let's take a look:

$ dd if=/dev/zero of=disk-tmp bs=1024k count=300
300+0 records in
300+0 records out
314572800 bytes (315 MB) copied, 0.430909 s, 730 MB/s

  # so, the theoretical maximal raw performance of this partition on my disk is ~ 730 MB/s.

$ sparc64-softmmu/qemu-system-sparc64 -drive file=../boot-disk.qcow2,if=virtio,index=0 -drive file=disk-tmp,if=virtio,index=1
root@debian-wheezy:~#  mkfs.ext4dev /dev/vdb1
root@debian-wheezy:~#  mount /dev/vdb1  /mnt/
root@debian-wheezy:~#  dd if=/dev/zero of=/mnt/100M-file bs=1024k count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 3.2384 s, 32.4 MB/s

# while it seems to be 20 times slower than the host speed it is still a lot (the number 730MB/s is the theoretical limit, without performing regular syncs. QEMU does execute syncs to ensure the data integrity for the unlikely case of the crash.

Let's take a look on the read performance:

root@debian-wheezy:~#  umount /mnt/
root@debian-wheezy:~#  mount /dev/vdb1  /mnt/
[  582.673975] EXT4-fs (vdb1): mounted filesystem with ordered data mode. Opts: (null)
root@debian-wheezy:~#  dd if=/mnt/100M-file of=/dev/null bs=1024k
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 1.08423 s, 96.7 MB/s

This looks even better. But what happens with the usual ext4?

root@debian-wheezy:~#  mkfs.ext4 /dev/vdb1
root@debian-wheezy:~#  mount /dev/vdb1  /mnt/
root@debian-wheezy:~#  dd if=/dev/zero of=/mnt/100M-file bs=1024k count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 3.54709 s, 29.6 MB/s

Ok, that's a bit less than the newer version. And what about ext3?

root@debian-wheezy:~#  mkfs.ext3 /dev/vdb1
root@debian-wheezy:~#  dd if=/dev/zero of=/mnt/100M-file bs=1024k count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 13.8049 s, 7.6 MB/s

Wow. That's slow. The ext3 partition access with the default options is more than 4 times slower than the ext4 partition access with the default options.

The short conclusion: if you need performance, don't use the ext3 file system with the default settings for your virtual machines.

/ happy hacking

Thursday, January 10, 2013

CruiseControl 2.8.4 and JDK7

Upgraded system JDK to Java 7 and noticed that the dashboard in your CruiseControl stays gray?
Well there is even an official bug CC-1049 in the JIRA (currently it seems to be down, I used Google cache to access it).

With -debug switch on there is a hint error message:

Cannot construct net.sourceforge.cruisecontrol.BuildLoopInformation as it does not have a no-args constructor.

The root of the problem: Oracle changed the vendor name from "Sun Microsystems Inc." to “Oracle Corporation”. The solution is really simple, either add

-Djava.vm.vendor="Sun Microsystems Inc." -XX:+OverrideVMProperties

to your java call in , or replace xstream-1.2.2.jar and xpp3_min- files both in lib/ and webapps/dashboard/WEB-INF/lib/ with xstream-1.4.1.jar and kxml2-2.3.0.jar respectively.

Sunday, May 13, 2012

Networking in Linux/sparc64 under qemu...

... is also possible. The bulit-in ne2k-pci network card doesn't work, but hey there is an even faster virtio-net alternative. At the OpenBIOS "ok" prompt, before the "boot" command in the previous post, type

cd /pci@1fe,0/pci1af4,1000
0 encode-int " interrupts" property

Will send this patch upstream as soon as we clear whether "0" is allowed for the "interrupts" property.

Saturday, May 12, 2012

Booting Linux/sparc64 on todays OpenBIOS

Historical day for everyone interested in the vanilla qemu-system-sparc64 emulator. For the first time it can boot Linux!

Although the patches I sent upstream are  fixing CPU and IOMMU, there is still one missing piece: cmd646 IDE. Luckily it's not a showstopper at all: qemu-sparc64 is a PCI machine, which means one could use the fast virtio instead! Now, this is the command line:

$  sparc64-softmmu/qemu-system-sparc64 -m 256 -nographic -prom-env 'auto-boot?=false' -kernel /path/to/kernel/image -drive file=/path/to/debian-disk,if=virtio,index=0 -append 'root=/dev/vda1 init=/bin/sh'

Some time ago, I used Forth to workaround missing qemu features  to get OBP working.
Now it's pretty similar: OpenBIOS doesn't have all the properties necessary for Linux to get the interrupt mapping.

So at the ok command prompt where you'd get after the command above, type:
cd /
1 encode-int " #interrupt-cells" property
cd /pci@1fe,0/ebus
000001fe encode-int 020003f8 encode-int encode+ 1 encode-int encode+
ffe29140 encode-int encode+ 2b encode-int encode+ " interrupt-map" property
000001ff encode-int ffffffff encode-int encode+ 00000003 encode-int encode+
" interrupt-map-mask" property

cd /pci@1fe,0/ebus@3/su
1 encode-int " interrupts" property
cd /pci@1fe,0/pci-ata@5
0 encode-int " interrupts" property

cd /pci@1fe,0/pci1af4,1001
0 encode-int " interrupts" property

The only magical constant above is actually "ffe29140" - the address of the root pci node in the OpenBIOS device hierarchy. Although it the same in all the recent builds, in theory it could be moved somewhere else one day. But I guess till that day OpenBIOS will have the missing properties... ;-)


Oh, and a few words to /path/to/kernel/image and /path/to/debian-disk:

I found no Linux/sparc64 distribution which has a built in virtio driver. This makes installing from a  CD/DVD image not possible. The solution is build your own kernel with the virtio driver compiled in and put it at /path/to/kernel/image.

As for the disk, the user space utilities from the sparc32 world can be used in the sparc64 world as is. So, you can install Debian (or your favorite sparc32 distribution) in the /path/to/debian-disk, using qemu-system-sparc (no 64 at the end), and then use it with qemu-system-sparc64 as described above.

If you know a Linux/sparc64 distribution with the virtio support, please let me know.

/Happy hacking

Sunday, May 6, 2012

Qemu is going to boot Linux/sparc64

After considering it a bit, I thought, it's a good marketing strategy: the free QEMU version shall run the free OS - Linux. And if anyone is interested in running something else, feel free to ask me for a [paid] support. :-)

So, Linux is going to be the second OS which vanilla qemu-system-sparc64 would boot - HelenOS was the first one. But, unlike HelenOS, Linux will be fully functional, having not just a stdout, but a stdin as well. And a serial port support!

At the moment OpenBIOS has some missing features - it doesn't describe the interrupt mappings - and a regression - currently it can't even boot HelenOS from a command line. But both are not show-stoppers.

Once my patches are accepted I'll publish the OpenBIOS command to boot Linux (not because it's top secret, but due to the dependency to a certain version).

Sunday, April 22, 2012

Sent y2k10 fix upstream

It occurred to me that I've described the fix for y2k10 bug in qemu-system-sparc which makes Solaris 2.5.1 (and prior) hang, but never managed to submit it upstream.

So, let's fix it. For those one and half users who care ;-) .

Friday, April 6, 2012

The 100th post (cycling in a City is dangerous)

Allright, this is my 100th post. I was going to look back, summarize and talk about the future.

But I won't.

Yesterday on my way to the office I broke my leg. Multiple fractures. After all cycling in a city on an asphalt is more dangerous than cycling on an ice.

The good news: I'm going to have more time to read books and work on Open Source projects...