Sunday, March 24, 2013

Debian/sparc64 Wheezy under QEMU How-To

The steps to install Debian Wheezy RC1 / SPARC64 under QEMU.

Since the installation process is not obvious for the current QEMU and Debian versions, I gathered this How-To. Feel free to send any feedback.

 0. Getting stuff

Get the latest QEMU from git (released versions up to 1.4.x won't work).

Download debian-wheezy-DI-rc1-sparc-DVD-1.iso.

 1. Extracting the kernel and initrd

Currently qemu-system-sparc64 does not emulate the hardware devices the Debian can boot from. This is not a show stopper at all! We just going to need some extra steps for setting the things up.  Mount the image on your host machine. As root,

# mkdir -p /mnt/wheezy
# mount -o loop,ro /path-to/debian-wheezy-DI-rc1-sparc-DVD-1.iso /mnt/wheezy

(if you don't want to do it as root, use isoinfo, fuseiso, 7z or whatever utility your OS has for extracting the files from ISO images)

As user which will run qemu,

$  mkdir ~/debian-wheezy-DI-rc1-sparc
$  cp  /mnt/wheezy/boot/sparc64 ~/debian-wheezy-DI-rc1-sparc
$  cp  /mnt/wheezy/boot/initrd.gz ~/debian-wheezy-DI-rc1-sparc

 2.  Adding the VirtIO modules to initrd

Unfortunately Debian Wheezy RC1 doesn't have virtio modules in the installation initrd.  This won't stop us, as there is a package with them on the install media:

$  cd ~/debian-wheezy-DI-rc1-sparc

$  ar -p /mnt/wheezy/pool/main/l/linux/virtio-modules-3.2.0-4-sparc64-di_3.2.35-2_sparc.udeb  data.tar.xz |tar xfJ -
$  umount /mnt/wheezy

 now we have the modules:

$  ls -l lib/modules/3.2.0-4-sparc64/kernel/drivers/virtio/
total 56
-rw-r--r--. 1 user user 13184 Dec 19 17:10 virtio_balloon.ko
-rw-r--r--. 1 user user 11208 Dec 19 17:10 virtio.ko
-rw-r--r--. 1 user user 14480 Dec 19 17:10 virtio_pci.ko
-rw-r--r--. 1 user user 11368 Dec 19 17:10 virtio_ring.ko

 add them to initrd:

$  gunzip -c initrd.gz > initrd+virtio
$  find lib/ |cpio -o -H newc --append -O initrd+virtio
$  rm -r lib
$  gzip initrd+virtio

Done. Now we have initrd+virtio.gz which contains the original initial RAMdisk contents and the virtio modules.

3. Creating image for the virtual guest disk

$ mkdir ~/debian-wheezy
$ qemu-img create -f qcow2 ~/debian-wheezy/debian-wheezy.qcow2 4G

4. Launching QEMU for the installation

# Please note that you'll need another command line for running the installed system

$  sparc64-softmmu/qemu-system-sparc64 -L pc-bios -m 384 -nographic \
-monitor telnet::4440,server,nowait -kernel ~/debian-wheezy-DI-rc1-sparc/sparc64 \
-initrd ~/debian-wheezy-DI-rc1-sparc/initrd+virtio.gz \
-drive file=~/debian-wheezy/debian-wheezy.qcow2,if=virtio,index=0 \
-drive file=/path/to/debian-wheezy-DI-rc1-sparc-DVD-1.iso,if=virtio,readonly=on,index=1 \
-net nic,model=virtio -net user -append 'modprobe.blacklist=pata_cmd64x'

5. Answering Installer's questions

Some seconds after launching the installer will be asking you about the locale settings. Nothing tricky about them. Then it will be trying hard to find a CD-ROM drive...

After this point comes a blank screen. Be patient it may take some minutes. And then... it won't find the CD/DVD drive, as we didn't give it any. Actually it was on purpose. The "pata_cmd64x" driver crashes under QEMU because of some missing functionality. So it's not only important not to launch QEMU with a '-cdrom' option, but also blacklist  the pata_cmd64x driver (I didn't find the way to completely disable DMA. In PIO modes cmd64x might even be working).

The proper answer here is 'No'. We don't need to load additional drivers. The next question is whether we want to select  the driver manually:

The answer is 'Yes'. We do want to select it manually...

...sort of. The module needed for accessing the CD-ROM is 'none' (we haven't plugged a real CD-ROM). The next question is the Device Name:

The default '/dev/cdrom' won't work (did I mention that we haven't plugged a real CD-ROM?). Use the virtio device /dev/vdb instead.
The next few steps are not QEMU-specific. You will be asked ...

for the Hostname, ...

password for the root user, ...

and disk partitioning. Here it makes more sense to use the "Manual" method. The "Guided" method creates a /boot partition (from which QEMU can't  boot using virtio anyway) and too much swap for a virtual machine. So, unless you are the SILO developer/tester, you don't need a special /boot partition.

The installer warns that it will re-label your empty disk image.

Ext4 under QEMU is significantly faster than Ext3 (see a quick test here).

That's an example how it can be partitioned without /boot, ...

...and here comes the warning that SILO can not boot from an ext4 partition. Just ignore it.

Answer yes to write partition information. After this point the installation starts. It takes some tens of minutes (at least on my machine), so be patient. You'll figure as you see the progress monitor.

Not QEMU-specific. Select standard system utilities.

After taking the 'Continue' option the system will panic with the message Reboot failed:

Requesting system reboot
[ 3205.593245] Restarting system.
BOOT[ 3205.597929] Kernel panic - not syncing: Reboot failed!

This is expected, because the OpenBIOS can not reset system. So just use the QEMU monitor. From another shell execute

$ telnet localhost 4440
Connected to localhost.
Escape character is '^]'.
QEMU 1.4.50 monitor - type 'help' for more information

And enter 'q' here to quit the QEMU process gracefully.

6. Getting the installed kernel and initrd 


Use the guestfish to get the files from the qcow2 image:

 $  cd  ~/debian-wheezy/
 $  guestfish  --ro -a debian-wheezy.qcow2 -m /dev/vda1 copy-out /boot/vmlinuz-3.2.0-4-sparc64 /boot/initrd.img-3.2.0-4-sparc64 .

     # -- QEMU can't load compressed kernel, so uncompress it

 $  gunzip -c vmlinuz-3.2.0-4-sparc64 > vmlinux-3.2.0-4-sparc64

7. Booting from the installed Debian Wheezy / sparc64 image

Use the kernel and initrd extracted at the step 6.

$  sparc64-softmmu/qemu-system-sparc64 -L pc-bios -m 384 -nographic \
-monitor telnet::4440,server,nowait -kernel ~/debian-wheezy/vmlinux-3.2.0-4-sparc64 \
-initrd ~/debian-wheezy/initrd.img-3.2.0-4-sparc64 \
-drive file=~/debian-wheezy/debian-wheezy.qcow2,if=virtio,index=0 \
-net nic,model=virtio -net user -append 'modprobe.blacklist=pata_cmd64x root=/dev/vda1'

optionally you can add
-drive file=/path/to/debian-wheezy-DI-rc1-sparc-DVD-1.iso,if=virtio,readonly=on,index=1
to make the installer DVD available for your guest OS.

/ Happy hacking


Sparcher said...

> Unfortunately Debian Wheezy RC1 doesn't have virtio modules in the installation initrd.
> This won't stop us, as there is a package with them on the install media

Hmm, this should probably be reported as bug (if it hasn't been reported yet)...

atar said...

Yeah, reported twice. #673320 and #704672.
Looks like this how-to helped to push it. Steve McIntyre promised that in rc2 it will be fixed.

mrRadio said...

Здравствуйте Артем !
А незнаете ли вы, есть ли возможность(хотя-бы теоритическая) запустить в qemu sparc - Solbourne OS/MP 4.1.

atar said...

Привет Дмитрий! (что за официальность, кстати? Мы же вроде в прошлый раз были на "ты"?)

Искал Solbourne несколько лет назад, но не нашёл.

OS/MP 4.1 умеет запускаться на одном CPU? Если да - шанс есть. Носитель - CD или лента? Если CD, QEMU его надо подсовывать как жёсткий диск. Потому что размер сектора, скорее всего, 512 байт.

Если OS/MP слишком привязана к Solbourne, может потребоваться родная прошивка ПЗУ (у меня её нет) и немного колдунства с соединением компонентов QEMU (но там всё просто, если знать, что по какому адресу подключать).

А если OS/MP не может запускаться на одном проце - надо сначала найти, из-за какого бага, OBP под QEMU не опознаёт наличие более одного процессора.

mrRadio said...

У меня CD, на образе UFS(big endian), поэтому посмотреть что на диске у меня возможности нет.
По твоей инструкции о установке Solaris, я попытался загрузиться с образа но получил:
ok boot disk1:d -vs
Boot device: /iommu/sbus/espdma@5,8400000/esp@5,8800000/sd@1,0:d File and args: -vs
The file just loaded does not appear to be executable.
Возможно все дело в фирмваре, у меня есть дампы солборновских железок, но мне это дампы кажутся странными, да и Sparc я полный 0

Я посчитал, что лишним оффициоз не будет, так как своим странным хобби возможно отвлекаю вас от важных дел.

atar said...

Вообще-то, UFS под линуксом прочитать довольно легко. (Изменять труднее и опаснее). Под CentOS я просто делаю

mount -o loop,ro /path/to/img /mnt

иногда ещё надо указать ufstype=sun (dmesg про это скажет).

Sparcher said...

> The "pata_cmd64x" driver crashes under QEMU because of some missing functionality.

Probably, it would be better for QEMU to emulate some SCSI controller instead of this PATA one...

atar said...

>Probably, it would be better for QEMU to emulate some SCSI controller instead of this PATA one...

PATA is used just to be close to the original Ultra-5 machine. QEMU can emulate multiple PCI SCSI controllers.

But virtio is faster than emulated SCSI devices anyway (especially since due to the current limitations in the upstream SPARC PCI controller they would only be able use PIO mode).

Karel Gardas said...

Hi, excellent blog article, however now, the final one is needed! How to run Solaris 10 on qemu/sparc64? Would be most welcome here! Thanks! :-)

Anonymous said...

thank you for the article,

i tried debian 7.8 (last sparc release)
because 7.7 and earlier versions are not available anymore
debian 7.8 does still not include the virtio modules in initrd

with qemu git (2015/07/02) build under ubuntu 15.04 x64

i followed your virtio installation step by step but the installation breaks with "installtion failed" while installing the "standard system utilities" and the resulting images is not bootable

any avices for using debian 7.8 with virtio?

installing without virtio directly booting from dvd works but is way to slow - test installation runned for hours and it would be nice to have a little bit faster system

atar said...

Can you please check the logs after the "installtion failed" message? What is failing exactly? I haven't tried it recently but from your report it looks like it aborts after successful loading the virtio drivers. It may be a regression in a sparc kernel or QEMU sparc emulation (the later is less likely because there were no sparc - related changes recently).

Anonymous said...

i will get back to my computer in two weeks and will check the installation logs (console 4 or /var/log/syslog i think) while getting the error, using a freshly git qemu - and 3 shell-scripts for initrd virtio preperation, installtion-start and after-installation-extraction + some screenshots - anything else i can prepare?

Anonymous said...

back to my computer i tested the installation again but i've got problems to get to the log console - ctrl-alt-f1...f4 does nothing, esc-1..2 works sometimes, also qemu monitor sendkey does not work

tested on ubuntu itself and a putty (for the setup and another for qemu monitor) connection from windows

Anonymous said...

got the syslog :)

debian 7.8 sparc install fail syslog

my install start script

the non-virtio installtion with
qemu-system-sparc64 -m 1024 -hda debian-7.8.0.qcow2 -nographic -cdrom debian-7.8.0-sparc-DVD-1.iso -boot d
works/runs without any problem

Anonymous said...

better compareable (diffable) syslogs of virtio and non-virtio installation
i replaced alle PIDs/ect. with !better_diffable!



it seems that the virtio installation (with working ethernet) tries
to uses different (broken?) security packages from the internet - is that the problem?

Anonymous said...

and another info:

the non-virtio installation show different options while installation

for example:

the "Participate in package survey" question does come not up when using virtio

the non-virtio installation gives me the complete list of installable software like Desktop Environment, SSH Server, etc. the virtio just "Basic System utilities"

atar said...

Yes, for some reason it says:
"Unable to locate package popularity-contest"
which would start the survey. What is strange, the non-virtio install is failing to connect to , so it doesn't seem to install any updates.
One idea is to try a virtio install where qemu is not connected to the net, to check whether security updates break anything.

Anonymous said...

the virtio installation works if i disable network

but i still get only "Basic System Utilities" not the complete list and
no package popularity-contest question like in the non-virtio installation


virtio with net:
not the complete list of installabale software - only basic utilities
no package popularity-contest question
-->install fail some time after "basic system utilities" installation

virtio without net
not the complete list of installabale software - only basic utilities
no package popularity-contest question
-->install works and is bootable

non-virtio with net
complete list of installabale software
package popularity-contest question
-->install works and is bootable

anything else i can do?

Anonymous said...

and still the question why non-virtio fails to connect to

atar said...

Report a bug to qemu-devel@ and . :-)

I'll try installing the version 7.8 tonight. And I think I'll have to update this how-to because meanwhile QEMU can perfectly boot from the IDE drives, so using an IDE drive for /boot and a VirtIO drive for / might be easier than extracting the kernel/initrd from the VirtIO disk.

atar said...

I think it doesn't connect to because of your command line. Try adding "-net user" to it.

atar said...

Yes, I can reproduce the setup messing up when a virtio device is used instead of cdrom. It seems to be a regression in the Debian installer. What is also interesting, the late stage just ignores device explicitly given as a cdrom from which the base system is installed. Without modprobe.blacklist=pata_cmd64x, it keeps trying /dev/sr0.

Well, since there is a workaround - using the ide cdrom - and since sparc is not an official Debian port anymore it's quite unlikely that it's gonna be fixed. But who knows...

Anonymous said...

will you update or post a new blog about the debian 7.8 installation / and the known problems - could help the bug report to have an known sparc/qemu authoritiy behind :)

so the current workaround is

using virtio for the hda
using ide for cdrom
blacklist pata_cmd64x


qemu-system-sparc64 -m 1024 -nographic -drive file=~/image.qcow2,if=virtio,index=0 -cdrom~/debian-7.8.0-sparc-DVD-1.iso -net nic,model=virtio -net user -append 'modprobe.blacklist=pata_cmd64x'


atar said...

Not quite. Blacklisting partially solves problems in virtio-only install, but shall not be used with -cdrom option (with it the ide driver is blacklisted). So it's just

qemu-system-sparc64 -m 1024 -nographic -drive file=~/image.qcow2,if=virtio,index=0 -cdrom ~/debian-7.8.0-sparc-DVD-1.iso -net nic,model=virtio -net user

Anonymous said...

and kernel sparc64 + virtio-initrd

atar said...

Haven't tried, but it should be not necessary when booting from an ide cdrom.

Anonymous said...

do i still need to disable network with -net user,restrict=y then?

atar said...

No, disabling the network is not necessary. I did an install from an IDE cd to a VirtIO hdd with network. Chosen a German mirror when the installer asked for it, but I think it should not matter.
The install is a bit slower than a pure VirtIO install, but not 6 times slower as I was afraid (I wrote about VirtIO vs. IDE performance some time ago).

Anonymous said...

>No, disabling the network is not necessary.
>I did an install from an IDE
>cd to a VirtIO hdd with network.

that seems to work

>Chosen a German mirror when
>the installer asked for it

me too

>but not 6 times slower
>as I was afraid

i've installed the whole stuff serveral times an
would say at least 3-4 times slower

thx for the help

Anonymous said...

the installation works - thx

is there anything i can do to speedup qemus emulation (attach cores or stuff like that)?

its still very slow

atar said...

3-4 times slower is strange: I did a quick benchmarking and in case of a DVD iso image the throughput seems to be only 20% less:

# dd if=/dev/sr0 of=/dev/null bs=1024k count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 2.0884 s, 50.2 MB/s
# dd if=/dev/vdb of=/dev/null bs=1024k count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 1.81383 s, 57.8 MB/s

And for some reason a virtio HDD access is 2 times faster than virtio DVD access.

As for speeding up, I'm afraid currently there is no way to do it. You can try using more aggressive gcc optimization options or even using icc for compilation (some tests shown 15% performance gain with it on my hardware), but yeah. QEMU is not that quick when emulating sparc64 CPU.

Maybe it gets better when the upstream switches to the multi-threaded TCG.

Anonymous said...

>the throughput seems to be only 20% less

the virtio-dvd (which fails in the end) is definitiely multiple times faster - i reinstalled the whole stuff serveral times

also the rest of the system is very slow
currently compiling CMake - for the last 2 hours i think - under the x86 host itself a thing of few minutes

Anonymous said...

cmake 3.2.2 tooked more then 10 hours to build - but it works :)

atar said...

That's slow. Have you had an experience with building it on a real sun4u? Is actually cmake the goal, or are you going to use it for building something else?

Can you please try looking in "perf top" during the build to find out where does QEMU spend time the most? Maybe there are some easy ways to gain more performance.

Anonymous said...

>That's slow. Have you had an experience with building it on a real sun4u?

hardcore slow
no i don't have a sun4u available
have you got one with debian 7.8.0 for comparison available :)

>Is actually cmake the goal,
>or are you going to use it for building something else?

want to build something other - bigger :(

>Can you please try looking in "perf top"
>during the build to find out where does QEMU
>spend time the most? Maybe there are some
>easy ways to gain more performance.

im still unable to switch to another virtual terminal

1. qemu monitor sendkey
2. ALT + left or right arrows on the keypad
3. stick" it to the "virtualized window" followed by the other keys e.g. CTRL+CTRL+CTRL+ALT+F1

anything else i can do to get to another login

atar said...

I usually set port forwarding (either with qemu command line or via ssh from guest to host) and ssh from host to guest as many sessions as I need.

But when I was writing about "perf top", I meant the host, not the guest. That would show where QEMU spends its time while building your software.

atar said...

And, btw, if you are tunneling with ssh (, it makes sense to use an faster cipher .

Anonymous said...

im not using tunneling - but maybe i should

running "top perf" after boot gives:
53.16% [kernel] [k] touch_nmi_watchdog
39.27% [kernel] [k] touch_softlockup_watchdog
2.78% perf_3.2 [.] 0x42e5c
1.22% perf_3.2 [.] dso__find_symbol

now i need to setup ssh access and perf the cmake (or something smaller build)

atar said...

Why do you need ssh to run perf top on the host? The idea is to profile QEMU process itself and not the guest OS. But to make it sure you are profiling something useful and not the OS idle loop, the guest has to run something useful (the build in your use case).

Anonymous said...

mit Host und SPARC

Anonymous said...

Pugixml(and test) builds for ages

Anonymous said...

if you want to have a look

i uploaded my debian image for the pugixml developer to help solving some big-endian/unaligned access problems

using the free file hoster depositfiles (seems to be the only one with ftp upload feature)

press the "regular download" button, wait some seconds, solve the chapca, "download file in regular mode by browser"

there is pugi_sparc.txt in the 7z which describes how to start and what is installed in the image

Anonymous said...

i benchmarked the pugixml.cpp ( compile

qemu-system-sparc64: >180sek
x64 native : ~ 2sek

that explains my 10h cmake build perfectly :)

atar said...

Have you looked how much RAM does gcc need for compiling the file? Could it be that the virtual Debian starts swapping/paging?

Anonymous said...

root@debian:~# free -mh

total: 1.0G
used: ~140...150MB
free ~800...900MB

total: 86M
usage: 0M (never in use)

while compiling

atar said...

If you are not limited to Linux, there is a faster emulator - TME. I wrote about it a few years back. But afair it can only boot NetBSD.