AIX under QEMU boots up to NFS

Launching a proprietary OS under QEMU is never boring because every next problem has to do with yet another component. A few weeks ago I was mostly doing Forth to make a bootable firmware, then I fought with the missing residual data, at which point it was mostly debugging ODM database using the PPC assembly, then extracted the mock residual data from the live system, then spent some time with NCR/LSI script, and now after fixing the PCI layout it gets to the point of starting the NIS and NFS services. It works much slower than the real machine, and also much slower than Linux/PPC, but still:

MOT PowerStack2 (e0), Serial #0, 62 MiB memory installed
Open Firmware , Built  September 01, 2017 16:11:38
Copyright (c) 1995-2000, FirmWorks.
Copyright (c) 2014,2017, Artyom Tarasenko.

Rebooting with command: boot /pci/scsi@2/disk@0,0
Boot device: /pci/scsi@2/disk@0,0  Arguments:

+ swcons -c

Saving Base Customize Data to boot disk
Starting the sync daemon
Starting the error daemon
System initialization completed.
Starting Multi-user Initialization
 Performing auto-varyon of Volume Groups
 Activating all paging spaces
swapon: Paging device /dev/hd6 activated.
 Performing all automatic mounts
mount: 1831-010 server axxxxs01 not responding:
RPC: 1832-018 Port mapper failure - RPC: 1832-006 Unable to send
mount: backgrounding
axxxxs01:/home
Multi-user initialization completed
Checking for srcmstr active...complete
Starting tcpip daemons:
0513-056 Timeout waiting for command response.
0513-056 Timeout waiting for command response.
vmtune:  current values:
  -p       -P        -r          -R         -f       -F       -N        -W
minperm  maxperm  minpgahead maxpgahead  minfree  maxfree  pd_npages maxrandwrt
   2968    11872       2          8        115      123     524288        0

  -M       -w       -k       -c         -b          -B          -u
maxpin   npswarn  npskill  numclust  numfsbufs   hd_pbuf_cnt  lvm_bufcnt
  12692     1536      384        0       93           64           9

number of valid memory pages = 15864    maxperm=74.8% of real memory
maximum pinable=80.0% of real memory    minperm=18.7% of real memory
number of file memory pages = 1443      numperm=9.1% of real memory

vmtune:  new values:
  -p       -P        -r          -R         -f       -F       -N        -W
minperm  maxperm  minpgahead maxpgahead  minfree  maxfree  pd_npages maxrandwrt
   2968    11872       2          8        115      123     524288       64

  -M       -w       -k       -c         -b          -B          -u
maxpin   npswarn  npskill  numclust  numfsbufs   hd_pbuf_cnt  lvm_bufcnt
  12692     1536      384        1       93           64           9

number of valid memory pages = 15864    maxperm=74.8% of real memory
maximum pinable=80.0% of real memory    minperm=18.7% of real memory
number of file memory pages = 1444      numperm=9.1% of real memory

Starting NIS services:
Starting NFS services:

0513-056 Timeout waiting for command response.
NIS: server not responding for domain "axxxxs01"; still trying.
NIS: server not responding for domain "axxxxs01"; still trying.
0513-056 Timeout waiting for command response.
NIS: server not responding for domain "axxxxs01"; still trying.
NIS: server not responding for domain "axxxxs01"; still trying.
NIS: server not responding for domain "axxxxs01"; still trying.

Woo-hoo! I could be proud that I made the second proprietary OS boot under QEMU. Or even the third one, because Solaris/sun4m is quite different from Solaris/sun4v, so probably each one counts for one.

But now I'm puzzled. Initially I planned to build a throw-away prototype for Powestack II Utah, fix the QEMU bugs preventing booting AIX with the existing machines, and dispose the prototype. The Motorola AIX I've got seems to expect the machine to be called "MOT PowerStack2 (e0)". Not sure if pushing such a model upstream wouldn't cause any copyright/trademark issues.

Now it boots to the same point as on the physical machine, but except for the PCI layout problems, I haven't found any bugs. And the PCI layout can not be the reason of AIX failing on the 40p machine, because it fails much earlier.

So obviously it's time to change the plan. And here is where things are going to slow down. I don't have the install media for the Motorola AIX 4.2, so I can not make my disk usable - it waits for NFS/NIS servers forever. I can not share the HDD image because it may contain the private data, so publishing the Powerstack II Utah target for QEMU makes a little sense for now. So, either I find the Motorola AIX 4.2 install CD (and probably a physical UW-SCSI drive if it won't work at the first attempt under QEMU), or the work has to be done again for the 40p target. Yes, now I have some know-how about the AIX boot process, but still at the moment I don't feel like starting again from scratch with the 40p target.

The good news is that AIX can definitely be booted under qemu-system-ppc.

From SCSI to PCI

20 years ago SCSI devices ruled the world. Reading the NCR53c810 manual, I see that it was basically a computer in a computer. It can be programmed to transfer data from/to/between the disks without using the CPU at all.

Trying to understand what happens in the NCR/LSI script:

lsi_scsi: Select LUN 0
lsi_scsi: Extended message 0x1 (len 3)
lsi_scsi: SDTR (ignored)
lsi_scsi: SCRIPTS dsp=81c3b924 opcode 80080000 arg 81c3b9a4
lsi_scsi: Jump to 0x81c3b9a4
lsi_scsi: SCRIPTS dsp=81c3b9a4 opcode 870b0000 arg 81c3b9c4
lsi_scsi: Compare phase 2 == 7
lsi_scsi: Control condition failed
lsi_scsi: SCRIPTS dsp=81c3b9ac opcode 860a0000 arg 81c3b94c
lsi_scsi: Compare phase 2 == 6
lsi_scsi: Control condition failed
lsi_scsi: SCRIPTS dsp=81c3b9b4 opcode 98080000 arg 00000022
lsi_scsi: Interrupt 0x00000022
...
lsi_scsi: SCRIPTS dsp=81c3bb6c opcode 0e000002 arg 81c3bdb4
lsi_scsi: MSG out len=2
lsi_scsi: Select LUN 0
lsi_scsi: MSG: ABORT tag=0x0
lsi_scsi: SCRIPTS dsp=81c3bb74 opcode 80080000 arg 81c3bbbc
lsi_scsi: Jump to 0x81c3bbbc
lsi_scsi: SCRIPTS dsp=81c3bbbc opcode 60000008 arg 00000000
lsi_scsi: Clear ATN

Looks like it aborts if the selected SCSI target doesn't change phase to MSG_OUT or MSG_IN. So I implemented a hack for SDTR reply and it doesn't abort here. But indeed it's a red herring. AIX can also work with the devices which do not support the synchronous or wide transfers.

The actual problem happens later:

lsi_scsi: SCRIPTS dsp=81c43444 opcode c0000004 arg 010000dc
lsi_scsi: memcpy dest 0x81c435fc src 0x010000dc count 4

Or, with a bit more enhanced logging:

lsi_scsi: memcpy dest 0x81c475fc (Mem) src 0x010000dc (IO) count 4
lsi_mem_read, address_space_read status 2
lsi_scsi: the first 4 bytes: 00 00 00 00

It tries to read the port 0x10000dc and save it. QEMU doesn't have anything at the port 0x10000dc, so no wonder the NCR script fails. But what is supposed to be there? The Motorola Ultra 603/Ultra 603e/Ultra 604 Programmer’s Reference Guide suggests it must be the PCI I/O space.

So looks like I've learned enough of NCR/LSI script. Time to see how the PCI bus mastering is supposed to work on this machine.

Wiretapping AIX

Identified a couple of kernel and shared library functions, so I'm not poking in the dark anymore:

First of all I found execv. It gives a lot of insights about the AIX boot process. The process is quite different from Linux or Solaris boot. Kernel is small, and actually is already loaded, even under QEMU. The most other operating systems would write a greeting once a kernel is loaded. AIX does it all silently. On IBM machines there is a LED panel showing one byte of a status. On the Motorola there are just two LEDs which can light green or yellow, which altogether gives just 9 combinations. Not very informative. But even if I had one byte,  it still would not help. I look for error messages like "missing property", "unknown PCI chip", "missing residual data", etc.

The initialization of the PCI bus happens long after  the kernel spawns the /etc/init process.

Breakpoint 20, 0x0008cd38 in ?? ()
(gdb) x/s $r3
0x20051d08:     "/etc/methods/defsys"
(gdb) c
Continuing.
Breakpoint 20, 0x0008cd38 in ?? ()
(gdb) x/s $r3
0x2ff22090:     "/bin/sh"
(gdb) c
Continuing.
Breakpoint 20, 0x0008cd38 in ?? ()
(gdb) x/s $r3
0x20051d28:     "/usr/lib/methods/cfgsys_MOT3F00"       <= here is where it can't find the PCI bus

Then I found the printf and sprintf functions. Although AIX doesn't write anything on the screen, it still collects the boot log messages, so wiretapping  printf and fprintf helps to see them.

The house is still dark but now I have a search light. So whatever bugs are there, beware, you are going to be seen soon!

Debugging AIX 4.2 boot

I wonder if it is possible to make the AIX 4.2 boot more verbose.
The various sources say that it should be done via

mw enter_dbg

under KDB. The AIX version I have doesn't have it. In fact it even doesn't have an option to disassemble a piece of code. Just the hardcore hex-dump, pretty much like it was in eighties.

That feeling when you started with a retro-computing and ended up with a steam punk computing.

ok  boot /scsi/disk@6 -s trap
Trap instruction interrupt.
> mw enter_dbg
032-001  You entered a command «mw» that is not valid.
> help
alter   … (a)lter — alter memory
back    … (b)ack — decrement the IAR
ditto   … «» — blank repeats the last command
break   … (br)eak — set a breakpoint
breaks  … (breaks) — list currently set breakpoints
buckets … (bu)ckets — display kmembucket structures
clear   … (c)lear — clear breakpoint(s)
display … (d)isplay — display a specified amount of memory
dmodsw  … (dm)odsw — display Streams dmodsw table
drivers … (dr)ivers — display device driver (devsw) table
find    … (f)ind — find a string in memory
float   … (fl)oat — display floating point registers
fmodsw  … (fm)odsw — display Streams fmodsw table
fs      … fs — display file system data structures
go      … (g)o — start executing the program
help    … (h)elp — display the list of valid commands
loop    … (l)oop — execute until control returns to this point
map     … (m)ap — display the system loadlist
mblk    … (mb)lk — display mblk/kmemstat structures
next    … (n)ext — increment the IAR
origin  … (o)rigin — set the origin
proc    … (p)roc — process table display
quit    … (q)uit — end the debugger session
queue   … (que)ue — display Streams queues
reset   … (r)eset — release a user defined variable
restore … (re)store — restore or do not restore the screen
screen  … (s)creen — display a screen containing registers and memory
set     … (se)t — define an/or set a variable
sregs   … (sr)egs — display segment registers
st      … (st) — store a full word into memory
stack   … (sta)ck — formatted stack trace
stc     … (stc) — store one byte into memory
step    … (ste)p — perform an instruction single-step
sth     … (sth) — store a half word into memory
stream  … (str)eam — display Stream head structures
swap    … (sw)ap — switch from the current display/keyboard to RS-232 port
thread  … (th)read — thread table display
trace   … (tr)ace — print traceback buffer
trb     … (trb) — display formatted timer request block info
tty     … (tt)y — Display tty struct
user    … (u)ser — formatted user area
uthread … (ut)hread — formatted uthread area
vars    … (v)ars — display a listing of the user_defined variables
vmm     … vmm — display virtual memory data structures
xlate   … (x)late — display the real address of a memory location
>

Booting OFW from OFW

In order to use my new Motorola Powerstack II Utah machine as a reference for improving qemu there are two ways:

a) make qemu run the Powerstack II firmware
b) make Powerstack II run my firmware

I quickly tried running the Powerstack II firmware under qemu. After all that was the way which did let me run Solaris/SPARC under qemu 7 years ago. The firmware sort of starts, but gets into some very limited debugger. It looks to me that the debugger is from Motorola and it starts before launching the OFW. Last week I found out that the Powerstack II Utah firmware is no good for anything but one version of AIX, so this particular version is really not worth of launching.

So went for the option b) and made a firmware which can be netbooted on the Uhah.
It's even bootable on both Powerstack I and II, which was tricky. For some reason the different Powerstacks have slightly different ideas about the layout of the 0x41 partition. For instance, Solaris floppy image can not be netbooted. But there is one layout which works both for floppy any netbooting on both Powerstacks.

So now I netboot my OFW from the Motorola OFW, and then try booting AIX from the SCSI disk.
And then it hangs with the residual data. No wonder here - I used the device tree from qemu, so at least some devices are different or wired differently. But if I remove the creation of the residual data, it even boots AIX to the same point as the Motorola OFW.

The SCSI host on both Powerstacks is different than on the 40p machine.  The 40p (and qemu) have Vendor Id: 0x1000, Device ID 0x0001, which is according to the pcidatabase a LSI53C810 chip. The Powerstacks have Vendor Id: 0x1000, Device ID 0x0003, which is supposedly LSI53C1010-33. On the chip is written Symbios Logic 53C825A.

I may hit the difference beween the LSI (formerly known as Symbios and NCR) chips later, but at least 53C825A is reverse compatible to 810, otherwise my firmware would not be able to load anything.

My new toys: Motorola PowerStack II

The second gift from Jochen is a Powerstack II mainboard with an AT power supply unit and a SCSI disk. The SCSI disk has "AIX" written on it, which looks promising, but Jochen doesn't remember if it was really installed, or just planned.

The board has Serial/Parallel/Ethernet/SCSI and even a couple of unsoldered IDE connectors.
The boot log shows it has a Firmworks based Open Firmware:

WARNING: NVRAM Header Test Failed - Auto Initializing
Starting real time clock...
screen not found.
Can't open input device.
Keyboard not present.  Using com1 for input and output.
, Serial #0, 64 MB memory
Power Firmware(TM) by FirmWorks , Built  Thu Jun 4 10:20:43 MST 1998
Copyright (c) 1995-1996 FirmWorks.  All Rights Reserved.
PowerPC Open Firmware
Version 1.2 RM11   Thu Jun 4 10:20:43 MST 1998
Copyright Motorola 1995-96, All Rights Reserved
Copyright FirmWorks 1995-96, All Rights Reserved

 CPU . . . . . . . . . . . . . . . . . . . . . . . . . . . =PowerPC,604e
 MicroProcessor Internal Clock Speed (MHZ) . . . . . . . . =300
 MicroProcessor External Clock Speed (MHZ) . . . . . . . . =67
 PCI Bus Clock Speed (MHZ) . . . . . . . . . . . . . . . . =33
 Local Memory Size . . . . . . . . . . . . . . . . . . . . =4000000 (64 MB)
 Memory Type . . . . . . . . . . . . . . . . . . . . . . . =EDO
 Memory Error Checking . . . . . . . . . . . . . . . . . . =ECC
 Memory Speed. . . . . . . . . . . . . . . . . . . . . . . =50 NS
 L2 Cache Size . . . . . . . . . . . . . . . . . . . . . . =256KB
 L2 Cache Type . . . . . . . . . . . . . . . . . . . . . . =Asynchronous
 L2 Cache Parity . . . . . . . . . . . . . . . . . . . . . =Disabled
 Configuration Checksum. . . . . . . . . . . . . . . . . . =Failed

Then it gets to a windowed menu interface (which doesn't look like the typical OFW at all), but under "Administrative options" it's possible to choose "Invoke the Command Line Prompt", which gives the famous "ok" prompt.

AIX starts booting from the SCSI disk:

Trying..., fdisk0 Recalibrate failed.  The floppy drive is either missing,
improperly connected, or defective.
Failed
Trying..., hdisk0 Booting
Please wait while the system is booting
Boot device: /pci/scsi@2/disk@6,0  File and args:
   

 ******* Please define the System Console. *******

 Type a 1 and press Enter to use this terminal as the
  system console.

cvga0
+ swcons -c

Saving Base Customize Data to boot disk
Starting the sync daemon
Starting the error daemon
System initialization completed.
Starting Multi-user Initialization
 Performing auto-varyon of Volume Groups
 Activating all paging spaces
swapon: Paging device /dev/hd6 activated.
 Performing all automatic mounts

And here it hangs. Probably it tries to perform a NFS mount, which I don't have.
Anyways it's much further than QEMU currently gets, so it's definitely can be used as a reference.

I don't have a UW-SCSI cdrom drive to boot from the Powerstack II  media. But it can be netbooted via tftp.

Surprisingly booting the Solaris/PPC did not work out. The floppy is not recognized, tried to netboot SOLARIS.ELF from the cd got an interesting error:

Rebooting with command: boot /pci/ethernet@4:172.22.0.20,SOLARIS.ELF,172.22.134.1
Boot device: /pci/ethernet@4:172.22.0.250,SOLARIS.ELF,172.22.134.51  File and args:
Trying to get Internet/Ethernet Address ...
       Contact your system administrator to see if
       a Boot Host and network is setup correctly.

so, obviously after switching to the Little-Endian mode the Motorola network driver doesn't work anymore. Looks like in 1998 netbooting Solaris and Windows NT was not relevant for Motorola anymore, otherwise it would have been tested.

Overall it looks like Motorola did heavily modify the OFW. For instance, there are no hidden words. Which is nice. It should be possible to peek if it has any quirks in creation of the residual data. Or better to say it could have been. It is all a one single quirk, there is no residual data.

Initially I thought that this would be a perfect firmware which would boot both PReP images and the later OFW-compatible ones. But alas. After poking around, I googled and found a couple of mails on the NetBSD mailing list stating that:
 1. The firmware doesn't provide any residual data
 2. The firmware doesn't have the PCI, DMA and interrupt mapping properties in the device tree.
Looking at the code I see that the first point is clearly caused by the second one. In the OFW the residual data is generated from the device tree. The code was not removed, but Motorola forgot to add the properties. 

Which makes it a worst possible firmware.

But still it can boot the AIX from the supplied SCSI disk. This explains at least one reason for a custom AIX: the Motorola version should be able to live without the residual data.

Probably the developers were in a rush, so instead of fixing the firmware properties, they just added a hack to the OS. Maybe the OS department had more resources than the firmware one, or maybe the developers who were able to do Forth, were on vacation or fired.

The result is ugly, but I think every software developer has done something similar at least once.

Anyway now I have a sort of reference machine which can sort of boot AIX.

P.S. And by the way, if you wonder why I keep writing  "Powerstack II Utah" instead of just "Powerstack II". It turned out multiple machines called "Powerstack II" were produced. And indeed they are incompatible. More gory details in the Linux kernel sources.