My main workstation, an amd64 system, runs with two SATA disks. However one of them, hosting most of the Gentoo system seemed to have a few problems. Time to upgrade.
Current setup :
sda1 : /windows (games)
sda2 : /share (network drive)
sdb1 : /
sdb2 : /home
sdb3 : /space
sdb4 : swap
Since there wasn't much space left anyway, I bought a new 200GiB disk and added it to the machine. My Asus K8N-E motherboard still has 4 SATA connectors that can be used for RAID, or apparently for standalone disks.
While I was at it, I burned a fresh Gentoo disk. A clean install every now and then can't hurt.
So theoretically, the new disk should be partitionned pretty much like sdb currently is. Then I'll have three disks on the machine until one dies or stops acting up.
Step one, plugging it in.
As expected, the "extra" SATA interfaces work as advertised. Unexpectedly though, the new disk now pops up first in the BIOS list. Apparently I can boot from a blank HD, a floppy or a CD.
A bit of rewiring later, I can boot again. I'd rather have the disk with the XP partition first, I'm not sure how happy Windows it would be if it wasn't. Maybe it matters, maybe not.
Anyway, I'm installing a fresh Linux system so it's no big deal.
Once inside the Gentoo install CD, everything goes smoothly until I setup Grub. Then there's something wrong which I don't quite get, just a general feeling.
As expected, the machine won't boot.
After a lot of head scratching, it dawns on me that the machine sees the disks differently whether it's through Linux or through the BIOS (or through Grub which "kind of" uses the BIOS).
In grub notation, the BIOS sees this :
hd0: new 200 GiB disk
hd1: old linux disk
hd2: old Windows disk
And Linux sees this :
hd0: old Windows disk
hd1: new 200 GiB disk
hd2: old linux disk
I was first blindly relying on Linux to locate the disks relative to one another and needless to see that the whole thing ended up in a mess. The boot time editing of the grub configuration finally saved me by letting me poke around (grub is ugly and not fun and all but it does have its redeeming features, dynamic boot parameter editing has been a life saver more times than I can count, whereas LiLo would die and remain dead in the same conditions).
So finally the system boots. Or would have booted if I hadn't forgotten a digit at the end of a disklabel (disklabels bay be great because they're unique but a mouthful of hex isn't user friendly... couldn't we just name the disk ?) in the fstab (yes, well, it was late). So I rebooted the CD again, fixed that, fixed the grub.conf while I was at it (at least the linux booting bit, I'll get to Windows later) and restarted.
And it all works as advertised.
So out of spite, after finalising a few things (users and such), I issued an "emerge gnome" and went to buy groceries.
Why the BIOS and the system disagree on the disk order shall remain a mystery.