Mounting from zfs:zroot/ROOT/default failed with error 2: unknown file system after upgrade to 10.1-RELEASE

Discussion:

Anton Eliasson

2014-11-18 22:05:25 UTC

Hi!
After upgrading a server from FreeBSD 10.0-RELEASE to 10.1-RELEASE it no
longer starts up. The bootloader shows its usual menu and then proceeds
to load zfs.ko, opensolaris.ko and some third module. Then kernel boots
and continues all the way to mountroot where it fails with the message
"Mounting from zfs:zroot/ROOT/default failed with error 2: unknown file
system." The entire message:
https://cloud.solid.lth.se/public.php?service=files&t=27c15a131ca63b19f619a7ee558fe8d1

Entering zfs:zroot, zfs:zroot/ROOT, zfs:zroot/ROOT/default or
zfs:anythingelse results in the same error.

FreeBSD 10.0 was installed this summer. I believe I used the installer's
guided procedure for installing on a mirrored ZFS root. Each drive has a
GPT table with a 512k freebsd-boot, a freebsd-swap and a freebsd-zfs
partition. No separate /boot.

There is no vfs.root.mountfrom directive in /boot/loader.conf and no /
in /etc/fstab. If I set vfs.root.mountfrom="zfs:zroot" it fails with the
same message but a slightly different path (zfs:zroot).

The motherboard is a Supermicro X9SRH-7TF. I can import and mount the
root filesystem with a FreeBSD live CD without problems.

Some config files pulled from a very similar installation (same server
model, installed at the same time). They should be more or less
identical on the server that won't boot:

/etc/rc.conf:

hostname="..."
keymap="swedish.iso.kbd"
# Use synchronous DHCP; pause boot until DHCP is completed.
ifconfig_ix0="SYNCDHCP"
sshd_enable="YES"
ntpd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"

# IPFW
[...]

/boot/loader.conf:

zfs_load="YES"
geom_mirror_load="YES"

/etc/fstab:

fdesc /dev/fd fdescfs rw 0 0
/dev/mirror/swap none swap sw 0 0

Thanks!

--
Best Regards,
Anton Eliasson

Trond Endrestøl

2014-11-19 07:22:28 UTC

Permalink

Post by Anton Eliasson
Hi!
After upgrading a server from FreeBSD 10.0-RELEASE to 10.1-RELEASE it no
longer starts up. The bootloader shows its usual menu and then proceeds to
load zfs.ko, opensolaris.ko and some third module. Then kernel boots and
continues all the way to mountroot where it fails with the message "Mounting
from zfs:zroot/ROOT/default failed with error 2: unknown file system." The
https://cloud.solid.lth.se/public.php?service=files&t=27c15a131ca63b19f619a7ee558fe8d1
Entering zfs:zroot, zfs:zroot/ROOT, zfs:zroot/ROOT/default or zfs:anythingelse
results in the same error.
FreeBSD 10.0 was installed this summer. I believe I used the installer's
guided procedure for installing on a mirrored ZFS root. Each drive has a GPT
table with a 512k freebsd-boot, a freebsd-swap and a freebsd-zfs partition. No
separate /boot.
There is no vfs.root.mountfrom directive in /boot/loader.conf and no / in
/etc/fstab. If I set vfs.root.mountfrom="zfs:zroot" it fails with the same
message but a slightly different path (zfs:zroot).
The motherboard is a Supermicro X9SRH-7TF. I can import and mount the root
filesystem with a FreeBSD live CD without problems.
Some config files pulled from a very similar installation (same server model,
installed at the same time). They should be more or less identical on the
hostname="..."
keymap="swedish.iso.kbd"
# Use synchronous DHCP; pause boot until DHCP is completed.
ifconfig_ix0="SYNCDHCP"
sshd_enable="YES"
ntpd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"
# IPFW
[...]
zfs_load="YES"
geom_mirror_load="YES"
fdesc /dev/fd fdescfs rw 0 0
/dev/mirror/swap none swap sw 0 0
Thanks!

According to the screenshot, the bootfs property on zroot is set to
zroot/ROOT/default. Does that dataset exist?

While at the loader prompt, what does these commands tell you?

lszfs zroot
lszfs zroot/ROOT

If you see some other potential dataset, try setting currdev using
this as a template before booting:

set currdev="zfs:zroot/ROOT/some-other-dataset:"
^ ^
These :'s are important.

Next, unload the old stuff, and try to load the new stuff:

unload
load /boot/kernel/kernel
load /boot/kernel/opensolaris.ko
load /boot/kernel/zfs.ko

Boot into single user mode:

boot -s

Remember, the console uses US keyboard layout, set up by the firmware.

Anton Eliasson

2014-11-19 16:10:25 UTC

Permalink

Post by Trond EndrestÃ¸l

According to the screenshot, the bootfs property on zroot is set to
zroot/ROOT/default. Does that dataset exist?
While at the loader prompt, what does these commands tell you?
lszfs zroot

$MOS
$FREE
$ORIGIN
ROOT
tmp
usr
var

Post by Trond EndrestÃ¸l
lszfs zroot/ROOT

default

And `lszfs zroot/ROOT/default` shows nothing.

Post by Trond EndrestÃ¸l
If you see some other potential dataset, try setting currdev using
set currdev="zfs:zroot/ROOT/some-other-dataset:"
^ ^
These :'s are important.

`show currdev` shows that it is already set to zfs:zroot/ROOT/default:
(with a colon at the end)

Post by Trond EndrestÃ¸l
unload
load /boot/kernel/kernel
load /boot/kernel/opensolaris.ko
load /boot/kernel/zfs.ko

It looks like it loads the kernel, but then it fails on the following
two commands because the kernel is not loaded:
https://cloud.solid.lth.se/public.php?service=files&t=5028527feac7982c76206b691212d15b
https://cloud.solid.lth.se/public.php?service=files&t=0c0d50aac583167a11a8eaa4de5f08ce

Doing boot -s after that gives the same error as before.

However, loading the old kernel with the following commands works better:

unload
load /boot/kernel.old/kernel
load /boot/kernel.old/opensolaris.ko
load /boot/kernel/zfs.ko
lsmod

Screenshots:
https://cloud.solid.lth.se/public.php?service=files&t=9c6c1bbc96b2c816aae05d2718215e2a
https://cloud.solid.lth.se/public.php?service=files&t=8aee8df4bacf65b9ceae02fa65b53711

Issuing boot -s after that shows some text and then suddenly reboots the
computer. I can see "ZFS filesystem version 5" in the output so it looks
like it is successfully loaded.

Selecting the old kernel in the boot menu and trying to boot normally
fails because init is not found:
https://cloud.solid.lth.se/public.php?service=files&t=81fd35bb6348ae7426553c2f59fb2a18

Could this be a mismatch in kernel and kernel module versions? How would
I fix that?

--
Best Regards,
Anton Eliasson

Trond Endrestøl

2014-11-19 17:17:50 UTC

Permalink

Post by Anton Eliasson

Post by Trond EndrestÃ¸l

According to the screenshot, the bootfs property on zroot is set to
zroot/ROOT/default. Does that dataset exist?
While at the loader prompt, what does these commands tell you?
lszfs zroot

$MOS
$FREE
$ORIGIN
ROOT
tmp
usr
var

Post by Trond EndrestÃ¸l
lszfs zroot/ROOT

default
And `lszfs zroot/ROOT/default` shows nothing.

Post by Trond EndrestÃ¸l
If you see some other potential dataset, try setting currdev using
set currdev="zfs:zroot/ROOT/some-other-dataset:"
^ ^
These :'s are important.

`show currdev` shows that it is already set to zfs:zroot/ROOT/default: (with a
colon at the end)

Post by Trond EndrestÃ¸l
unload
load /boot/kernel/kernel
load /boot/kernel/opensolaris.ko
load /boot/kernel/zfs.ko

It looks like it loads the kernel, but then it fails on the following two
https://cloud.solid.lth.se/public.php?service=files&t=5028527feac7982c76206b691212d15b
https://cloud.solid.lth.se/public.php?service=files&t=0c0d50aac583167a11a8eaa4de5f08ce
Doing boot -s after that gives the same error as before.
unload
load /boot/kernel.old/kernel
load /boot/kernel.old/opensolaris.ko
load /boot/kernel/zfs.ko
lsmod
https://cloud.solid.lth.se/public.php?service=files&t=9c6c1bbc96b2c816aae05d2718215e2a
https://cloud.solid.lth.se/public.php?service=files&t=8aee8df4bacf65b9ceae02fa65b53711
Issuing boot -s after that shows some text and then suddenly reboots the
computer. I can see "ZFS filesystem version 5" in the output so it looks like
it is successfully loaded.

To break into the kernel debugger, enter this before booting the
kernel:

set debug.debugger_on_panic="1"

Judging from the screenshots you've provided and from what you wrote
in your reply, it seems vfs.root.mountfrom is overridden somehow.
/boot/loader.conf would be a natural candidate.

If the bootfs property is indeed set to zroot/ROOT/default, then it's
very strange that the kernel would proclaim:

Trying to mount root from zfs:zroot []...

instead of:

Trying to mount root from zfs:zroot/ROOT/default []...

And it's even stranger that nothing appears on lszfs' radar when asked
to list zroot/ROOT/default.

Post by Anton Eliasson
Selecting the old kernel in the boot menu and trying to boot normally fails
https://cloud.solid.lth.se/public.php?service=files&t=81fd35bb6348ae7426553c2f59fb2a18
Could this be a mismatch in kernel and kernel module versions? How would I fix
that?

If you can boot from a recent snapshot image or live image, and import
the zpool read-only, you could walk around and see if
zroot/ROOT/default/boot/loader.conf contains anything strange.

This zpool command would allow you to go spelunking inside /zroot:

zpool import -o readonly=on -f -R /zroot zroot

You might need to mount readonly any dataset you wish to examine.

If you see the need for changing any of the files, you must export the
zpool and re-import it readwrite, i.e. leave out the -o readonly=on
option.

I admit I'm in unknown territory, even if I have managed multiple ZFS
systems over the past years, so be careful.

Anton Eliasson

2014-11-19 17:47:22 UTC

Permalink

Post by Trond EndrestÃ¸l
To break into the kernel debugger, enter this before booting the
set debug.debugger_on_panic="1"
in your reply, it seems vfs.root.mountfrom is overridden somehow.
/boot/loader.conf would be a natural candidate.
If the bootfs property is indeed set to zroot/ROOT/default, then it's
Trying to mount root from zfs:zroot []...
Trying to mount root from zfs:zroot/ROOT/default []...
And it's even stranger that nothing appears on lszfs' radar when asked
to list zroot/ROOT/default.

If you can boot from a recent snapshot image or live image, and import
the zpool read-only, you could walk around and see if
zroot/ROOT/default/boot/loader.conf contains anything strange.
zpool import -o readonly=on -f -R /zroot zroot
You might need to mount readonly any dataset you wish to examine.
If you see the need for changing any of the files, you must export the
zpool and re-import it readwrite, i.e. leave out the -o readonly=on
option.
I admit I'm in unknown territory, even if I have managed multiple ZFS
systems over the past years, so be careful.

Hi!
I had forgot to remove the vfs.root.mountfrom that I experimented with.
This probably caused the failure to load init.

I got it to work by chrooting into the system from a live cd and rolling
back the upgrade with `freebsd-update rollback`. Then it booted fine.
Now freebsd-version shows 10.0-RELEASE-p12 and uname -r shows
10.1-RELEASE. It looks a bit strange, but at least it works for now.

I may have mixed up the commands used to upgrade the system. Is the
following the correct procedure for doing a major upgrade (with a stock
kernel)?

freebsd-update -r 10.1-RELEASE upgrade
freebsd-update install
(reboot)
freebsd-update install
(rebuild ports)
(done)

Thanks for your help!

--
Best Regards,
Anton Eliasson

Trond Endrestøl

2014-11-19 18:29:16 UTC

Permalink

If you can boot from a recent snapshot image or live image, and import
the zpool read-only, you could walk around and see if
zroot/ROOT/default/boot/loader.conf contains anything strange.
zpool import -o readonly=on -f -R /zroot zroot
You might need to mount readonly any dataset you wish to examine.
If you see the need for changing any of the files, you must export the
zpool and re-import it readwrite, i.e. leave out the -o readonly=on
option.
I admit I'm in unknown territory, even if I have managed multiple ZFS
systems over the past years, so be careful.

Hi!
I had forgot to remove the vfs.root.mountfrom that I experimented with. This
probably caused the failure to load init.
I got it to work by chrooting into the system from a live cd and rolling back
the upgrade with `freebsd-update rollback`. Then it booted fine. Now
freebsd-version shows 10.0-RELEASE-p12 and uname -r shows 10.1-RELEASE. It
looks a bit strange, but at least it works for now.
I may have mixed up the commands used to upgrade the system. Is the following
the correct procedure for doing a major upgrade (with a stock kernel)?
freebsd-update -r 10.1-RELEASE upgrade
freebsd-update install
(reboot)
freebsd-update install
(rebuild ports)
(done)

Sadly, I wouldn't know.

I use the Subversion base/stable source tree for the OS, be it
base/stable/{8,9,10}, and the ports collection (ports/head) for any
3rd party SW.

On my ZFS systems I always install a new world and kernel into a new
dataset, zroot/ROOT/YYYYMMDD-rRRRRRR, (maybe I'll eventually settle
for a snapshot + clone + install dance), adjust the bootfs property on
the root pool, copy select files from /boot, /etc, and /root to the
new dataset, let the new dataset inherit the mountpoint (legacy) from
its parent, and reboot into single user mode to perform basic
maintenance such as mergemaster and make delete-old, and maybe I even
upgrade any outdated ports using portupgrade and a homegrown shell
script, before I cold boot the system to verify the system's ability
to bootstrap itself, and to allow the harddrives to do their power-on
maintanance.

In stable/10 and head, you can now have options ZFS in a custom kernel
configuration, removing the need to load opensolaris.ko and zfs.ko
along with the kernel. The freebsd-update route allows only for the
GENERIC kernel and a precompiled world. :-/

Thanks for your help!

NP. :-)