FreeBSD 13-RELEASE install forgest SSD-based ZFS volume

[Scott Gasch]

Hi all,

I'm trying to upgrade a system from 12.2 to 13.0. The system has two ZFS
volumes: z and zssd. The former is a mirror set based on spinning platter
disks and the latter is a mirror set based on two ssd devices.

I've gotten as far as installing the new kernel and rebooting at which
point the system is screwed up because, on boot, zssd no longer exists.
Note: the kernel is 13.0 while userland is still 12.2 at this point.

I have backups for everything so my first inclination was to recreate
zssd. I did this including all volumes with mountpoints on it and rebooted
again. When it came back up it has lost zssd again.

I'm not sure how to proceed here. Any advice would be greatly
appreciated. I'd like to avoid a full reinstall if at all possible.

Scott

[Dean E Weimer via freebsd-questions]

Sent from my iPhone

Post by Scott Gasch
Hi all,
I'm trying to upgrade a system from 12.2 to 13.0. The system has two ZFS
volumes: z and zssd. The former is a mirror set based on spinning platter
disks and the latter is a mirror set based on two ssd devices.
I've gotten as far as installing the new kernel and rebooting at which
point the system is screwed up because, on boot, zssd no longer exists.
Note: the kernel is 13.0 while userland is still 12.2 at this point.
I have backups for everything so my first inclination was to recreate
zssd. I did this including all volumes with mountpoints on it and rebooted
again. When it came back up it has lost zssd again.
I'm not sure how to proceed here. Any advice would be greatly
appreciated. I'd like to avoid a full reinstall if at all possible.
Scott
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-questions

Have you tried doing an import of the volume after reboot?

zool import zssd

Not sure why it would forget it though I have a similar setup a ssd mirror that’s my boot drive and an hdd mirror that stores less used files files and don’t have any problems. Make sure your /boot/loader.conf has the zfs_load_enable option set and rc.conf has zfs_enable option set. I did my upgrade from source an installed it to a new boot dataset kernel and user land then rebooted after changing pools boot dataset option so I never ran with 13 kernel and 12.2 user land.

[andrew clarke]

Post by Scott Gasch
I'm trying to upgrade a system from 12.2 to 13.0. The system has two ZFS
volumes: z and zssd. The former is a mirror set based on spinning platter
disks and the latter is a mirror set based on two ssd devices.
I've gotten as far as installing the new kernel and rebooting at which
point the system is screwed up because, on boot, zssd no longer exists.
Note: the kernel is 13.0 while userland is still 12.2 at this point.

My experience upgrading to 13.0 was similar, but resolved itself after
upgrading userland and rebooting:

I have two ZFS pools (both mirrors): zroot and tank. FreeBSD boots from zroot.

I upgraded from 12.2 to 13.0 with "freebsd-update upgrade -r 13.0-RELEASE", ran
"freebsd-update install" then rebooted to load the 13.0 kernel. At this point
userland was still at 12.2, and tank was missing from "zpool status". Only
zroot was visible.

I didn't try to import tank manually.

After running "freebsd-update install" a second time to install 13.0 userland
and rebooting again, FreeBSD 13.0 booted from zroot and both it and tank was
visible once more with "zpool status", and from there it was business as usual.

I imagine I'd have been a bit stuck if /var/db/freebsd-update/ happened to be a
symlink to a location on the "tank" pool, because freebsd-update wouldn't be
able to complete the upgrade. But that wasn't the case, and spreading
the base system over multiple pools seems unwise anyway.

I don't know if any of this is helpful. If not you'll need to elaborate on
"the system is screwed up".

[Scott Gasch]

Thanks. It's as you say: when I rebooted with the 13.0 kernel and the 12.2
userland the /var mountpoint (and /home, and /root, and /usr/src, etc...)
sit on the zssd pool which has disappeared. The system comes up but my
home directory is gone and so is root's. I can login but the missing pool
means I can't finish upgrading userland to 13.0.

My first instinct was to restore /home which is when I found that creating
the missing pool anew does not survive a reboot in this state. Apparently
creating an ssd based mirror (or stripe, or single provider pool) with 12.2
zpool and creating some volumes on it with 12.2 zfs then rebooting the
machine causes the newly created pool to vanish again.

I'm considering mounting /var on the HDD zfs pool temporarily and trying to
freebsd-update -r 13.0-RELEASE upgrade again so that I can get userland to
13.0 on the hope that, in that configuration, I'll be able to finish the
update and then maybe restore the missing data from backups.

Thanks for any ideas and suggestions, much appreciated.

Scott

Post by Scott Gasch

Post by Scott Gasch
I'm trying to upgrade a system from 12.2 to 13.0. The system has two ZFS
volumes: z and zssd. The former is a mirror set based on spinning

platter

Post by Scott Gasch
disks and the latter is a mirror set based on two ssd devices.
I've gotten as far as installing the new kernel and rebooting at which
point the system is screwed up because, on boot, zssd no longer exists.
Note: the kernel is 13.0 while userland is still 12.2 at this point.

My experience upgrading to 13.0 was similar, but resolved itself after
I have two ZFS pools (both mirrors): zroot and tank. FreeBSD boots from zroot.
I upgraded from 12.2 to 13.0 with "freebsd-update upgrade -r
13.0-RELEASE", ran
"freebsd-update install" then rebooted to load the 13.0 kernel. At this point
userland was still at 12.2, and tank was missing from "zpool status". Only
zroot was visible.
I didn't try to import tank manually.
After running "freebsd-update install" a second time to install 13.0 userland
and rebooting again, FreeBSD 13.0 booted from zroot and both it and tank was
visible once more with "zpool status", and from there it was business as usual.
I imagine I'd have been a bit stuck if /var/db/freebsd-update/ happened to be a
symlink to a location on the "tank" pool, because freebsd-update wouldn't be
able to complete the upgrade. But that wasn't the case, and spreading
the base system over multiple pools seems unwise anyway.
I don't know if any of this is helpful. If not you'll need to elaborate on
"the system is screwed up".

[Ireneusz Pluta/wp.pl]

Post by Scott Gasch
Thanks. It's as you say: when I rebooted with the 13.0 kernel and the 12.2
userland the /var mountpoint (and /home, and /root, and /usr/src, etc...)
sit on the zssd pool which has disappeared. The system comes up but my
home directory is gone and so is root's. I can login but the missing pool
means I can't finish upgrading userland to 13.0.
My first instinct was to restore /home which is when I found that creating
the missing pool anew does not survive a reboot in this state. Apparently
creating an ssd based mirror (or stripe, or single provider pool) with 12.2
zpool and creating some volumes on it with 12.2 zfs then rebooting the
machine causes the newly created pool to vanish again.
I'm considering mounting /var on the HDD zfs pool temporarily and trying to
freebsd-update -r 13.0-RELEASE upgrade again so that I can get userland to
13.0 on the hope that, in that configuration, I'll be able to finish the
update and then maybe restore the missing data from backups.
Thanks for any ideas and suggestions, much appreciated.
Scott

from what you and the others say, what the system misses seem not to be an "SSD-based" pool, but
just a pool(s) other than it boots from. And manual zpool import of a missing pool just works
(zpool.cache format change?). So import it and proceed with `freebsd-update install` as usual.

Also, it is probably worth noting the below change:

[***@myhost:~/freebsd/git/src]{main}$ git log -U -1 --grep zpool.cache
commit a784185078e566103b7f8abffc7c0a4a1e813eb1
Author: Cy Schubert <***@FreeBSD.org>
Date:   Thu Aug 27 14:33:46 2020 +0000

    /etc/zfs/zpool.cache is the preferred (and new) location of zpool.cache.
    Check for it first. Only use /boot/zfs/zpool.cache if the /etc/zfs
    version is not found and good.

    Reported by:    avg
    Suggested by:   avg, kevans

Notes:
    svn path=/head/; revision=364867

diff --git a/libexec/rc/rc.d/zpool b/libexec/rc/rc.d/zpool
index 01028f8633ea..8aab58080a0a 100755
--- a/libexec/rc/rc.d/zpool
+++ b/libexec/rc/rc.d/zpool
@@ -20,9 +20,9 @@ zpool_start()
 {
    local cachefile

-   for cachefile in /boot/zfs/zpool.cache /etc/zfs/zpool.cache; do
+ for cachefile in /etc/zfs/zpool.cache /boot/zfs/zpool.cache; do
        if [ -r $cachefile ]; then
-           zpool import -c $cachefile -a -N
+         zpool import -c $cachefile -a -N && break
        fi
    done
 }


As I just practised on a vm machine, the /etc/zfs/zpool.cache installed by update seems to be a copy
of the prior /boot/zfs/zpool.cache. But they slightly differ this is what diff of zdb dumps of both
shows (however this is after full update, I missed a step to compare before userland update).

$ diff -u zdb.0 zdb.1
--- zdb.0       2021-04-15 20:04:59.296702000 +0200
+++ zdb.1       2021-04-15 20:05:01.421878000 +0200
@@ -2,10 +2,11 @@
     version: 5000
     name: 'z2'
     state: 0
-    txg: 4
+    txg: 165
     pool_guid: 2328267395261012403
+    errata: 0
     hostid: 1467704166
-    hostname: ''
+    hostname: 'v8'
     com.delphix:has_per_vdev_zaps
     vdev_children: 1
     vdev_tree:
@@ -47,9 +48,9 @@
     version: 5000
     name: 'zroot'
     state: 0
-    txg: 2172637
+    txg: 2175239
     pool_guid: 686285197683857337
-    hostid: 1467704166
+    errata: 0
     hostname: 'v8'
     com.delphix:has_per_vdev_zaps
     vdev_children: 1

Anyway, thank you guys for early adopting and giving me very useful pre-warning about what is going
to happen when I start upgrading my production systems :-).

[Scott Gasch]

Thank you, Ireneusz. You're definitely right about this -- I just noticed
that on a different freebsd box I had already upgraded to 13.0 has a
similar problem. It's boot device is UFS but it has a ZFS pool that it
uses to host backups. That machine didn't see the ZFS pool until I
followed you zpool import at which point everything was fine.

I really wish I had know about that before killing my zssd pool on the
other machine so I wouldn't be waiting on the restores at this point. But
at least it gave me a chance to test out my backup strategy.

Thanks for the help, everyone. Much appreciated.

Post by Scott Gasch

Post by Scott Gasch
Thanks. It's as you say: when I rebooted with the 13.0 kernel and the

12.2

Post by Scott Gasch
userland the /var mountpoint (and /home, and /root, and /usr/src, etc...)
sit on the zssd pool which has disappeared. The system comes up but my
home directory is gone and so is root's. I can login but the missing

pool

Post by Scott Gasch
means I can't finish upgrading userland to 13.0.
My first instinct was to restore /home which is when I found that

creating

Post by Scott Gasch
the missing pool anew does not survive a reboot in this state.

Apparently

Post by Scott Gasch
creating an ssd based mirror (or stripe, or single provider pool) with

12.2

Post by Scott Gasch
zpool and creating some volumes on it with 12.2 zfs then rebooting the
machine causes the newly created pool to vanish again.
I'm considering mounting /var on the HDD zfs pool temporarily and trying

to

Post by Scott Gasch
freebsd-update -r 13.0-RELEASE upgrade again so that I can get userland

to

Post by Scott Gasch
13.0 on the hope that, in that configuration, I'll be able to finish the
update and then maybe restore the missing data from backups.
Thanks for any ideas and suggestions, much appreciated.
Scott

from what you and the others say, what the system misses seem not to be an
"SSD-based" pool, but
just a pool(s) other than it boots from. And manual zpool import of a
missing pool just works
(zpool.cache format change?). So import it and proceed with
`freebsd-update install` as usual.
commit a784185078e566103b7f8abffc7c0a4a1e813eb1
Date: Thu Aug 27 14:33:46 2020 +0000
/etc/zfs/zpool.cache is the preferred (and new) location of
zpool.cache.
Check for it first. Only use /boot/zfs/zpool.cache if the /etc/zfs
version is not found and good.
Reported by: avg
Suggested by: avg, kevans
svn path=/head/; revision=364867
diff --git a/libexec/rc/rc.d/zpool b/libexec/rc/rc.d/zpool
index 01028f8633ea..8aab58080a0a 100755
--- a/libexec/rc/rc.d/zpool
+++ b/libexec/rc/rc.d/zpool
@@ -20,9 +20,9 @@ zpool_start()
{
local cachefile
- for cachefile in /boot/zfs/zpool.cache /etc/zfs/zpool.cache; do
+ for cachefile in /etc/zfs/zpool.cache /boot/zfs/zpool.cache; do
if [ -r $cachefile ]; then
- zpool import -c $cachefile -a -N
+ zpool import -c $cachefile -a -N && break
fi
done
}
As I just practised on a vm machine, the /etc/zfs/zpool.cache installed by
update seems to be a copy
of the prior /boot/zfs/zpool.cache. But they slightly differ this is what
diff of zdb dumps of both
shows (however this is after full update, I missed a step to compare
before userland update).
$ diff -u zdb.0 zdb.1
--- zdb.0 2021-04-15 20:04:59.296702000 +0200
+++ zdb.1 2021-04-15 20:05:01.421878000 +0200
@@ -2,10 +2,11 @@
version: 5000
name: 'z2'
state: 0
- txg: 4
+ txg: 165
pool_guid: 2328267395261012403
+ errata: 0
hostid: 1467704166
- hostname: ''
+ hostname: 'v8'
com.delphix:has_per_vdev_zaps
vdev_children: 1
@@ -47,9 +48,9 @@
version: 5000
name: 'zroot'
state: 0
- txg: 2172637
+ txg: 2175239
pool_guid: 686285197683857337
- hostid: 1467704166
+ errata: 0
hostname: 'v8'
com.delphix:has_per_vdev_zaps
vdev_children: 1
Anyway, thank you guys for early adopting and giving me very useful
pre-warning about what is going
to happen when I start upgrading my production systems :-).