Determining process preventing umount of busy partition

[Polytropon]

I'd like to ask how to determine which process (or program) keeps a partition
in state "busy" so that umount will refuse to unmount this partition. I found
this when going into SUM for checking and maintenance, so I think it would be
good to check which program still accesses files on a specific partition
allthough it should already be terminated due to the different "stop" mechanisms
run for the services in /etc/rc.d and /usr/local/etc/rc.d respectively, which
is performed by init, if I understood this correctly.

Example:

% shutdown now

... going SUM, starting sh ...

# umount /home
# umount /tmp
# umount /var
# umount /usr
umount: unmount of /usr failed: Device busy
# umount -f /usr
# mount -o ro /
# fsck ... blah blah ...

It would be good to be able to check why the partition is in state "busy" and
possible terminate / kill processes that cause this. Using the force (-f) seems
to be unneccessarily unfriendly. =^_^=

Thanks for suggestions!

[Robert Huff]

Post by Polytropon
I'd like to ask how to determine which process (or program) keeps
a partition in state "busy" so that umount will refuse to unmount
this partition.

The traditional tool for doing this is sysutils/lsof.
(Please let me know if it compiles.)


Robert Huff

[Polytropon]

Post by Robert Huff

Post by Polytropon
I'd like to ask how to determine which process (or program) keeps
a partition in state "busy" so that umount will refuse to unmount
this partition.

The traditional tool for doing this is sysutils/lsof.
(Please let me know if it compiles.)

Sadly not, but my Ports tree is not up to date, so I tried to compile
it in PORTVERSION=4.82A, with this error:

===> Building for lsof-4.82A,3
(cd lib; make DEBUG="-O2" CFGF="-fno-strict-aliasing -pipe -march=pentium4 -march=pentium4 -DHASEFFNLINK=i_effnlink -DHASF_VNODE -DHASCPUMASK_T -DHASSBSTATE -DHAS_KVM_VNODE -DHAS_UFS1_2 -DHAS_NO_SI_UDEV -DHAS_SI_PRIV -DHAS_SYS_SX_H -DHAS_ZFS -DHAS_V_LOCKF -DHAS_LOCKF_ENTRY -DFREEBSDV=7000 -DHASFDESCFS=2 -DHASPSEUDOFS -DHASNULLFS -DHASIPv6 -DHAS_STRFTIME -DLSOF_VSTR=\"7.0-STABLE\"")
cc -fno-strict-aliasing -pipe -march=pentium4 -march=pentium4 -DHASEFFNLINK=i_effnlink -DHASF_VNODE -DHASCPUMASK_T -DHASSBSTATE -DHAS_KVM_VNODE -DHAS_UFS1_2 -DHAS_NO_SI_UDEV -DHAS_SI_PRIV -DHAS_SYS_SX_H -DHAS_ZFS -DHAS_V_LOCKF -DHAS_LOCKF_ENTRY -DFREEBSDV=7000 -DHASFDESCFS=2 -DHASPSEUDOFS -DHASNULLFS -DHASIPv6 -DHAS_STRFTIME -DLSOF_VSTR="7.0-STABLE" -I/usr/src/sys -O2 -c ckkv.c
In file included from ckkv.c:33:
../machine.h:62: error: redefinition of typedef 'cpumask_t'
/usr/src/sys/sys/types.h:146: error: previous declaration of 'cpumask_t' was here
*** Error code 1

Stop in /usr/ports/sysutils/lsof/work/lsof_4.82A.freebsd/lib.
*** Error code 1

Stop in /usr/ports/sysutils/lsof/work/lsof_4.82A.freebsd.
*** Error code 1

Stop in /usr/ports/sysutils/lsof.




After I updated my Ports (just right now) I saw that lsof didn't change,
still same version number.




Installation via pkg_add -r worked without problems.

# pkg_add -r lsof
Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-7-stable/Latest/lsof.tbz... Done.

I've got lsof-4.82A,3 now. The manpage reveals that this seems to be
exactly what I've been searching for, so "lsof | grep usr" should to
the trick. Thanks!

[Polytropon]

Replying to my own message... I'm so stupid: How do I *use* lsof which
is /usr/local/sbin/lsof when actually trying to umount /usr? Can I
put a copy of it into, let's say, /root/bin? I've checked library
dependencies:

# which lsof | xargs ldd
/usr/local/sbin/lsof:
libkvm.so.4 => /lib/libkvm.so.4 (0x280a2000)
libc.so.7 => /lib/libc.so.7 (0x280aa000)

So this is on the / partition. This should work, am I right?

[Matthew Seaman]

Post by Polytropon
Replying to my own message... I'm so stupid: How do I *use* lsof which
is /usr/local/sbin/lsof when actually trying to umount /usr? Can I
put a copy of it into, let's say, /root/bin? I've checked library
# which lsof | xargs ldd
libkvm.so.4 => /lib/libkvm.so.4 (0x280a2000)
libc.so.7 => /lib/libc.so.7 (0x280aa000)
So this is on the / partition. This should work, am I right?

Don't use lsof for that then. Use fstat(1) which is part of the base
system:

# fstat -f /usr

Cheers,

Matthew

[Polytropon]

Post by Matthew Seaman
Don't use lsof for that then. Use fstat(1) which is part of the base
# fstat -f /usr

Cool! I didn't know about how to use fstat for *this* purpose.

[Matthew Seaman]

Post by Matthew Seaman

Post by Polytropon
Replying to my own message... I'm so stupid: How do I *use* lsof which
is /usr/local/sbin/lsof when actually trying to umount /usr? Can I
put a copy of it into, let's say, /root/bin? I've checked library
# which lsof | xargs ldd
libkvm.so.4 => /lib/libkvm.so.4 (0x280a2000)
libc.so.7 => /lib/libc.so.7 (0x280aa000)
So this is on the / partition. This should work, am I right?

Don't use lsof for that then. Use fstat(1) which is part of the base
# fstat -f /usr

Must stop replying to these things before I've had enough coffee.
fstat(1) is of course installed as /usr/bin/fstat so unmounting /usr
will seriously cramp its style. As you say -- there are no shlib
dependencies on stuff under /usr so just slap a copy somewhere on
the root partition and away you go.

Cheers,

Matthew, who avoids having to think about any of this stuff
by the simple expedient of not having a separate /usr
partition.

[Tim Judd]

Post by Polytropon
I'd like to ask how to determine which process (or program) keeps a partition
in state "busy" so that umount will refuse to unmount this partition. I found
this when going into SUM for checking and maintenance, so I think it would be
good to check which program still accesses files on a specific partition
allthough it should already be terminated due to the different "stop" mechanisms
run for the services in /etc/rc.d and /usr/local/etc/rc.d respectively, which
is performed by init, if I understood this correctly.
% shutdown now
... going SUM, starting sh ...
# umount /home
# umount /tmp
# umount /var
# umount /usr
umount: unmount of /usr failed: Device busy
# umount -f /usr
# mount -o ro /
# fsck ... blah blah ...
It would be good to be able to check why the partition is in state "busy" and
possible terminate / kill processes that cause this. Using the force (-f) seems
to be unneccessarily unfriendly. =^_^=
Thanks for suggestions!

Most commonly for me is because my $PWD (or CWD) is in the filesystem i
intend to umount
so as a habit now, i move myself to the root partition (when logged in
as root) via the following, and assuming I want to umount /usr


# umount /usr
umount: unmount of /usr failed: Device busy
# cd
# umount /usr


cd, with no arguments, move you to ~ (aka $HOME)

--Tim

[Polytropon]

First of all, I checked both lsof's and fstat's output: NOTHING seems to
have a file open in the /usr partition. Very strange. Of course, I've tried
the copies of both tools in /root/bin so they don't cause any access on /usr
theirselves.

Post by Tim Judd
Most commonly for me is because my $PWD (or CWD) is in the filesystem i
intend to umount

I've checked this: In SUM, $CWD was /, and root's $HOME is /root on
the / partition. Users' home directories are on /home which is separated
from /usr (and can be unmounted without problems). At no time, a $CWD
was on /usr partition.

Post by Tim Judd
so as a habit now, i move myself to the root partition (when logged in
as root) via the following, and assuming I want to umount /usr
# umount /usr
umount: unmount of /usr failed: Device busy
# cd
# umount /usr
cd, with no arguments, move you to ~ (aka $HOME)

Which would be /root in case of SUM.



As I said, very strange...

[Mel]

Post by Polytropon
First of all, I checked both lsof's and fstat's output: NOTHING seems to
have a file open in the /usr partition. Very strange. Of course, I've tried
the copies of both tools in /root/bin so they don't cause any access on
/usr theirselves.

Post by Tim Judd
Most commonly for me is because my $PWD (or CWD) is in the filesystem i
intend to umount

I've checked this: In SUM, $CWD was /, and root's $HOME is /root on
the / partition. Users' home directories are on /home which is separated
from /usr (and can be unmounted without problems). At no time, a $CWD
was on /usr partition.

Post by Tim Judd
so as a habit now, i move myself to the root partition (when logged in
as root) via the following, and assuming I want to umount /usr
# umount /usr
umount: unmount of /usr failed: Device busy
# cd
# umount /usr
cd, with no arguments, move you to ~ (aka $HOME)

Which would be /root in case of SUM.
As I said, very strange...

Is this a one-time event or 100% reproducable?
A likely scenario is:
- You have squid running
- You have rc_shutdowntimeout at default (30 seconds)
- rc hits the watchdog while squid is being shutdown
- you unmount
- get busy
- call fstat at which point squid has been shutdown.

Replace squid with anything that takes 30+ seconds to shutdown. Allthough,
they would probably already fail at umount /var. Squid with defaults is fully
contained in /usr/local.

[Polytropon]

Post by Mel
Is this a one-time event or 100% reproducable?

I've tried it several times, it can always be reproduced.

Post by Mel
- You have squid running
- You have rc_shutdowntimeout at default (30 seconds)

I'm not sure if this setting (?) will have an effect after trying
the umount operation in SUM. Even if umount is retried after a
several time, /usr is still "busy".

Post by Mel
- rc hits the watchdog while squid is being shutdown

No, nothing running. All applications have terminated.

Post by Mel
- you unmount
- get busy
- call fstat at which point squid has been shutdown.

I've used fstat and lsof to check for open files on /usr, nix,
nada, nitshewo.

Post by Mel
Replace squid with anything that takes 30+ seconds to shutdown. Allthough,
they would probably already fail at umount /var. Squid with defaults is fully
contained in /usr/local.

I can't imagine which application should still be running when nothing
on /usr is accessed (lsof, fstat); I'll check on running applications
using ps.

[Mel]

On Mon, 16 Feb 2009 10:44:02 -0900, Mel

Post by Mel
Is this a one-time event or 100% reproducable?

I've tried it several times, it can always be reproduced.

Post by Mel
- You have squid running
- You have rc_shutdowntimeout at default (30 seconds)

I'm not sure if this setting (?) will have an effect after trying
the umount operation in SUM. Even if umount is retried after a
several time, /usr is still "busy".

To explain the setting:
If the shutdowntimeout is hit, rc stops killing daemons like it's supposed to
do (effectively, stops calling rc.d scripts with SHUTDOWN keyword).

This is weird, though. New theories (where are Chase, Cameron and Foreman when
you need them!):

fstat is lying, instead use:
fstat -f /usr -m -v

You have a mount on top of /usr, ie.: /usr/local or /usr/ports.

[Polytropon]

Post by Mel
This is weird, though. New theories (where are Chase, Cameron and Foreman when

Spying around in someone else's house. :-)

Post by Mel
fstat -f /usr -m -v

Well, I've taken that pill. This is the result:

# /root/bin/fstat -m -v -f /usr
USER CMD PID FD MOUNT INUM MODE SZ|DV R/W
# _

It shows NOTHING. I have made a copy of fstat binary in /root/bin,
which is possible because all needed libs are in /.

Furthermore, I've carefully studied the output of "ps ax" and even
of "top -t", but as well, nothing that indicates some activity on
/usr...

Post by Mel
You have a mount on top of /usr, ie.: /usr/local or /usr/ports.

No. From /etc/fstab:

# Device Mountpoint FStype Options Dump Pass#
# ----------- --------------------- ------ ------------- ----- -----
/dev/ad0s1b none swap sw 0 0
/dev/ad0s1a / ufs rw 1 1
/dev/ad0s1d /tmp ufs rw 2 2
/dev/ad0s1e /var ufs rw 2 2
/dev/ad0s1f /usr ufs rw 2 2
/dev/ad0s1g /export/home ufs rw 2 2

These are the only partitions on ad0. /usr has its own partition,
nothing mounted "on top" of it. (You mentioned a valid point: I
sometimes have another disk mounted inside /export/home, and I
cannot umount /export/home while this partition is mounted. But
that's not the case here.)



This is REALLY strange, I should get a whiteboard, some pens and
start making a drawing of the symptoms, until Dr. Cuddy tells me
not to do so. :-)

[Mel]

On Tue, 17 Feb 2009 13:30:55 -0900, Mel

Post by Mel
This is weird, though. New theories (where are Chase, Cameron and Foreman

Spying around in someone else's house. :-)

Post by Mel
fstat -f /usr -m -v

# /root/bin/fstat -m -v -f /usr
USER CMD PID FD MOUNT INUM MODE SZ|DV R/W
# _
It shows NOTHING. I have made a copy of fstat binary in /root/bin,
which is possible because all needed libs are in /.

Well, that rules that out.

Furthermore, I've carefully studied the output of "ps ax" and even
of "top -t", but as well, nothing that indicates some activity on
/usr...

Post by Mel
You have a mount on top of /usr, ie.: /usr/local or /usr/ports.

# Device Mountpoint FStype Options Dump
Pass# # ----------- --------------------- ------ ------------- -----
----- /dev/ad0s1b none swap sw 0
0 /dev/ad0s1a / ufs rw 1
1 /dev/ad0s1d /tmp ufs rw 2
2 /dev/ad0s1e /var ufs rw 2 2
/dev/ad0s1f /usr ufs rw 2 2
/dev/ad0s1g /export/home ufs rw 2 2
These are the only partitions on ad0. /usr has its own partition,
nothing mounted "on top" of it. (You mentioned a valid point: I
sometimes have another disk mounted inside /export/home, and I
cannot umount /export/home while this partition is mounted. But
that's not the case here.)

Can you show mount -p before trying to unmount /usr? On the off-chance /export
or /export/home is really a symlink to /usr/home (mount -p shows realpath(3)
for mounts).

This is REALLY strange, I should get a whiteboard, some pens and
start making a drawing of the symptoms, until Dr. Cuddy tells me
not to do so. :-)

Lol, I'm actually enclined to talk to Wilson, since this smells like something
up his alley (and free lunch).

[Polytropon]

Post by Mel
Can you show mount -p before trying to unmount /usr? On the off-chance /export
or /export/home is really a symlink to /usr/home (mount -p shows realpath(3)
for mounts).

Hm, I keep /home out of /usr, so there's only a symlink (for the
obvious compatibility reasons) /home@ -> export/home.

BUT, and now the big surprise, maybe a possibility:

% mount -p
/dev/ad0s1a / ufs rw 1 1
devfs /dev devfs rw 0 0
/dev/ad0s1d /tmp ufs rw 2 2
/dev/ad0s1e /var ufs rw 2 2
/dev/ad0s1f /usr ufs rw 2 2
/dev/ad0s1g /export/home ufs rw 2 2
devfs /var/named/dev devfs rw 0 0
linprocfs /usr/compat/linux/proc linprocfs rw 0 0
^^^^ Haha!!!

It seems that the linprocfs prevents umounting of /usr because its
mountpoint /usr/compat/linux/proc is de facto INSIDE /usr.

I've checked this while in SUM: When umounting linprocfs prior to
/usr, no problems occur.

I'll speak to Mr. Tritter so he can stop his investigations. Seems
that we found the reason.

Thanks for your help.

Seems that I'm too stupid to own a computer. :-)

[Wojciech Puchar]

Post by Polytropon
that we found the reason.
Thanks for your help.
Seems that I'm too stupid to own a computer. :-)

i wish more people will be as stupid as you ;)