Discussion:
OpenAFS cache manager cold vs warm shutdown
Mark Vitale
2014-07-03 21:45:47 UTC
Permalink
(cross-posted to openafs-devel and port-solaris)

The AFS Unix cache manager has two ways to shutdown:

- 'afsd -shutdown' requires that /afs already be umounted; it then sets afs_cold_shutdown=1 and calls afs_shutdown().

- 'umount /afs' also calls afs_shutdown() on most platforms. Some of them set afs_cold_shutdown=1 first, while others do not.

If afs_shutdown() is called "warm" (afs_cold_shutdown==0), the shutdown logic skips the clearing and releasing of some resources. I see no rhyme or reason to which resources AFS leaves unreleased. Nor do I understand the (possibly historical) reason for why there is a distinction between cold and warm shutdown.

For example, on Solaris, umount /afs currently performs a "warm" shutdown. This makes it effectively impossible (catch-22) to perform a "cold" shutdown.

Then there is the question of when it is safe to rmmod/modunload the libafs kernel module. Does warm or cold shutdown affect the answer to this question?

Can anyone shed some insight on this situation?

thanks,
--
Mark Vitale
***@sinenomine.net
Russ Allbery
2014-07-03 22:01:04 UTC
Permalink
Post by Mark Vitale
If afs_shutdown() is called "warm" (afs_cold_shutdown==0), the shutdown
logic skips the clearing and releasing of some resources. I see no
rhyme or reason to which resources AFS leaves unreleased. Nor do I
understand the (possibly historical) reason for why there is a
distinction between cold and warm shutdown.
For example, on Solaris, umount /afs currently performs a "warm"
shutdown. This makes it effectively impossible (catch-22) to perform a
"cold" shutdown.
Then there is the question of when it is safe to rmmod/modunload the
libafs kernel module. Does warm or cold shutdown affect the answer to
this question?
My experience on Linux is that cold shutdowns are bad news, and that,
following a cold shutdown, AFS on that system is probably hosed and will
never recover without a reboot.

See, for example:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=608173 (RT #128838)
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=732072
--
Russ Allbery (***@eyrie.org) <http://www.eyrie.org/~eagle/>
Andrew Deason
2014-07-04 00:23:17 UTC
Permalink
On Thu, 3 Jul 2014 21:45:47 +0000
Post by Mark Vitale
(cross-posted to openafs-devel and port-solaris)
I don't think this has anything to do with Solaris, but I'll keep it on
the cc list for now so this doesn't just look responseless. (And in case
this came from some Solaris historical artifact or something.)
Post by Mark Vitale
- 'afsd -shutdown' requires that /afs already be umounted; it then
sets afs_cold_shutdown=1 and calls afs_shutdown().
- 'umount /afs' also calls afs_shutdown() on most platforms. Some of
them set afs_cold_shutdown=1 first, while others do not.
If afs_shutdown() is called "warm" (afs_cold_shutdown==0), the
shutdown logic skips the clearing and releasing of some resources. I
see no rhyme or reason to which resources AFS leaves unreleased. Nor
do I understand the (possibly historical) reason for why there is a
distinction between cold and warm shutdown.
My understanding is that a WARM shutdown is via 'umount' (i.e., the
normal way), and a COLD shutdown is via afsd -shutdown. That is the
definition of those terms; that's all they mean (or maybe the defining
factor is "if /afs is mounted" or some other
similar-yet-technically-different distinction). Note that this (and
below) just comes from my own experience with the code; not from any
proper documented or otherwise authoritative source.

Normally you can never shutdown via 'afsd -shutdown' while /afs is
mounted. The reason 'afsd -shutdown', and thus, the COLD shutdown
procedure, exists at all is:

- If mounting /afs fails in the middle of initialization, you can't
umount /afs. So you can 'afsd -shutdown' instead to de-initialize
some things that came up before afs was mounted.

- You can run some of the client daemons without actually mounting /afs
in "normal" operation. I'm not sure if I've ever done this, but I
have a vague recollection from some code comment or post somewhere
referencing running the nfs xlator PAG manager this way. Maybe there
are other reasons, other ways.

The reason that some cleanup procedures are different is because during
the normal WARM shutdown procedure, we assume some daemons or other
things will cleanup after themselves; but during a COLD procedure we're
not sure if they're really there or functioning properly, so there's
some extra cleanup along the way. I may not be remembering that properly
and there may be mistakes; you'll have to be more specific if you want
more info.

The reason why I think Russ associates COLD shutdown procedures with
brokenness is probably because that would happen if something broke
during initialization. Init scripts usually try both ways of shutting
down on 'stop':

umount /afs
afsd -shutdown

Just to handle both cases. If AFS started successfully, the 'afsd
-shutdown' would do nothing, because either the 'umount' would succeed
and we'd already be shutdown, or the 'umount' failed and 'afsd
-shutdown' would refuse to run because /afs was still mounted.
Conversely, if AFS did not start successfully, the 'umount' would fail
because /afs is not mounted, and we'd try the 'afsd -shutdown'.

For unsuccessful starts, in the past, our handling of errors during init
wasn't great (and still isn't now, but it used to be worse), so things
would be left in an inconsistent state, but wouldn't break in a panic or
whatever until we tried to shutdown. So a lot of the time on Linx in the
past, whenever you saw a COLD shutdown, it was because something broke
during startup in a way we did not handle, and we panic'd on shutdown.
For WARM shutdowns, we started up successfully so there were no such
issues.

If you look at the platforms that set COLD shutdown during umount, you
just see AIX, obsd, and nbsd, and the 'COLD' shutdown was enabled when
support was added for a certain platform version. I would guess that
doing so is "incorrect", and someone just set it to workaround some
problem and never mentioned it, and possibly didn't really know why it
was there. (Of course, I'm not one to talk, since most of what I'm
saying here is also just guessing.)
Post by Mark Vitale
Then there is the question of when it is safe to rmmod/modunload the
libafs kernel module. Does warm or cold shutdown affect the answer to
this question?
It's supposed to be safe after either shutdown is finished. But there
can always be bugs with that, and the ability to stop the client at all
on Solaris is only a few years old (which is why some older mentions of
AFS on Solaris say you can't do it), and there have been a bug or two
already in its shutdown support in that time.

And since COLD shutdowns are more rare (from my understanding above), of
course the code path is less exercised and more prone to bugs.
--
Andrew Deason
***@sinenomine.net
chas williams - CONTRACTOR
2014-07-07 13:04:30 UTC
Permalink
On Thu, 3 Jul 2014 19:23:17 -0500
Post by Andrew Deason
Post by Mark Vitale
If afs_shutdown() is called "warm" (afs_cold_shutdown==0), the
shutdown logic skips the clearing and releasing of some resources. I
see no rhyme or reason to which resources AFS leaves unreleased. Nor
do I understand the (possibly historical) reason for why there is a
distinction between cold and warm shutdown.
I have noticed this oddness a few years when looking at the linux
allocator. It keeps track of all its allocations since it can't rely
seem to rely on the other parts to reliably release their memory
allocations.
Post by Andrew Deason
- You can run some of the client daemons without actually mounting /afs
in "normal" operation. I'm not sure if I've ever done this, but I
have a vague recollection from some code comment or post somewhere
referencing running the nfs xlator PAG manager this way. Maybe there
are other reasons, other ways.
I think just loading the kernel module will get you token management
(PAGs et al) which is potentially useful even without afs mounted.
Post by Andrew Deason
If you look at the platforms that set COLD shutdown during umount, you
just see AIX, obsd, and nbsd, and the 'COLD' shutdown was enabled when
support was added for a certain platform version. I would guess that
doing so is "incorrect", and someone just set it to workaround some
problem and never mentioned it, and possibly didn't really know why it
was there. (Of course, I'm not one to talk, since most of what I'm
saying here is also just guessing.)
Possibly afsd -shutdown should go away. What it does should happen
during module removal. Why did it ever exist? Perhaps there wasn't a
mechanism to hook module removal or perhaps the developers wanted some
sort of partial shutdown in order to remount afs with different
parameters (computers were slower at one point).

Loading...