Re: How best to ensure s6-managed services are shut down cleanly?

From: Laurent Bercot <ska-supervision_at_skarnet.org>
Date: Fri, 01 Feb 2019 19:46:00 +0000

>I _think_ that with my naive current setup, what actually happens is:
>
>- systemd sends a SIGTERM to s6-svscan;
>- s6-svscan sends a SIGTERM or SIGHUP to all s6-supervise processes,
>depending on what they are supervising, and then runs the finish program;
>- the s6-supervise for postgresql sends a SIGTERM and a SIGCONT to the main
>database process. It then waits for the postgresql process to terminate,
>runs its finish program if there is one, and exits;
>- because postgresql responds to SIGTERM by disallowing new connections but
>permitting existing ones to keep running, it continues doing that until
>being killed.

That sounds accurate.


>Reviewing the current docs for s6, I see that I can improve this situation
>a bit by using a "down-signal" file to tell s6-supervise to send a SIGINT
>instead of a SIGTERM.

Yes, being able to customize the signal that kills the service was a
highly requested feature. I postponed it for a long time because I
couldn't find a model that didn't jeopardize the supervisor's stability.
(The original implementation of this feature is runit's control/
scripts,
but a bad control script can hang runsv.) down-signal is not as
flexible,
but it's safe.


> That's cool! But what I would really _like_ to do is
>wait for up to a minute to allow the database to shut down cleanly before
>the system shutdown proceeds

The question is, how does systemd decide to proceed with the rest of
the shutdown? If it's just waiting for s6-svscan to die, then it's
easy: don't allow s6-svscan to die before all your services are
properly shut down. That can be done by a single s6-svwait invocation
in .s6-svscan/finish:

#!/bin/sh
exec s6-svwait -D -T60000 /scandir/*

and s6-svscan's death won't be reported to systemd before all your
services are really down, or one minute, whichever happens sooner.

--
Laurent
Received on Fri Feb 01 2019 - 19:46:00 UTC

This archive was generated by hypermail 2.3.0 : Sun May 09 2021 - 19:44:19 UTC