Struct ChildReaper 

pub(crate) struct ChildReaper {
    grandchildren: Arc<Mutex<HashMap<Box<str>, Vec<ReapableChild>>>>,
    forgotten: Arc<Notify>,
    pending: Arc<Mutex<HashMap<Pid, usize>>>,
    sweep: Notify,
    shutting_down: AtomicBool,
    in_flight: Arc<Sender<Calls>>,
    stop_calls: CancellationToken,
}

Watches the containers and execs of the server until they exit and reaps them.

Fields

grandchildren: Arc<Mutex<HashMap<Box<str>, Vec<ReapableChild>>>>

The registered children, each until its watcher ended.

forgotten: Arc<Notify>

Notified whenever a watcher ended and forgot its child, which the shutdown waits for.

pending: Arc<Mutex<HashMap<Pid, usize>>>

The runtimes and container processes of calls in progress, which are not registered as grandchildren yet. Their exit status belongs to the call, so the orphan sweep leaves them alone. See PendingPid.

sweep: Notify

Wakes the orphan sweep up early, for example after a failed runtime call or on shutdown.

shutting_down: AtomicBool

Set by the shutdown while it holds the lock of the grandchildren, so a child registered later knows that the shutdown missed it.

in_flight: Arc<Sender<Calls>>

The runtime calls in progress, including the registration of their child, which the shutdown waits for.

stop_calls: CancellationToken

Cancelled by the shutdown once the runtime calls in progress did not end within ChildReaper::IN_FLIGHT_WAIT. A call whose runtime still runs fails then and kills it on its own, which only the call can do safely, see ChildReaper::kill_pending.

Implementations

impl ChildReaper

const ORPHAN_GRACE: Duration

How long a zombie nobody claimed stays unreaped before the sweep takes it, so that an owner which is about to wait for it still can.

pub(super) const ORPHAN_MAX_DEFER: Duration

How long the sweep leaves a zombie nobody claimed alone at most, for the runtime calls which were in progress when it was first seen. The process of a call can exit before the call claims it, like an exec which ends before its runtime exited, and its exit status belongs to the call. Each of these calls defers the zombie only until its own runtime deadline and the ChildReaper::CLAIM_MARGIN. This bound counts from when the zombie was first seen, the earlier of both ends the deferral. That is the one of the call for a zombie first seen late in its call, like the process of a slow exec runtime which exited right before it, and this one for a create, whose runtime gets 10 minutes: a runtime which hangs must not keep the zombies an init of a PID namespace waits for that long. An exec whose process exits more than this before its runtime does loses its exit code.

pub(super) const CLAIM_MARGIN: Duration

How long after the deadline of its runtime a call may still claim its process: a runtime which exits right before it may still take the ChildReaper::RUNTIME_GRACE to connect the terminal, the ChildReaper::ORPHAN_GRACE on top covers the scheduling.

pub(crate) fn start_orphan_sweep(self: Arc<Self>)

Reap exited children which nobody waits for, in the background.

A runtime which fails after entering the PID namespace of a container, like an exec of a missing binary, leaves its helpers behind as children of this subreaper. They exit at any later time. The init of the namespace cannot exit until their zombies are reaped, so the container could never be stopped.

Every other child has an owner which reaps it: the runtimes are waited for by tokio and the container processes by their watchers. They are never zombies for long, so a zombie which is neither registered nor pending for ChildReaper::ORPHAN_GRACE is an orphan, once no runtime call which may still claim it is in progress, at most for ChildReaper::ORPHAN_MAX_DEFER. Once the server shuts down, those calls cannot register a process anymore and are not waited for.

pub(super) async fn sweep_orphans( &self, candidates: &mut HashMap<Pid, Candidate>, grace: Duration, max_defer: Duration, )

Reap the zombie children which have been unclaimed for grace, and up to max_defer while a call which was in progress when the zombie was first seen still is and may still claim it, unless the server shuts down. candidates remembers when a zombie was first seen and the calls which were started then.

pub(super) fn defers( &self, candidate: Candidate, now: Instant, max_defer: Duration, ) -> bool

Whether the sweep leaves the zombie candidate alone at now for a call which may still claim it, at most for max_defer since it was first seen, see ChildReaper::ORPHAN_MAX_DEFER.

pub(super) fn is_claimed(&self, pid: Pid) -> bool

Whether a watcher or a call in progress waits for the child. A registered child which its watcher reaped already does not count, another child may have its PID by now.

Both sets get checked under the lock of the grandchildren, which watch_grandchild holds while it moves a child from one to the other.

fn claimed(&self) -> HashSet<Pid>

The children which a watcher or a call in progress waits for, see ChildReaper::is_claimed.

fn watched( map: &HashMap<Box<str>, Vec<ReapableChild>>, ) -> impl Iterator<Item = Pid>

The registered children which their watchers did not reap yet.

impl ChildReaper

pub(crate) const CREATE_RUNTIME_TIMEOUT: Duration

How long the runtime of a create may take before it gets killed, well above the time CRI-O gives the creation of a container, so that a runtime which hangs does not keep its call in progress forever.

pub(super) const EXEC_RUNTIME_GRACE: Duration

How long the runtime of an exec may take beyond its timeout before it gets killed. It only starts the process, so this bounds an exec without a timeout as well.

pub(super) const RUNTIME_GRACE: Duration

How long a runtime gets for what follows its exit or its failure: one which exited successfully to connect the terminal, one which failed to exit on its own and one which got killed to exit. The helpers of a runtime which exited get it to close its output, see RuntimeOutput::finish_unless_stopped.

pub(super) const STOPPED_EXIT_WAIT: Duration

How long a call which the shutdown stopped waits for the exit of its killed runtime, and then for the output of the runtime, instead of the ChildReaper::RUNTIME_GRACE, see ChildReaper::STOP_BUDGET.

pub(super) const STOPPED_OUTPUT_WAIT: Duration

pub(crate) fn exec_runtime_deadline(timeout: Option<Instant>) -> Instant

When the runtime of an exec with the timeout gets killed. A runtime which hangs must not keep its call in progress forever, which defers the reaping of orphans and delays the shutdown.

fn spawn_runtime( runtime: &Path, args: &[String], stdin: bool, ) -> Result<RuntimeChild>

Spawn the runtime, see ChildReaper::create_child.

pub(super) fn runtime_command( runtime: &Path, args: &[String], stdin: bool, ) -> Command

The command of a runtime, see ChildReaper::spawn_runtime.

pub(crate) async fn create_child( &self, call: RuntimeCall<'_>, container_io: &mut ContainerIO, ) -> Result<StartedChild>

Run the runtime with the I/O of the process it starts, and return the process once the runtime exited. The caller registers it through ChildReaper::watch_grandchild.

pub(super) fn check_reported_pid(pid: Pid, subreaper: bool) -> Result<()>

Fail for a PID the runtime reported which is no child of this server, see is_child. Only a child gets registered and signalled later, its PID cannot get reused before its watcher reaped it. A runtime which reports the PID of another process, by a bug or on purpose, must not make the server signal that process. A process which exited already is still a child until reaped, like a restored container may have. The processes get reparented to the server as their subreaper, without that flag, like in the tests, they are never its children, so there is nothing to check.

The call claims the PID before it checks it, so the cleanup of the failed call finds it claimed and does not signal it, see ChildReaper::cleanup_failed_grandchild. The signal through Process::child would refuse a PID which is no child anyway.

async fn connect_io( child: &mut Runtime, container_io: &mut ContainerIO, stdin: bool, token: &CancellationToken, deadline: RuntimeDeadline, stop: &CancellationToken, ) -> Result<(), RuntimeFailure>

Connect the I/O of the process which the runtime starts: wait until the runtime connected the terminal, or read the pipes of the runtime, which the process shares. Fails once the deadline passed or the shutdown cancelled stop, like ChildReaper::wait_runtime.

async fn wait_runtime( child: &mut Runtime, deadline: RuntimeDeadline, stop: &CancellationToken, ) -> Result<ExitStatus, RuntimeFailure>

Wait for the exit of the runtime. A runtime which hangs gets killed once its deadline passed, and a wait which fails gets cleaned up like a runtime which failed. A runtime which still runs once the shutdown stops the call through stop gets killed right away as well, by this call, which holds it until it got reaped, see ChildReaper::kill_pending.

async fn stop_runtime( child: Runtime, output: Option<RuntimeOutput>, grace: Duration, stop: &CancellationToken, ) -> Vec<u8>

Stop a runtime which failed before it exited, or whose call the shutdown stopped: give it grace to exit on its own, so that it still writes the PID file the cleanup needs, kill it otherwise, and return its stderr. It gets killed through child, which tokio did not reap yet, so neither its PID nor its group got reused. Its pending mark ends once it got reaped, or once a killed runtime did not exit within ChildReaper::RUNTIME_GRACE, before waiting for its output. Once the shutdown stopped the call through stop, the runtime gets killed right away and the waits get shorter, so that the call ends within the ChildReaper::STOP_BUDGET.

fn fail_create(&self, pidfile: &Path, error: Error, stderr: &[u8]) -> Error

Clean up after a failed runtime call and build its error, with the stderr of the runtime appended.

pub(super) fn cleanup_failed_grandchild(&self, pidfile: &Path)

Kill and reap the container process of a failed runtime call, if the runtime got as far as writing its PID. It got reparented to this subreaper and nobody else waits for it.

Only this PID is waited for. Reaping any exited child instead would take the exit status of runtimes of concurrent calls, or of grandchildren which are not registered yet.

impl ChildReaper

const IN_FLIGHT_WAIT: Duration

How long the shutdown waits for runtime calls in progress to end on their own, within its deadline. Those which take longer get stopped, see ChildReaper::kill_pending, and get the rest of the deadline to end, at least the ChildReaper::STOP_BUDGET.

pub(crate) const STOP_BUDGET: Duration

How long before the deadline of the shutdown the runtime calls in progress get stopped at the latest. A stopped call kills its runtime right away, waits for its exit and its output only briefly, and kills the process the runtime left, see stop_runtime, which this covers with some time to spare for the scheduling.

pub(crate) fn get(&self, id: &str) -> Result<RegisteredContainer>

Retrieve the container registered under id. Execs share the id of their container and are skipped. The error starts with CONTAINER_NOT_FOUND if there is none.

fn begin_call(&self, deadline: Instant) -> Result<InFlight>

Start a runtime call whose runtime gets killed at the deadline, unless the server shuts down. The shutdown waits for the calls in progress, see ChildReaper::kill_grandchildren.

pub(crate) fn watch_grandchild( &self, child: Child, call: InFlight, ) -> Result<Receiver<ExitChannelData>>

Register and watch a child which create_child started, which ends its call.

pub(crate) fn watch_exec( &self, id: Box<str>, started: StartedChild, timeout: Option<Instant>, io: ContainerIO, ) -> Result<WatchedExec>

Register and watch an exec which create_child started under the id of its container, which gets killed at its timeout, see ChildReaper::watch_grandchild.

fn kill_late_grandchild(pid: Pid) -> bool

Kill a grandchild registered after the shutdown killed the others, unless it is gone already, and return whether it was signalled. The sweep does not defer orphans while the server shuts down, so the process may have exited and got reaped before its call claimed it. Opening a pidfd fails with ESRCH then, and neither the PID nor the group of the process gets signalled. A PID which got reused that fast only gets signalled if it belongs to an unreaped child of the server, see kill_grandchild.

fn forget_grandchild( locked_grandchildren: &Arc<Mutex<HashMap<Box<str>, Vec<ReapableChild>>>>, id: &str, seq: u64, )

Forget the child of the registration seq under id. Not by its PID, which may be in use by a new child already.

pub(crate) async fn terminate_containers(&self, deadline: Instant)

Forward SIGTERM to the containers like conmon, to their PIDs only, and wait for their watchers until the deadline. Containers registered meanwhile get neither, the shutdown kills them and what is left afterwards. A container stays registered after its exit while its watcher finishes the output, its PID may be in use by another child by then, so it gets signalled through its pidfd, see ReapableChild::process.

pub(crate) async fn kill_grandchildren(&self, deadline: Instant) -> Result<()>

Kill every grandchild and wait for their watchers and the runtime calls in progress until the deadline of the shutdown, also for the execs killed at their timeout which were not reaped yet. A process which does not exit on SIGKILL, like one stuck in the kernel, must not keep the server from stopping. It fails if they did not end in time.

fn kill_pending(&self)

Kill the processes which the runtime calls that did not end in time started but did not register yet, like a created container whose call waits for the output of its runtime. The server stops without the calls, a process they leave would run without a watcher.

The calls got stopped before, see ChildReaper::stop_calls. Their runtimes are not signalled here: tokio reaps them without the REAP_LOCK of Process, so the PID of one may belong to another process by now. A call kills its runtime itself while it still holds it unreaped, see stop_runtime. The processes which the calls claimed get reaped only by the orphan sweep and by their watchers, under that lock, see kill_grandchild. Under the lock of the grandchildren no call registers its process meanwhile, whose watcher could reap it then, and under the one of the calls no call ends, after which the sweep could reap it. A call which claims its process later kills it itself, see InFlight::claim.

async fn wait_for_watchers( &self, deadline: Instant, filter: impl Fn(&ReapableChild) -> bool, ) -> Vec<i32>

Wait until the watchers of all grandchildren which match the filter ended, every watcher forgets its child at its end. Return the PIDs of the children left at the deadline.

Trait Implementations

impl Debug for ChildReaper

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for ChildReaper

fn default() -> Self

Returns the “default value” for a type.