Struct LogFile 

pub(super) struct LogFile {
    path: PathBuf,
    mode: Option<u32>,
    pub(super) file: Option<Arc<File>>,
    max_size: Option<usize>,
    bytes_written: usize,
    buf: Vec<u8>,
    torn: bool,
    rotation_failed: bool,
}

A log file on disk which is written in whole chunks.

The drivers assemble their entries into buf, account each one with LogFile::commit_entry and write the buffer with a single call per read chunk, see LogFile::end_chunk, or per LogFile::FLUSH_SIZE of entries for chunks of short lines. The writes, the opens and closing a replaced file run on the blocking pool, so writeback throttling, a slow disk or replacing a huge log only stalls this log and not the thread which drives every container of the pod and the RPCs. The buffer moves to the blocking task and back, unlike an async file, which copies every write into a buffer of its own that keeps the size of the largest one.

Fields

path: PathBuf

mode: Option<u32>

The mode of a file this creates, an existing one keeps its own.

file: Option<Arc<File>>

max_size: Option<usize>

Maximum allowed log size in bytes.

bytes_written: usize

The size of the file, counted from the one it had when it got opened, including the committed entries of buf.

buf: Vec<u8>

The pending output of the current chunk, allocated on first use and released at its end, see LogFile::buf.

torn: bool

Whether a failed write left the file in the middle of an entry, or it had such an end when it got opened, so that the next entry has to start on a line of its own.

rotation_failed: bool

Whether the rotation failed during the current chunk, see LogFile::commit_entry.

Implementations

impl LogFile

const ERR_UNINITIALIZED: &'static str = "logger not initialized"

const BUF_CAPACITY: usize

Initial capacity of the chunk buffer, it grows to what the read chunks need with their line overhead, up to about twice LogFile::FLUSH_SIZE for the entry which crosses it.

const FLUSH_SIZE: usize

The entries get written once they take this much, a chunk of very short lines would grow the buffer to many times its size otherwise.

pub(super) fn new( path: impl AsRef<Path>, mode: Option<u32>, max_size: Option<usize>, ) -> Self

A log file at path, created with mode, which gets replaced once it grows beyond max_size. Nothing gets opened before LogFile::open.

fn buf(&mut self) -> &mut Vec<u8>

The buffer to append the entries of a chunk to. A log file holds one only while it writes a chunk, see LogFile::end_chunk.

pub(super) async fn open(&mut self) -> Result<()>

Open the file and append to its existing content, like conmon does. The content counts for the maximum size, the first entry which exceeds it rotates the file. Content which does not end with a newline, like of a server which crashed in the middle of a write, counts as torn, see LogFile::write_buf. A symlink in place of the log is not followed, the writes would hit its target. A temporary file which a crashed server left behind gets removed, the kubelet would never clean it up.

fn ends_torn(file: &File) -> bool

Whether the file has content which does not end with a newline. One which cannot be read counts as complete.

fn tmp_path(path: &Path) -> PathBuf

The new file of a replacement before it gets renamed over the path: .<name>.conmonrs-tmp next to it. The kubelet removes *.tmp files and treats <name>.* ones as rotated logs, a name which matches neither keeps it from deleting the file while it gets renamed.

async fn replace(&mut self) -> Result<()>

Replace the file with a new empty one, like conmon does: create a temporary file, see LogFile::tmp_path, and rename it over the path. Unlike a truncation in place, a reader which follows the log, like kubectl logs -f, keeps the old file with its offset and reads it to its end, before it notices the new one. A rename does not follow a symlink at the path either.

fn create_at(options: &OpenOptions, tmp: &Path, path: &Path) -> Result<File>

Create a new file at tmp and rename it to path, see LogFile::replace.

fn open_options(&self, truncate: bool) -> OpenOptions

The writes append, so that they land at the end also after the file got truncated in place, like by an admin freeing disk space, instead of at the old offset, which leaves a hole of zeros in front of them. The standard library refuses append together with truncate, the kernel takes both.

async fn commit_entry(&mut self, start: usize) -> Result<()>

Account for the entry appended to buf since start. If it would exceed the maximum log size, the entries of the chunk before it go to the old file, which readers following it still read to its end, and the file gets replaced, the entry becomes the first one of the new file.

An error drops only this entry, or the entries of a failed write, the writers still commit the later entries of the chunk and report the first error once they ended it.

async fn end_chunk(&mut self) -> Result<()>

Write the entries of the chunk which are still pending, see LogFile::commit_entry, and release the buffer. The line overhead of the entries grows it beyond LogFile::BUF_CAPACITY for most full read chunks, which an idle container would keep otherwise. The allocation of the next chunk costs little next to its write.

pub(super) async fn write_entries<T>( &mut self, items: impl IntoIterator<Item = T>, entry: impl FnMut(&mut Vec<u8>, T), ) -> Result<()>

Write a chunk of one entry per item, which entry appends to the buffer, and end it. An entry which fails to commit does not stop the later ones, the first error is returned.

fn file_size(&self) -> Option<usize>

The size of the open file. A stat of an open file does not wait for the disk, unlike its writes.

async fn flush(&mut self) -> Result<()>

Write the pending buffer to the file and clear it. The writers end their chunks with LogFile::end_chunk instead.

async fn write_buf(&mut self) -> (usize, Result<()>)

Write the buffer and return how much of it got written. A write which fails in the middle of an entry, like on a full disk, leaves the file torn, the next write then starts with a newline of its own, so that the next entry does not get appended to the cut one.

fn write_out(file: &File, torn: bool, buf: &[u8]) -> (bool, usize, Result<()>)

Write buf to the file, after a newline if the file is torn. Return whether the newline got written and how much of the buffer.

pub(super) async fn reopen(&mut self) -> Result<()>

Write the pending buffer and start a new file at the path. The kubelet renamed the old one already, a file still at the path gets replaced, like by the rotation.

The file is not synced to disk: an fsync of a large log can take long and the data sits in the page cache either way. Only the exit of the container syncs the log, see ContainerLog::sync.

A failed flush dropped the buffer already, failing the reopen would not bring it back but stop the rotation of the kubelet, which moves the rotated file back then.

Trait Implementations

impl Debug for LogFile

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.