SEV-3 · 3 min read

Fluent Bit is silently skipping your rotated logs

This is a dummy post. Replace with a real write-up — the structure below is the template every SEV-3 (quick tip) entry follows.

Symptom

Log volume in OpenSearch dips during traffic spikes — exactly when you need logs most. No Fluent Bit errors. Nothing in the metrics that screams.

Cause

The tail input tracks files by inode. When a chatty pod rotates its log file faster than Fluent Bit's refresh interval, the new inode appears and the old one vanishes between scans — those lines are simply never read.

Fix

[INPUT]
    Name              tail
    Path              /var/log/containers/*.log
    Refresh_Interval  5        # default 60 — way too slow for chatty pods
    Rotate_Wait       30       # keep reading the rotated file this long
    Buffer_Max_Size   256k
    Skip_Long_Lines   On

Then watch fluentbit_input_records_total against the emitting pod's own counter for a day. If the lines diverge under load, lower Refresh_Interval further or raise the pod's log rotation size.

Rule of thumb: rotation interval must be at least 6× your refresh interval, or you're gambling with your audit trail.