Truncate/prevent super long log files in "Vector" container
The timberio/vector
container seems to be a log aggregation system that is used by Cairn, Some times, it gets stuck in a "started watching, stopped watching" loop, with messages like these:
2022-08-17T14:07:09.192618Z INFO vector::internal_events::docker_logs: Started watching for container logs. container_id=9d69b394bca3d28762cc6d9d2be3587fd3924b02e6d9e06582fe29287cafb57e
2022-08-17T14:07:09.193371Z INFO vector::internal_events::docker_logs: Stopped watching for container logs. container_id=9d69b394bca3d28762cc6d9d2be3587fd3924b02e6d9e06582fe29287cafb57e
2022-08-17T14:07:09.193563Z INFO vector::internal_events::docker_logs: Started watching for container logs. container_id=9d69b394bca3d28762cc6d9d2be3587fd3924b02e6d9e06582fe29287cafb57e
2022-08-17T14:07:09.195059Z INFO vector::internal_events::docker_logs: Stopped watching for container logs. container_id=9d69b394bca3d28762cc6d9d2be3587fd3924b02e6d9e06582fe29287cafb57e
These messages quickly fill up the disk, the log files we've seen were 30-50 GB, in filenames like: /var/lib/docker/containers/faf827405b2b711f18a26592bb7de126a71c2bf630ca2dc6b2445279e6e649ad/faf827405b2b711f18a26592bb7de126a71c2bf630ca2dc6b2445279e6e649ad-json.log
.
We should figure out:
- If we can prevent the loop
- If we can automatically truncate this log (even without this looping quickly, the log could fill up the disk eventually if configured incorrectly)