steady state pattern
Every server should be able to run forever without manual cleanup. Logs rotate; caches evict; tables get archived. The machine should never need an operator to come around and free up disk space.
Every server should be able to run forever without manual cleanup. Logs rotate; caches evict; tables get archived. The machine should never need an operator to come around and free up disk space.
symptoms
- recurring incidents from disk fill / unbounded growth
- manual cleanup tasks accumulating in runbooks
- memory leaks dressed up as "needs a weekly restart"
causes
- no log rotation
- unbounded in-memory caches
- audit/event tables that grow forever
- no eviction policies anywhere
fixes
- log rotation by size + age
- TTL on caches, bounded sizes
- archive / partition + drop old data
- soft-delete + scheduled hard-delete
you might say
- it ran out of disk again
- the table never gets cleaned up
- logs filled the volume