The third time the compactor volume filled up, I checked what was using the space before making it bigger.
The alert was KubePersistentVolumeFillingUp, on the compactor volume of our log store. The first time, we resized the volume. That bought about a month of quiet. It also made the diagnosis feel settled: more logs, not enough disk.
Then the alert came back. And came back sooner.
This time I opened a shell in the pod and compared two numbers:
$ df -h /data
Filesystem Size Used Avail Use% Mounted on
/dev/nvme2n1 50G 46G 3.2G 94% /data
$ du -sh /data
6.1G /data
The filesystem reported roughly 46 GB in use. The files I could reach under /data accounted for about 6 GB. Before another resize, I needed to explain the gap.
The two commands count different things
df reports usage for the filesystem containing the path. du walks the directory tree and totals the files it can reach. They answer related questions, but they don’t measure exactly the same thing.
A large difference is a clue, not a diagnosis. First check that you’re looking at the same filesystem, that du isn’t reporting permission errors, and that another mount isn’t hiding files. Filesystem metadata and snapshots can also affect the comparison. On GNU systems, du -xsh /data keeps the walk on one filesystem; it still only counts what is reachable under that directory.
In our case, the missing space belonged to deleted files that the compactor still had open.
Deleting a filename doesn’t necessarily free its data. If a process still holds the file open, Linux keeps the file available through that open reference. du can’t find it through its old name, but its blocks still count in filesystem usage. The unlink(2) manual describes this distinction between removing a name and releasing the file.
Find the process holding the files
Inside this container, the compactor was PID 1. I checked its open descriptors:
$ ls -l /proc/1/fd | grep ' (deleted)$' | wc -l
14312
That was more than fourteen thousand descriptor entries pointing to deleted files. The count alone didn’t tell me how many bytes they occupied, or how many distinct files there were. It did give me a process to investigate.
/proc/1/fd is specific to this container’s process layout. In another container, PID 1 might be a wrapper or an init process. Use the actual application’s PID, and check other processes if the first one doesn’t explain the usage. The /proc/<pid>/fd documentation explains the entries and their access restrictions.
Where lsof is installed, this is another useful starting point:
lsof +L1
+L1 selects open files with no remaining directory links. Inspect the process, device and file information to connect the results to the affected mount. You need permission to inspect the relevant processes, and a container may not expose every process that uses the volume. An empty result doesn’t rule out the problem if your view is incomplete.
The compactor was removing temporary files while retaining references to them. Each compaction cycle left more space allocated.
A restart recovered space; the update addressed the leak
Restarting the compactor dropped usage from 94% to below 15%. The process exited, its open references went away, and the filesystem could release the deleted files. Updating the compactor addressed the underlying leak in this incident.
A restart needs the same care here as any other change to a stateful service. Capture the evidence first. Check how the component handles interruption and how it resumes work. If the leak remains, the volume can fill again after the restart.
The drop supported what the open-file inspection had shown. It wasn’t a universal test for every df/du mismatch. If usage stays high, another process may still hold the files, your inspection may have missed part of the filesystem, or a different cause may be responsible.
Compare usage before discussing capacity
The misleading part was that resizing worked for a while. The graph stopped approaching the limit, the alert cleared, and there was less reason to question the explanation.
I now want the runbook to answer these questions before treating the next alert as a sizing problem:
- Are
dfanddulooking at the same filesystem and a comparable scope? Did the directory walk finish without access errors? - Are processes holding deleted files on that filesystem? Which process owns them?
- Does the application’s version history describe the same symptom? Compare the details before choosing an upgrade.
- If a controlled restart is needed to recover space, what evidence should be saved first, and what will prevent the buildup from returning?
There are real capacity problems too. Retention may be too long, ingestion may have grown, or a workload may need more working space. In this incident, though, I had been sizing the volume around files the compactor should already have released.
The next time the alert fires, compare the two numbers before deciding what the disk needs.