https://blog.thalamus.am/tag/troubleshooting/Recent content in Troubleshooting on ThalamusHugo -- gohugo.ioen-USThalamus TMSun, 20 Sep 2026 00:00:00 +0000https://blog.thalamus.am/posts/the-200-that-nobody-received/Sun, 20 Sep 2026 00:00:00 +0000https://blog.thalamus.am/posts/the-200-that-nobody-received/<img src="https://blog.thalamus.am/posts/the-200-that-nobody-received/editorial-cover.png" alt="Featured image of post The 200 that nobody received" /><p>Clients of <code>api.thalamus.am</code> were occasionally timing out. The gateway access log showed successful requests.</p> <p>The request had arrived. The upstream had answered. The log recorded a 200. There was no obvious pattern by endpoint or time of day, and the failures were rare enough to disappear into otherwise healthy dashboards.</p> <p>I spent too long looking for an application error. The useful question turned out to be: which machines did the connection depend on after the application had done its work?</p>https://blog.thalamus.am/posts/df-says-full-du-disagrees/Sat, 08 Aug 2026 00:00:00 +0000https://blog.thalamus.am/posts/df-says-full-du-disagrees/<img src="https://blog.thalamus.am/posts/df-says-full-du-disagrees/editorial-cover.png" alt="Featured image of post df says full, du disagrees" /><p>The third time the compactor volume filled up, I checked what was using the space before making it bigger.</p> <p>The alert was <code>KubePersistentVolumeFillingUp</code>, on the compactor volume of our log store. The first time, we resized the volume. That bought about a month of quiet. It also made the diagnosis feel settled: more logs, not enough disk.</p> <p>Then the alert came back. And came back sooner.</p> <p>This time I opened a shell in the pod and compared two numbers:</p>