Kubernetes on Thalamus

The 200 that nobody received

Sun, 20 Sep 2026 00:00:00 +0000

Clients of api.thalamus.am were occasionally timing out. The gateway access log showed successful requests.

The request had arrived. The upstream had answered. The log recorded a 200. There was no obvious pattern by endpoint or time of day, and the failures were rare enough to disappear into otherwise healthy dashboards.

I spent too long looking for an application error. The useful question turned out to be: which machines did the connection depend on after the application had done its work?

df says full, du disagrees

Sat, 08 Aug 2026 00:00:00 +0000

The third time the compactor volume filled up, I checked what was using the space before making it bigger.

The alert was KubePersistentVolumeFillingUp, on the compactor volume of our log store. The first time, we resized the volume. That bought about a month of quiet. It also made the diagnosis feel settled: more logs, not enough disk.

Then the alert came back. And came back sooner.

This time I opened a shell in the pod and compared two numbers:

Set up a Kubernetes cluster with MetalLB

Sat, 29 Apr 2023 00:00:00 +0000

A Kubernetes cluster on your own VMs needs a way to give services an address that other machines can reach. This lab uses kubeadm to build the cluster, Calico for pod networking, and MetalLB for LoadBalancer services.

The setup had three Ubuntu VMs: one control plane node and two workers. These notes keep the 2023 versions, including Kubernetes 1.26.4 and Calico 3.25.1. Some steps were missing from the first version, especially the MetalLB network configuration. I explain those gaps below, but the complete setup still needs a fresh test.

Deploy a microservices demo with Rancher

Sat, 09 May 2020 00:00:00 +0000

This lab follows a microservices demo from three Ubuntu machines to an application deployed on Kubernetes. Rancher provides the interface for creating the cluster and registering its nodes.

The notes and screenshots come from my 2020 demo. The package repositories and Rancher screens have changed since then. The sequence is still useful to read, but it needs a fresh lab run before it can serve as a current tutorial.

The three nodes have different jobs

The original cluster assigned these roles:

CI/CD for a Java project: manual installation

Sun, 12 Apr 2020 00:00:00 +0000

In this Java CI/CD lab, Jenkins runs the build jobs, Docker builds container images, and Nexus stores them. Nginx gives Jenkins and Nexus their own HTTPS addresses. Before connecting the pipeline, each service needs to work on its own.

I split the original project into two parts: install the components by hand first, then automate the setup with Terraform and Ansible. This page covers the manual installation recorded in 2020. It stops before the Java build job and Kubernetes deployment.