Incidents and lessons learned

A record of what has failed, why, and the rule that came out of each case. Most recent first.

2026-10-21 · Removing the Actions Runner Controller (summerwind)

Not an incident but a cleanup. The summerwind ARC (v0.27.6, an abandoned project, with 10 CRITICAL and 239 HIGH CVEs according to Trivy) only had one RunnerDeployment, for example-org/Bingx. That project is no longer used (BingX is not available in Spain or the EU), the runner had been offline since April, and the Helm values held a personal GitHub token in plain text.

Steps, in this order so nothing is left stuck: delete the RunnerDeployment while the controller is still alive (it deregistered itself from GitHub) → helm uninstall arc (controller, webhooks, RBAC, cert-manager certificate) → delete the 5 *.actions.summerwind.dev CRDs → delete the arc-systems namespace. Checked afterwards: API ok, no pods with problems and Argo CD with its 4 applications Synced.

The personal GitHub token it used had expired in May 2026, which is why the runner had been offline since then. It is no longer valid, so there is no risk. Same pattern as the Argo CD credential: credentials that expire without anyone noticing (→ alerts, R2).

If self-hosted runners are ever needed (deployments inside the LAN, unlimited minutes, ARM builds), install the official ARC (runner scale sets, GitHub App, ephemeral pods) and never use it on public repos.

2026-10-21 · Homepage intermittently loses the node list

Cause. The Nodes card comes from kube-state-metrics, which was running on master-1 (WiFi). Over the last hour, Prometheus (on a wired node) could only read what runs on the master-1 83-92 % of the time; the wired nodes, 100 %. When several scrapes fail, the series go stale and the card goes empty.

Fix. kube-state-metrics was moved to a wired worker (an affinity patch: never on master-1, preferably on a worker) and the duplicate release was removed. No helm upgrade, because it would have restarted Grafana, which had no persistence.

Lesson. While master-1 is on WiFi, don't put anything monitoring depends on there (Prometheus, kube-state-metrics, central exporters). Since 2026-10-27 master-1 is wired: the restriction is no longer needed because of the network, although it is still a heavily loaded master.

2026-10-18 → 21 · Jira and Confluence down from the Internet for 3 days (Cloudflare tunnel)

Symptom. jira.example.com and confluence.example.com returned 530 / error 1033 (Cloudflare can't find any active connector for the tunnel). Found by chance on 2026-10-21, while checking something else.

Cause. cloudflared (on master-3) was in a loop: it registered its 4 connections to Cloudflare and lost all of them after 10-20 s (timeout: no recent network activity, over QUIC/UDP). 1,000-1,700 reconnections a day since 2026-10-18, none before. Two things happened that day: apt upgraded cloudflared from 2026.3.0 to 2026.10.1 and the wired network outages started. The tunnel used QUIC (UDP), the default protocol.

Fix (2026-10-21). Force HTTP/2 (TCP), which copes much better with an unstable network:

sudo sed -i 's|tunnel run --token|tunnel --protocol http2 run --token|' /etc/systemd/system/cloudflared.service
sudo systemctl daemon-reload && sudo systemctl restart cloudflared

Since then: 4 protocol=http2 connections and no drops. Jira and Confluence respond again.

Lessons

• Third silent failure in a week, after Argo CD not syncing and Velero without backups: alerts are needed (R2), plus an external checker for *.example.com (R8).
• cloudflared runs on a single node and is upgraded with apt: watch its logs after every upgrade and consider deploying it in the cluster with 2 replicas (pending item no. 18).

2026-10-20 · Argo CD had not synced for months

Symptom. When deploying Homepage, Argo CD did nothing. All 4 applications were Unknown with ComparisonError: authentication required: Invalid username or token.

Cause. The credential Argo CD uses to read example-org/GitOps (Secret repo-*, created on 2026-01-18) had expired. The applications kept running with whatever was last deployed, so nobody noticed: any change to GitOps would have silently gone unapplied.

Fix. The credential was regenerated and updated in Argo CD (Settings → Repositories). Before reconnecting, kubectl diff confirmed that the repo and what was deployed matched, to avoid surprise changes when syncing again.

Lessons

• A credential that expires with no alert is a scheduled outage. Options: connect Argo CD through a read-only GitHub App (it doesn't expire) and add an alert for applications in Unknown or OutOfSync (R2).
• Same pattern as Velero (backups failing without anyone noticing): alerts that reach someone are needed.

2026-10-18/19 · Control plane in a restart loop (etcd losing its leader)

Symptom. k3s kept restarting on the masters: etcd lost its leader, the controller-manager logged leaderelection lost and k3s exited.

Causes found

1. filebeat (kube-system DaemonSet) with a 200 Mi limit, in an OOM loop. It was re-reading /var/log at ~490 MB/s on master-1, with 70 % I/O pressure. On top of that, Elasticsearch had been down for ~104 days, so filebeat had nowhere to send anything. → Paused with nodeSelector paused-by=claude-etcd-incident. To undo it, remove that nodeSelector.
2. master-3 with --flannel-iface=wlo1 while its WiFi was disconnected. k3s exited every ~30 s (196 restarts) and forced etcd elections. → Changed to enp2s0. Later, master-2 was also moved from wlp2s0 to enp1s0.
3. master-3's disk at 97 % I/O pressure. A low-end SSD, with Longhorn rebuilds and trivy-operator scanning at the same time. → trivy-operator scaled to 0. Re-enabled on 2026-10-26 with a light footprint (its own server with the database on worker-2's local disk, one scan at a time, preferably on worker-2/worker-1): seguridad/trivy-operator.yaml.
4. Network outages between the router and every wired machine: ~11 s every ~33 s, with 32 % packet loss from master-2 to the router. The wired machines see each other with no loss. etcd on master-1 (WiFi) loses the leader (master-2) every ~30 s (tls: bad record MAC/EOF). The cable between the router and the switch is the suspect. → Solved on 2026-10-27: the router ↔ switch cable was replaced with an S/FTP one. Afterwards, from master-2, 0 % loss to the router and to every node, with 1000 Mb/s links, and from master-1 (WiFi) 0 % over 90 s with a maximum of 91 ms. During the swap, master-3 and worker-2 lost the network (they had to be reconnected and worker-2 rebooted) and Longhorn rebuilt 4 volumes without issues. With the network stable, keepalived moved to the cable on master-2 and master-3, and master-1 moved entirely to the cable with scripts/master1_a_cable.sh (a first attempt just plugging the cable in left the master-1 with two IPs on the same network, .112 over WiFi and .185 over the cable, and cut the SSH session: it has to be done with the script).

Collateral damage

• The n8n and Portainer volumes went faulted because their only replica was on the worker-1, which had changed IP (.168 to .128). They recovered when the worker-1 rejoined.
• Filesystem damage on the Elasticsearch volume (input/output error under indices/*/_state).
• The MinIO image pinned on quay.io started returning 401. It was only cached on master-1 and worker-2, so MinIO was pinned to worker-2.
• keepalived ended up in FAULT on master-2 and master-3 (VRRP over a dead WiFi link). Found on 2026-10-20 and solved on 2026-10-27, after the cable swap, by moving VRRP to the cable.
• The worker-4 node was retired for good.

Lessons

• No I/O-heavy workloads without limits on a master. etcd is very sensitive to disk latency.
• --flannel-iface must point to an interface that is always up. If a node's physical network changes, check k3s and keepalived.
• Node IPs must be reserved on the router.

2026-07-04 · n8n loses its data when its nodeSelector is removed

What happened. The n8n Deployment had a clean-n8n init container (rm -rf /data/*) that mounted the same PVC as the main container, at a different path. Nobody saw it because the pod had gone 132 days without a restart. When the nodeSelector was removed, the first restart wiped the PVC.

Why it couldn't be recovered. Velero only stored manifests, not the volumes' contents. There was a Longhorn snapshot from 9 h earlier, but we decided to start from scratch because the workflows weren't critical.

Lessons

• Before forcing the first restart of a pod that has been up for a long time, review its initContainers and sidecars for destructive commands.
• A Velero backup without the node-agent doesn't protect data (see backups.md).

2026-07-04 · The VIP breaks the overlay network

What happened. keepalived added the VIP as a secondary IP, with preferred_lft forever, on the same interface as the node's real IP, which was dynamic. flannel prefers permanent addresses, so it picked the VIP as the node's public-ip and as the local address of the VXLAN tunnel. NodePort, the internal DNS and pod-to-pod traffic broke on every node that had held the VIP. Passing --node-ip and --flannel-iface wasn't enough, because flannel repeats the detection periodically.

Fix. Move the VIP to a dedicated dummy0 interface (nodes/masters/vip-dummy0.service). During the change, giving the VIP a /24 created a route for the whole subnet through dummy0 and broke etcd's raft for about 2 minutes. It has to be /32.

Lesson. On k3s with flannel, a floating IP always goes on a dummy interface, as a /32. After any network change, check that ip -d link show flannel.1 shows the real IP.

2026-07-03 · Migration to HA: data loss on local-path

What happened. To turn master-2 and master-3 from agents into servers, k3s-agent-uninstall.sh was run. It deletes the whole of /var/lib/rancher/k3s, including the local-path storage. The data of Kafka, Jenkins, Confluence and Portainer was lost.

Other problems in the migration from SQLite to etcd

• --cluster-init with a half-done db/etcd from a previous attempt fails with no bootstrap data found in datastore. Fix: move db/etcd aside and try again.
• A large state.db gives etcdserver: request is too large. Fix: --etcd-arg=max-request-bytes=52428800 --etcd-arg=quota-backend-bytes=2147483648.
• Installing without INSTALL_K3S_VERSION brings in the latest stable version, which can be 2 minor versions ahead of the rest.
• With 2 etcd members, if the third one doesn't finish joining, quorum is lost and the API with it. To recover it, --cluster-reset.

Lesson. See operacion.md.

2026-10-26 · Grafana: "database is locked" and its volume in a rebuild loop

What happened. Grafana moved to a Longhorn PVC (grafana_persistente.sh) and was running on master-1, which is on WiFi. First, a network drop with the replica on master-3 left writes hanging and SQLite threw database is locked. Then grafana_local.sh (2 replicas + dataLocality: best-effort) asked for a local replica on the master-1: Longhorn couldn't rebuild it over WiFi, the volume went into an AutoSalvaged / Detached / Attached loop and the kubelet even ran mkfs.ext4 on the volume (it failed with Input/output error).

Damage. None: Grafana was scaled to 0 replicas and, mounted read-only on worker-2, the filesystem was intact and grafana.db passed PRAGMA integrity_check (31 dashboards and 2 users). There was also a copy in /root/.config/grafana-copia/.

Fix. Grafana pinned to worker-2 (monitoring/grafana-persistente.yaml, nodeSelector), which is wired and holds the healthy replica: the local replica already exists and nothing has to be copied over WiFi.

Lesson. No Longhorn volume with data locality, and no SQLite workloads on Longhorn, on a WiFi node (the master-1 was one; since 2026-10-27 it is wired). See operations.

2026-10-28 · Falco doesn't start with kernel 7.0.0-42

What happened. After rebooting worker-1 and worker-2 with the new kernel (7.0.0-42-generic), Falco 0.45.0 went into CrashLoopBackOff on those two nodes: libbpf: failed to load BPF skeleton 'bpf_probe': -13 / libpman: failed to load BPF object (errno: 13 | message: Permission denied). On the nodes still on 7.0.0-38 it kept working: the new kernel's BPF verifier rejects the modern_ebpf probe.

Fix. Falco 0.45.1 (libs 0.26.1: fix signal_deliver verifier failure on kernels exposing the siginfo argument as a scalar) with scripts/falco_actualizar.sh. An exception was made to the two-week rule (released on 2026-10-19) because it was the fix for the security tool itself.

Lesson. After a kernel upgrade, check Falco on the rebooted nodes before rebooting the rest: a new kernel can leave the whole cluster without the eBPF probe.

2026-10-28 · The worker-2 drain hangs on orphaned Longhorn tickets

What happened. While upgrading the workers, kubectl drain on worker-2 never finished: the instance-manager PDB wouldn't let it be evicted. The MinIO, Confluence and es-snapshots volumes had snapshot-controller-* tickets in spec.attachmentTickets for snapshots that no longer existed, leftovers from the 27/10 outage. Longhorn considered them attached and protected the instance-manager.

Fix. Remove only the orphaned tickets, one by one (operations). Then the drain finished and worker-2 rebooted with the new kernel. No data was lost.

Lesson. Before a drain, look for snapshot-controller-* tickets whose snapshot doesn't exist.

2026-11-03 · A master is left cordoned and the playbook can't touch it again

What happened. master-3 turned up cordoned and empty (not rebooted since 18/10): a reboot from the portal had run the drain and never finished. The playbook returns the node to the cluster when it fails (always: uncordon), so that task was cut off before reaching that step; there is no record of why. Then Update system on the workers stopped at its first check ("no node out of the cluster"), and so did re-running Reboot on master-3, because the cordoned node was precisely the one to reboot. Two chained causes:

1. The drain was blocked by the last replica of a forgotten volume. A temporary Confluence volume (switched off, no pods, waiting to be deleted) had a single replica, on master-3. With node-drain-policy: block-if-contains-last-replica, Longhorn keeps the instance-manager PDB of a node holding the last copy of a volume, even a detached one. Its log said so: removing master-3 PDB is blocked: replica … has no pdb on another healthy replica. It was already known: since 2026-10-20, task no. 21 in the backlog asked to delete that unused volume. It sat under "Cleanup", the lowest priority, and was never done.
2. The playbook blocked itself. The pre-check requires no node to be cordoned. If a task is cut off after the drain without reaching its uncordon, the node is left in a state the portal can't get it out of.

Fix. node-drain-policy changed to allow-if-replica-is-stopped: a node can be drained if its last replica of a volume is stopped (a reboot doesn't wipe the disk), while volumes in use stay protected. Longhorn removed the PDB right away. Then a manual kubectl uncordon so the check would pass, and the reboot finished fine. Once back, five volumes were degraded for a few minutes while their replicas on master-3 caught up; none was left without healthy copies.

Lessons.

• Pending cleanup isn't harmless: a forgotten single-replica volume blocks maintenance of its node even if nobody uses it. Unused volumes get deleted or given replicas, and anything that blocks maintenance moves up in priority.
• A pre-check must accept the state the playbook itself leaves when it is cut off halfway (the target node cordoned).
• Update system didn't wait for Longhorn to be healthy again between nodes: until it does, it is run one node at a time, waiting in between.

2026-11-03 · No container can signal itself: AppArmor and kernel 7.0.0-42

What happened. After the x86 nodes finished rebooting onto kernel 7.0.0-42, Elasticsearch went into CrashLoopBackOff and the central Postgres into a restart loop. Kibana was left without its Elasticsearch. On the master-1, also, the node was still cordoned after the drain before its reboot, and Postgres, Pi-hole and Vector, pinned to it with nodeSelector, sat in Pending until kubectl uncordon. n8n and Confluence, the clients of that Postgres, were already switched off, so no other applications were affected. The symptoms looked nothing alike:

• Elasticsearch: Unable to attach entitlement agent, caused by IOException: Permission denied in sendQuitTo. The JVM could not send SIGQUIT to itself.
• Postgres: it hung after redo done, not accepting connections, until the liveness probe killed it. Each kill left another recovery pending, with the some data is corrupted warning.

Cause. Ubuntu turns on kernel.apparmor_restrict_unprivileged_unconfined=1. With that, runc can't switch to the container's profile and stacks it instead: processes end up labelled cri-containerd.apparmor.d//&unconfined. containerd's default profile only allows signals and ptrace towards the unstacked name, so the kernel denies them. According to containerd#14485, denials were rare on 7.0.0-38 and on 7.0.0-42 they hit every non-privileged container. The fix (#14487) is still unmerged: no k3s version includes it, so upgrading k3s wouldn't have helped.

How we got there. At first it looked like a resource problem: Postgres has 150m of CPU and Elasticsearch had just moved node. That was ruled out because there was no flood of clients (only the probes' errors), and no damage to the Elasticsearch volume, which was healthy. The clue was sendQuitTo. With short-lived test pods:

For Postgres the cause was inferred from the symptom, the containerd report, and the fact that it started as soon as the profile was loaded: without access to the kernel log on the nodes, the denial itself wasn't seen.

Fix. scripts/apparmor_senales.sh installs on the 5 x86 nodes a copy of the containerd v2.2.7 profile with the signal and ptrace rules also covering the stacked label. containerd doesn't load its own if one with that name already exists, and /etc/apparmor.d/ is loaded at boot, so it survives reboots. It is loaded live: without restarting anything, containers already running could send signals too. Postgres and Elasticsearch started on their next attempt. Postgres's recovery had nothing to redo and its log shows no errors afterwards; no integrity check has been run inside the database.

Setting the sysctl to 0 was rejected because it lowers the hardening of the whole host, and Unconfined per pod because it removes confinement and leaves every other container as it was.

Lessons.

• After a kernel update, also check the startup of something that signals itself, not just that pods are Running: most start fine even when they can't send signals.
• A Permission denied can come from the LSM even when files and capabilities are fine. A test pod with and without confinement settles it in minutes, without touching the real workload.
• A local copy of a profile has to be removed when the fix arrives: it stays in the backlog.