Highly available k3s homelab
A Kubernetes (k3s) cluster of six mixed machines at home: recycled x86 boxes, mini-PCs and a Raspberry Pi. I use it as a lab for platform engineering, observability, security and automation, and it hosts services I use every day. Everything here runs in home production: the incidents, the figures and the decisions are real.
Interactive dashboard:
index.html, a replica of the cluster's home dashboard (Homepage) with a snapshot of real data. It is the front page of this site. Click any card to see how it is built.
Anonymized showcase. This repository is generated automatically from the private operations repo. The IPs (RFC 5737 documentation ranges), the domain (
example.com), the node names and the users are fictitious; the content, the figures and the history are not. No file is edited by hand, and every publication is a single commit with no history. The code (scripts, playbooks and manifests) is shown as it is, with its comments in Spanish.
In numbers
| Nodes | 6: 3 masters with embedded etcd + 3 workers · amd64 and arm64 |
| Kubernetes | k3s v1.34 · ~36 namespaces |
| API endpoint | VIP with keepalived (VRRP) + haproxy, failover in ~3 s |
| Storage | Longhorn (3 replicas) and local-path |
| GitOps | Argo CD (Homepage, Ollama, Flink, Vector…) |
| Observability | Prometheus/Grafana · Elasticsearch/Kibana (ECK) · Kafka (Strimzi) → Flink → Vector |
| Security | Kyverno · Falco (eBPF) · Trivy Operator · Policy Reporter · key-only SSH |
| Automation | Ansible + Semaphore (portal with buttons) · automatic AI review of PRs |
Architecture
flowchart TB
ts((Tailscale<br/>2 subnet routers)) --> vip
ts --> lb
lan((LAN)) --> vip[VIP 192.0.2.250:6443<br/>keepalived + haproxy]
lan --> lb[MetalLB L2]
subgraph cp[HA control plane · 3-member etcd]
direction LR
m1[master-1<br/>VRRP 150]
m2[master-2<br/>VRRP 140]
m3[master-3<br/>VRRP 130]
end
vip --> cp
subgraph wk[Workers]
direction LR
w1[worker-1]
w2[worker-2<br/>local AI]
w3[worker-3<br/>arm64]
end
cp --- wk
lb --> traefik[Traefik Ingress]
lb --> dns[Pi-hole DNS]
traefik --> apps[Jira · Confluence · n8n]
flowchart LR fb[filebeat / elastic-agent] --> k[(Kafka KRaft)] k --> fl[Flink<br/>routing job] --> k k --> v[Vector] --> es[(Elasticsearch)] --> kb[Kibana] pr[Prometheus] --> gr[Grafana] & am[Alertmanager] fa[Falco eBPF] --> fs[Falcosidekick]
Layers and decisions in detail in docs/arquitectura.md.
Technical decisions
- k3s with embedded etcd and 3 masters: a single binary, low resource usage and mixed architectures. A master can be drained and rebooted without losing the API. Migrated from SQLite with the cluster running.
- VIP on a
dummy0interface as a/32: as a secondary IP on the real interface, flannel picked it as the node's public IP and broke the overlay network; as a/24it hijacked the route for the whole subnet. Both things happened (postmortem). - Nothing exposed to the Internet: services are only reachable from the LAN and through Tailscale (two subnet routers, in case one goes down). They used to be published with Cloudflare Tunnel; all routes were removed, and whatever goes back out will have Cloudflare Access (login first) in front.
- Stateful data according to its risk: Longhorn with 3 replicas for what must survive losing a node; local-path only where performance wins, documented, with the rule that changing its
nodeSelectorloses the data (storage). - Pinned versions with tag and
@sha256, released more than two weeks earlier; exceptions only for security advisories, and written down. - No secrets in git: the cluster state exporter replaces credentials with
<REDACTED>and fails if it finds a key or a token.
Postmortems
Format symptom → cause → fix → lesson, with the rule that came out of each one (docs/incidentes.md):
| Incident | Lesson |
|---|---|
Control plane in a restart loop: etcd losing its leader because of filebeat I/O, a dead WiFi link in --flannel-iface and a faulty cable |
No I/O-heavy workloads without limits on the masters; --flannel-iface always on a stable interface |
| The keepalived VIP breaks flannel's overlay network | Floating IP on dummy0, as a /32 |
| Cloudflare tunnel silently down for 3 days after an update (QUIC over an unstable network) | Force HTTP/2; alerts and an external check are needed |
| Argo CD not syncing for months because of an expired credential | A credential that expires with no alert is a scheduled outage |
| Grafana volume in a rebuild loop because of WiFi | No Longhorn volume with data locality on a WiFi node |
| Falco without its eBPF probe after a new kernel | Check Falco on the rebooted nodes before moving on to the rest |
kubectl drain stuck on orphaned Longhorn tickets |
Look for orphaned snapshot-controller-* tickets before a drain |
| Data loss on local-path when migrating to HA and because of a destructive init container | Review initContainers before restarting a pod that has been up for a long time |
Automation
- Maintenance portal (playbooks): Ansible behind Semaphore, with buttons to check, update and reboot nodes one at a time (cordon → drain → reboot → uncordon), checking etcd, Longhorn and the VIP before moving on to the next one. Automation user with a single key restricted by source.
- Automatic PR review: every change goes through a secret scanner and YAML and Kubernetes manifest validation, and then an AI review against the repo's rules. It never approves or merges: a person does that.
- Operations scripts (scripts/): a security traffic light for the whole cluster in ~20 s (
seguridad.py), a health check, a secret-free export of the cluster state, and key-only SSH on the 6 nodes enforced through Kubernetes without logging in over SSH.
Layout
.
├── docs/ Architecture, storage, operations and incidents
├── ansible/ Maintenance portal: playbooks and least-privilege Kubernetes permissions
├── scripts/ Operations, security and export
├── seguridad/ Trivy Operator and Grafana dashboard
└── monitoring/ Grafana with a persistent volume
License
MIT.