Operations

Day-to-day procedures. Every kubectl command assumes a kubeconfig pointing at the VIP (https://192.0.2.250:6443).

Overall status

scripts/check_cluster.sh runs a read-only check: nodes, etcd members, VIP, keepalived, pods with problems, Longhorn volumes and Velero.

./scripts/check_cluster.sh

For the nodes' system updates, the update portal (scripts/mantenimiento.sh). For security, scripts/seguridad.py (see security).

Manual checks:

kubectl get nodes -o wide
kubectl get pods -A | grep -vE 'Running|Completed'
kubectl -n longhorn-system get volumes.longhorn.io        # robustness: healthy
sudo k3s etcd-snapshot list                               # on a master
journalctl -u k3s --since '1 hour ago' | grep -iE 'leader|elect|timeout' | tail

Signs that etcd is struggling:

• leaderelection lost in the k3s logs.
• A sudden rise of NRestarts in systemctl show k3s -p NRestarts.
• Many simultaneous restarts in the operators (Longhorn CSI, Kyverno, cert-manager…).

Node maintenance (drain and reboot)

kubectl drain <nodo> --ignore-daemonsets --delete-emptydir-data
# … maintenance / reboot …
kubectl uncordon <nodo>

Before a drain:

• Check what local-path volumes that node holds. Those applications won't move: they stay Pending until the node comes back.
• If it is a master:
  • The other two must be Ready and etcd healthy (3 members). Never reboot two masters at once.
  • If it is master-1, first check that the VIP has a standby (keepalived in BACKUP on another master, not in FAULT). If it doesn't, use another master's IP in the kubeconfig for the duration of the maintenance.
• If it is the worker-1 (HDD), wait for Longhorn to finish rebuilding replicas before touching it again.
• Longhorn uses node-drain-policy: allow-if-replica-is-stopped (since 2026-11-03): a detached volume with its last replica on the node no longer blocks the drain; one in use still does. If the drain is stuck waiting on the instance-manager PDB, the longhorn-manager log says which replica is blocking it (removing … PDB is blocked: replica …).
• If a portal task is cut off after the drain (without reaching its uncordon), the node is left cordoned and neither Reboot nor Update system pass their initial check. Fix the cause, kubectl uncordon <node> and run it again (incident of 03/11).

If the drain is stuck waiting on Longhorn's instance-manager PDB, it is usually an orphaned snapshot-controller-* ticket in a volumeattachments.longhorn.io: it points to a snapshot that no longer exists (it happened after the 27/10/2026 outage). It is removed from the volume's spec.attachmentTickets, only if the snapshot doesn't show up in kubectl -n longhorn-system get snapshots.longhorn.io. It doesn't delete any data. See the 28/10 incident.

# Snapshot tickets and their volumes
kubectl -n longhorn-system get volumeattachments.longhorn.io -o json \
  | jq -r '.items[]|.metadata.name as $v|(.spec.attachmentTickets//{})|keys[]|select(startswith("snapshot-controller-"))|"\($v) \(.)"'
# For each one whose snapshot (the name without "snapshot-controller-") doesn't exist:
kubectl -n longhorn-system patch volumeattachments.longhorn.io <volumen> --type json \
  -p '[{"op":"remove","path":"/spec/attachmentTickets/<ticket>"}]'

Adding a worker

See nodes/README.md. Then:

kubectl label node <nodo> node-role.kubernetes.io/worker=worker

Turning a worker into a master

1. First copy /var/lib/rancher/k3s/storage/: the agent uninstaller deletes it without warning.
2. Drain the node and run k3s-agent-uninstall.sh.
3. Install the server with the same version as the cluster:

   curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.34.6+k3s1 K3S_TOKEN=<token> sh -s - server \
     --server https://192.0.2.250:6443 --disable=servicelb \
     --node-ip=<ip> --flannel-iface=<interfaz-cableada>
   

4. Add the node to the haproxy backend on the masters and install keepalived with its priority (nodes/masters/).
5. Check that ip -d link show flannel.1 | grep local shows the node's real IP.

With 2 etcd members, quorum is both of them. If the third one doesn't finish joining, the API is lost. To recover it: k3s server --cluster-reset on the healthy master, delete db/ on the node that failed and repeat the join.

Upgrading k3s

⚠️ Don't use curl -sfL https://get.k3s.io | sh -s - server without flags to upgrade. The install script rewrites /etc/systemd/system/k3s.service with whatever flags you pass it. Without them, the master loses --node-ip, --flannel-iface, --disable=servicelb, --server… (see nodes/README.md).

The cluster has 3 masters with embedded etcd: it hasn't been a single k3s (SQLite) since the migration on 2026-07-03. Upgrades are done in turns (rolling): one node at a time, while the other two masters keep etcd's quorum.

Rules

• Masters first, then workers. Never a worker ahead of the masters.
• No skipping minor versions: 1.34 → 1.35 → 1.36. Patches within the same branch (1.34.6 → 1.34.12) can be skipped.
• One master at a time, checking etcd before moving on to the next one. Recommended order: master-1 → master-2 → master-3 (all three wired since 2026-10-27). With the master-1 down, quorum is kept by the two wired masters, which see each other with no loss.
• Patches don't need a drain: when k3s restarts, the containers keep running (containerd keeps them). That avoids moving the applications tied to a node through local-path. For a branch (minor) change, yes: drain before and uncordon after.
• Before changing branch, check compatibility for Longhorn, Strimzi, ECK, Argo CD, Kyverno, Velero and Falco, and look for removed APIs (for example with kubent or pluto).

From the portal (recommended)

The "6 · Actualizar k3s" button in Semaphore (ansible/README.md), or ./scripts/mantenimiento.sh k3s -e version=v1.34.12+k3s1 (with --check first). It runs this same procedure: masters first, one at a time (etcd snapshot, wait for Ready with the new version and [+]etcd ok), then the workers, which aren't touched if any master is behind. nodo-mantenimiento k3s validates the version (format, never backwards, at most one branch ahead, GitHub binary with its sha256). For a branch change it drains and uncordons, but the compatibility review is still manual.

By hand, on a node: scripts/k3s_parche.sh (patches only; launch it with systemd-run and the script's full path).

Per-node procedure (replacing only the binary)

Keeps the systemd unit and all the flags.

V=v1.34.12+k3s1                      # target version
ARCH=amd64; BIN=k3s                  # on the Raspberry: ARCH=arm64; BIN=k3s-arm64
U="https://github.com/k3s-io/k3s/releases/download/${V/+/%2B}"

# 1. Download and verify
curl -sfLo /tmp/$BIN "$U/$BIN"
curl -sfL "$U/sha256sum-$ARCH.txt" | grep " $BIN\$" | sed "s| $BIN\$| /tmp/$BIN|" | sha256sum -c -

# 2. Masters only: etcd snapshot before touching anything
sudo k3s etcd-snapshot save --name pre-$(echo $V | tr + -)

# 3. Replace the binary and restart
sudo install -m 755 /tmp/$BIN /usr/local/bin/k3s
sudo systemctl restart k3s           # on workers: k3s-agent

Checks before moving on to the next node:

kubectl get nodes -o wide                                  # the node, Ready and on the new version
kubectl get --raw '/readyz?verbose' | grep -E 'etcd|failed'  # [+]etcd ok
./scripts/check_cluster.sh

If SSH is flaky: run the upgrade detached with systemd-run

If the SSH connection drops (not just slows down), the server sends SIGHUP to the session and kills the script halfway. So that it finishes no matter what, it is launched as a transient systemd service:

sudo -v    # enter the password first, so it isn't asked halfway through
sudo systemd-run --unit=k3s-upgrade --collect bash -c '
set -e; V=v1.34.12+k3s1; BIN=k3s; ARCH=amd64        # Raspberry: BIN=k3s-arm64; ARCH=arm64
U="https://github.com/k3s-io/k3s/releases/download/${V/+/%2B}"
curl -sfLo /tmp/$BIN "$U/$BIN"
curl -sfL "$U/sha256sum-$ARCH.txt" | grep " $BIN$" | sed "s| $BIN$| /tmp/$BIN|" | sha256sum -c -
# masters only: k3s etcd-snapshot save --name pre-$(echo $V | tr + -)
cp -a /usr/local/bin/k3s /usr/local/bin/k3s.anterior.bak
install -m 755 /tmp/$BIN /usr/local/bin/k3s
systemctl restart k3s-agent    # on masters: k3s
echo "OK: actualizado a $V"'

journalctl -u k3s-upgrade -f    # follow progress (also after reconnecting)

The result can also be checked without SSH, through the API: kubectl get nodes -o wide or the Nodes card in Homepage.

This protects the script, not the cluster: with the wired network dropping packets, don't upgrade masters (etcd's quorum depends on the other two). Measure first: ping -c 60 -i 0.5 192.0.2.1 from a wired node must give 0 % loss.

In Homepage (http://192.0.2.242), the Nodes card shows each node's version and Control plane the number of different versions (2 while the upgrade is running and 1 when it is done).

After upgrading a master, check that local-path hasn't become the default StorageClass again.

Rolling back a patch: reinstall the previous binary, the same way as in step 3. If etcd got corrupted: k3s server --cluster-reset --cluster-reset-restore-path=<snapshot pre-…> (see below).

Node system updates

With the update portal: the Grafana dashboard Node updates shows what is missing, and ./scripts/mantenimiento.sh actualizar installs it node by node, with a drain and a reboot when needed.

• It needs node-exporter's textfile collector, enabled with helm upgrade --reuse-values -f ansible/kubernetes/node-exporter-textfile.yaml on the monitoring/prometheus release. Every future helm upgrade of kube-prometheus-stack has to pass that file again (or use --reuse-values), or the dashboard is left without data.
• Before rebooting a master, the playbook checks that the network to it is stable (30 connections to the API without failures), and checks the VIP and etcd.
• When a node reboots, the services with local-path volumes on it stop until it comes back (the playbook lists them first), and the contents of its pods' emptyDir volumes are lost, such as Prometheus's history, which has no persistent volume (Grafana does, since grafana_persistente.sh).

Grafana: persistent volume

Grafana keeps the dashboards, users, folders and alerts created by hand in /var/lib/grafana (grafana.db). Until 2026-10-26 it was an emptyDir, which is lost every time the pod changes node. scripts/grafana_persistente.sh moves it to a Longhorn PVC without losing anything:

1. Copies grafana.db to /root/.config/grafana-copia/ (root only) and checks that its sha256 matches the pod's; if the network cuts the copy, it retries. Plugins aren't copied: Grafana reinstalls them on startup.
2. helm upgrade --reuse-values with monitoring/grafana-persistente.yaml (and node-exporter's textfile collector).
3. Moves aside the empty grafana.db on the new volume, uploads the copy in 48 KB chunks (through kubectl exec's standard input it arrives truncated), checks its hash and starts Grafana.

Applied on 2026-10-26: prometheus-grafana PVC on Longhorn, database restored with an identical hash and the copy in /root/.config/grafana-copia/.

The Prometheus and Alertmanager datasources are created by the chart and don't depend on this.

• Don't touch Grafana while the script runs. Anything created between the copy (step 1) and the helm upgrade (step 2) is lost: the Recreate strategy deletes the old pod before restoring.
• Rolling back: helm upgrade --reuse-values --set grafana.persistence.enabled=false leaves Grafana with an empty emptyDir. The .tar has to be restored by hand (kubectl exec -i … -- tar xf - -C /var/lib/grafana). The monitoring/prometheus-grafana PVC isn't deleted automatically: delete it by hand once it is no longer needed.

Grafana: "database is locked"

On 2026-10-26 (with the volume freshly created and 3 replicas on master-2, worker-2 and master-3), a network drop with the master-3 replica left writes hanging for a few seconds and Grafana started failing with database is locked (SQLite is very sensitive to that). It recovered by itself when Longhorn carried on with the other replicas. So that it doesn't depend on the network, scripts/grafana_local.sh leaves the volume with 2 replicas, one always on Grafana's node (dataLocality: best-effort), removing the spare one (master-3's if possible, never the local one). Grafana has to run on a wired node (pinned to worker-2): on the master-1 (WiFi) the local replica never finishes rebuilding (see incidents). It is a change on the Longhorn volume, not in Helm: a helm upgrade doesn't undo it, but recreating the PVC (another grafana_persistente.sh) brings it back to 3 replicas without locality.

Exporting the cluster state to this repo

python3 scripts/export_cluster.py    # regenerates cluster/ (no secrets)
git diff --stat                      # review the changes
git add -A && git commit -m "Exportación del estado del clúster $(date +%F)"

The script fails if it detects anything that looks like a private key or a token. Always review the diff before pushing.

Control plane recovery