Installer reference

install.sh is a single bash script that turns a fresh Ubuntu server into a Kwerft cluster. It also joins servers to a cluster, installs remote clusters in agent mode, restores a console from a backup, rescues a locked firewall and uninstalls. Every stage is idempotent: running the same command again repairs or upgrades the install. For a first install, start with Getting started.

Run it

The script runs as root. kwerft.dev/install.sh forwards to the script in the public ehilzinger/kwerft-install repository on GitHub:

curl -fsSL https://kwerft.dev/install.sh | sudo bash -s -- --domain ops.example.com --email ops@example.com --yes

Or download it first, read it, and run it:

curl -fsSLO https://kwerft.dev/install.sh
sudo bash install.sh --help

The whole script is a set of functions called on its last line, so a download that breaks off halfway never runs half a script.

Pin a version

https://kwerft.dev/install.sh is always the latest stable release (v0.4.0 at the time of writing). Every release, release candidates included, also has its own copy at https://kwerft.dev/<version>/install.sh:

curl -fsSL https://kwerft.dev/v0.4.0/install.sh | sudo bash -s -- --domain ops.example.com --yes

A pinned script installs its own version. To try what is in release candidates, pin the newest one, v0.6.0-rc.9:

curl -fsSL https://kwerft.dev/v0.6.0-rc.9/install.sh | sudo bash -s -- --domain ops.example.com --yes

v0.6.0-rc.9 contains everything from the v0.5.0 release candidates (Clusters & nodes, the Hetzner Cloud integration and several of the flags below) and adds backups, upgrades from the console, secret sets, templates and the Compose import, the infrastructure map and disk health for dedicated servers. Release candidates are for trying things out; expect changes before the release.

--version picks the Kwerft release (console image and Helm chart) independently of the script; it takes 0.4.0 or v0.4.0. The installer checks that the release is published before it changes anything and stops with exit code 50 if it is not. Prefer the script of the release you install, since flags and stages change between releases.

The console image is ghcr.io/ehilzinger/kwerft (amd64 and arm64) and the chart oci://ghcr.io/ehilzinger/charts/kwerft; both are public, so no registry login is needed.

Install from a Hetzner cloud config

A new Hetzner Cloud server can install Kwerft on its own while it boots: paste this into Cloud config when you create the server (Ubuntu image, 8 GB of RAM or more), and add your SSH key as usual.

#cloud-config
runcmd:
  - curl -fsSL https://kwerft.dev/install.sh | bash -s -- --yes --email you@example.com

• cloud-init runs the command as root, so there is no sudo. --yes skips every prompt.
• Pin a release with https://kwerft.dev/v<version>/install.sh (see Pin a version).
• Add --domain ops.example.com only if that name’s DNS record already points at the new server’s address, which you know only once the server exists. Without it the console comes up on <address>.sslip.io; set your own hostname later under Settings.
• The install takes about five minutes after the server starts. Follow it with ssh root@<address> tail -f /var/log/kwerft/install.log; cloud-init’s own output is in /var/log/cloud-init-output.log.
• Then create the owner account with the setup token: ssh root@<address> cat /etc/kwerft/setup-token, and open the /setup address the log’s last lines print (see The setup token).

Kwerft’s own node pools start their servers the same way; their cloud config holds only a short-lived join token.

Options

Flags marked v0.5.0 exist from the v0.5.0 release candidates on, flags marked v0.6.0 from v0.6.0-rc.1 on.

Install

Join an existing cluster

You rarely type these yourself: the console’s join command (Clusters & nodes › a cluster › Nodes › Join a server) and node pools pass them.

Connect a new cluster to a console (agent mode)

v0.5.0. Installs Kwerft without its own console; the cluster is managed from another console. The console’s Adopt a cluster dialog shows the whole command. See Clusters & nodes.

A server keeps the mode it was installed in: --agent on a console server, or a plain install on an agent server, stops with exit code 2.

Development

Restore onto a new server

v0.6.0. See Restore from a backup and Backups.

Maintenance

General

Environment variables

Options with a value can also come from the environment; flags win. This suits cloud-init and CI:

Switches such as --yes, --dry-run or --agent have no environment variable; pass them as flags. Before v0.6.0, --lite and --harden-ssh had none either.

curl -fsSL https://kwerft.dev/install.sh \
  | sudo KWERFT_DOMAIN=ops.example.com bash -s -- --yes

Config file

--config FILE reads a YAML file for unattended installs. Flags still win over it. Nested keys can be written as a block or inline (dns: { solver: hetzner, tokenFile: /root/dns.token }).

domain: ops.example.com
email: ops@example.com
appsDomain: apps.example.com
dns:
  solver: hetzner
  tokenFile: /root/dns.token
  records: true
hcloud:
  tokenFile: /root/hcloud.token

An apps domain or DNS solver given here replaces what was chosen in Settings; without them, Settings stays as it is. The token files stay where they are; the cluster keeps a copy that the console never shows. See Domains & TLS and Clusters & nodes.

With owner: the summary says owner you@example.com from --config and no setup token is created. Without it, or if the console rejects the owner (an invalid address or password, or accounts that already exist), the install hands out a setup token as usual.

Stages

The installer runs these stages in order and prints one line per stage. Completed stages are recorded in /var/lib/kwerft/stages/. The column on the right is v0.6.0’s behaviour; before it, System and Helm also ran only once.

In agent mode, Backups, Restore and Handoff do not run. In join mode the stages are Preflight, System, Firewall, Join cluster and Registry mirror.

--dry-run lists the stages it would run without running any.

The installer’s own log is /var/log/kwerft/install.log. When a stage fails, the installer names it, its exit code and the log, and tells you to re-run the same command.

Re-run to repair or upgrade

Run the same command again at any time. Re-running converges everything but Kubernetes:

• Repair: stages that always run converge the server back to the expected state; completed one-time stages are skipped.
• Upgrade: the script of a newer release upgrades Kwerft and the components pinned in it (Cilium, cert-manager, Traefik, monitoring, and from v0.6.0 Velero, the system-upgrade-controller, system packages and Helm). k3s keeps the version it was installed with, on every node; from v0.6.0 it is upgraded from the console. From v0.6.0 on, the console also upgrades Kwerft itself, with automatic rollback: see Upgrades.
• Settings are kept: without --domain, the console’s hostname stays what Settings chose. Other settings are remembered from v0.6.0 on (next section); before v0.6.0, --acme-server must be given on every run, or certificates come from Let’s Encrypt production again.
• The firewall stays up: reloading the firewall replaces Kwerft’s base rules in one step and keeps the console’s rules, so there is no moment without a firewall.

Remembered settings

v0.6.0. Every run writes its settings to /var/lib/kwerft/install.env (root only), and later runs reuse them, so a re-run needs no flags: --email, --acme-server, --platform, --private-iface, --lite, --harden-ssh, --channel, the mode (install, agent or join) and, in agent mode, --console. Flags and KWERFT_* environment variables win over the file.

Not remembered: the console’s hostname (Settings is the record), tokens, --config, --version and --k3s-version. An agent cluster reads its agent token back from the cluster; a joined server re-runs without --join and --token.

A server installed before v0.6.0 has no such file yet. On its first run of a v0.6.0 installer, give the flags you installed with (for example --acme-server staging or --lite); after that they are remembered.

The setup token

The handoff stage creates a single-use setup token in /etc/kwerft/setup-token (readable by root only), valid for 24 hours; only its hash goes into the cluster. If the file is lost or the token expired before anyone used it, a re-run creates a new one. Once the owner account exists, a re-run removes any leftover token and prints the console’s address instead.

Restore from a backup

v0.6.0. On a fresh server, --restore rebuilds a console from a Cluster backup in Object Storage: projects, apps with their volume data, members, settings and history.

curl -fsSL https://kwerft.dev/v0.6.0-rc.9/install.sh | sudo bash -s -- --config /root/kwerft.yaml --restore latest --yes

The config file needs a backups: block (see Config file):

backups:
  endpoint: https://fsn1.your-objectstorage.com
  bucket: acme-kwerft
  prefix: ops.example.com
  accessKeyFile: /root/s3.access
  secretKeyFile: /root/s3.secret
  recoveryKeyFile: /root/kwerft-recovery-key.txt

The installer runs the usual stages, then Restore before the Kwerft stage. It waits up to 10 minutes for Velero to read the bucket and up to 4 hours for the restore itself. The console keeps the hostname from the backup unless --domain says otherwise; everyone signs in with their existing accounts, and no setup token is created.

--restore is refused (exit code 2) on a server that already runs Kwerft, with --lite, and with --agent, --join, --uninstall or --reset-firewall. If the restore fails, the installer stops with exit code 60; run the same command again to continue. The full procedure and its limits are on Backups.

Exit codes

Exit codes are a stable contract for automation:

Firewall rescue

If a firewall change locked you out of SSH, open the server’s console from Hetzner (VNC for Cloud servers, KVM for dedicated ones) and run:

curl -fsSL https://kwerft.dev/install.sh | sudo bash -s -- --reset-firewall

It removes Kwerft’s nftables table and pauses the console’s firewall rules on this node, so they are not put back. Run your install command again to restore the baseline, and delete /var/lib/kwerft/firewall/paused to let the console manage this node’s firewall again. See Network › Server firewall.

Uninstall

curl -fsSL https://kwerft.dev/install.sh | sudo bash -s -- --uninstall --yes

This removes k3s with every workload on the server and the data on its local volumes, Kwerft’s firewall table, the registry mirror and AppArmor profile, /var/lib/kwerft, /etc/kwerft and Kwerft’s sysctl, module and SSH files. The logs in /var/log/kwerft stay. Without --yes it asks first.

A Hetzner Cloud Firewall or Load Balancer that Kwerft created stays in your Hetzner project. Turn them off in Settings before you uninstall, or delete them in the Hetzner Console afterwards.

Uninstalling deletes your apps’ data. From v0.6.0, Kwerft can back it up to Object Storage first; a backup in the bucket is not touched by an uninstall.

Pinned components

Each release pins the versions it installs at the top of the script. The v0.6.0 release candidates pin the following; v0.4.0 and the v0.5.0 release candidates pin the same versions of the components they have.