Why a bridge won’t do it

Incus puts containers behind a virtual bridge (usually incusbr0) that NATs them onto the host’s network. Great for wired access. What it can’t do is make the container authenticate to a Wi-Fi access point as its own client, because Wi-Fi association is a property of the radio itself. A bridge is layer 2 and 3; Wi-Fi client auth is layer 1 with a handshake. So if you want the container to show up on the AP as its own device, you hand it the physical dongle and let it do the talking.

Once passed through, the dongle vanishes from the host’s network stack and reappears inside the container as wlan0. The host can’t use that radio while it’s gone, which is exactly the point: the container owns it.

The trade-off you can’t skip

Attaching a physical device requires the container to run privileged (security.privileged=true). That loosens how much the container shares with the host’s kernel namespaces, and it visibly weakens the isolation that makes unprivileged containers worth using in the first place. Incus runs unprivileged by default for a reason. I’m fine with this on a box I own, running workloads I trust. I would not do it for anything I didn’t. If the code inside that container is untrusted, this is the wrong tool and you should stop here.

What actually happens, step by step

The mechanics are simple once you see the shape of it. Plug the dongle in, then find what the kernel called it:

ip link show        # look for a wlan*/wl* entry that appeared when you plugged it in
iw dev              # note the phy# (e.g. phy0) it's attached to

The thing people get wrong is the next line. You attach the device with nictype=physical, and the parent is the phy index from iw dev (phy0, phy1, …), not the wlan name. Mix those up and the device simply never shows up in the container, which is the most common way this whole exercise fails:

incus config set <container> security.privileged=true
incus config device add <container> wlan0 nic nictype=physical parent=phy0 name=wlan0

Inside, the device is a Wi-Fi radio, so I name it wlan0 to match what wpa_supplicant expects rather than leaving it as a generic eth0.

There’s a chicken-and-egg moment: before the dongle can authenticate, the container needs internet to install the supplicant tools. I borrow the host bridge just long enough to get them, then pull it back out:

incus config device add <container> tempnet nic network=incusbr0
incus exec <container> bash
  dhclient eth0
  apt update && apt install -y wpasupplicant wireless-tools iw
incus config device remove <container> tempnet

Then it’s just standard Wi-Fi from inside. Write the network config, bring the supplicant up on the passed-through interface, and ask the router for a lease:

wpa_passphrase "Your_WiFi_Name" "Your_WiFi_Password" > /etc/wpa_supplicant/wpa_supplicant.conf
wpa_supplicant -B -i wlan0 -c /etc/wpa_supplicant/wpa_supplicant.conf
ip addr flush dev wlan0
dhclient -v wlan0
iw dev wlan0 link      # confirm you're associated

Moving to another network later is the same dance with a fresh wpa_passphrase and a pkill wpa_supplicant before you restart it.

Things that will bite you

• parent= is the phy index, not the interface name. This is the number one reason the device “doesn’t appear” in the container.
• A privileged container holding a physical device is, for that radio, effectively standing on the host’s side of the network. Keep it on networks you trust.
• None of this survives a host reboot on its own. After a restart you re-run the supplicant steps, or you wire them into the container’s own init if you want it to come back automatically.

It’s a small amount of config for a genuinely useful capability: a container that is a first-class Wi-Fi citizen instead of a guest behind the host. Just remember you paid for it with isolation, and spend that privilege where it’s actually safe.

Why Incus over Proxmox

Incus is what came out of LXD, kept alive by the original LXC team at Linux Containers. Same mental model: one daemon (incusd) you talk to with a client (incus), running both system containers (LXC, sharing the host kernel, very light) and virtual machines (QEMU/KVM, full guests, stronger isolation). If you’ve touched LXD you already know most of it; the commands even match (incus launch, incus exec, incus config device add).

What pulled me over was that Incus doesn’t insist on being the whole OS. You drop the package onto the distro you already run, run incus admin init, and you’re set. Proxmox wants to own the machine. Incus is happy being a layer on top of one. That difference matters more in practice than it sounds on paper.

The Web UI caught me off guard

Proxmox ships a polished web UI and that’s a real reason people like it for day-to-day clicking. Incus doesn’t include one by default, but the Incus Web UI (the incus-ui-canonical frontend) is genuinely good once you switch it on:

incus config set core.https_address :8443
incus webui

Point a browser at the host on :8443, accept the cert, and you get a clean console for launching, shelling into, snapshotting, and networking instances, VMs and LXCs alike. I do most routine management there now instead of the CLI, which surprised me because I expected to live in the terminal. For a home lab, that UI is the thing that makes Incus feel as turnkey as Proxmox.

Running on two architectures

My lab isn’t a single box. I’ve got amd64 and arm64 hosts, and the arm64 ones are quiet and cheap, so they carry the always-on services. Incus handles both natively, and the image server tags architectures cleanly (images:debian/12/arm64, images:debian/12/amd64), so picking the right image per host is trivial. That mixed fleet is where Proxmox has historically lagged, and it’s a real reason Incus fits my setup better.

Where it broke down: clustering across architectures

I started the way you’d expect: stand up an Incus cluster, join the nodes, get one unified view, live-migrate VMs between hosts. The clustering story is arguably nicer than Proxmox’s Corosync-based one, with no hard node limit and the ability to rebalance VMs off a busy node.

Then I ran into the architecture wall, and the detail here changed between Incus versions so it’s worth getting right. Copying or moving an instance across architectures used to be refused outright, because Incus did an architecture check on incus copy/incus move. That check was removed in Incus 0.6 (early 2024), so today you can copy an arm64 container onto an amd64 host and vice versa. What still fails is running it there: a container’s binaries are built for one ISA, so an amd64 container won’t start on an arm64 host, and incus start throws an architecture mismatch. VMs under KVM are the same story, the guest OS has to target the host’s architecture to boot, even though the migration itself succeeds.

So the cluster gave me a tidy console and the ability to relocate instances across arch, which is handy for storage and backups or for starting them later on a same-arch node. What it did not give me was “run anything anywhere regardless of CPU.” After a while the operational overhead, keeping members healthy, wrangling certificates and trust, rebalancing that only helped VMs, stopped paying for itself against that hard limit. If you’re on an older Incus and hit Requested architecture isn't supported by this host on a copy, that’s the pre-0.6 behaviour; upgrade and the copy goes through, just don’t expect to start it on the wrong arch.

Two more reasons it beat Proxmox

Both come from the same root: Incus is a daemon on a normal distro, not an OS you boot into.

It runs OCI images directly. Since Incus 6.3 you can register Docker Hub as an OCI remote and launch application containers natively:

1incus remote add docker https://docker.io --protocol=oci
2incus launch docker:nginx my-nginx

No Docker engine required, Incus runs the image itself. Be clear-eyed about it, though. Incus’s main model is the system container, a long-lived, VM-like full OS, while OCI/Docker images are application containers, ephemeral and task-focused. The Incus team added OCI support but treat it as a secondary path, not the headliner, and it’s newer, so rough edges remain (GPU passthrough into OCI, local-image import, that sort of thing). For a throwaway nginx or a quick experiment it’s slick. For a stack of interdependent services I still reach for Docker Compose.

It also sits alongside Docker and Podman on the same host without complaint. Because incusd is just another service on the distro, I can apt install docker.io or podman next to it and run OCI containers as a peer of Incus, not nested inside it. That’s the clean way to do it: Incus owns the system containers and VMs, Docker/Podman owns the ephemeral app workloads, both sharing the host kernel through different mechanisms. What you don’t want is the Docker engine running inside an unprivileged Incus container, nested containers need privilege and cgroup fiddling and tend to surprise you on network isolation. Keep them side by side on the host and they get along.

1# On the Incus host, as a peer, not inside a container:
2apt install -y docker.io
3docker run -d -p 8080:80 nginx

What I run today

I dropped clustering and now run each host as a standalone Incus server, split by architecture:

The amd64 host carries the heavier services and the VMs that need x86. The arm64 host carries lightweight always-on LXC containers, DNS, small Go services, the odds and ends. I’ve made peace with a workload living on one architecture and staying there. When I need it elsewhere I rebuild the image for that arch, or run a multi-arch container image, rather than migrating. That’s a small, predictable tax instead of a cluster I maintain for a benefit I couldn’t fully use.

In hindsight the lesson is simple: clustering is worth it when your nodes are the same architecture, or when your VMs really are portable. With a mixed amd64 and arm64 fleet, running standalone per arch is just simpler, and for a home lab it’s honestly as effective.

What I’d tell myself at the start

Unprivileged is the default for a reason. Flip security.privileged=true only when you have a concrete need, like device passthrough for a Wi-Fi dongle (there’s a whole post on that). Most workloads never need it. And don’t suffer the CLI out of pride, the Web UI is one command away.

Plan around architectures from day one. Pick an image tag per host, /amd64 or /arm64, and decide where a service lives before you build it. You can copy or move a container to a different-arch host now that Incus 0.6 dropped that block, but you can’t start it there, so design workloads per ISA instead of promising yourself you’ll migrate later. The real productivity win, though, is snapshots and profiles: a decent default profile carrying storage, the bridge, and your SSH key turns a useful container into a one-liner, and incus snapshot has bailed me out more than once.

Bottom line

Incus replaced Proxmox in my home lab because it’s lighter, distro-agnostic, and native on arm64, and the Web UI closes the everyday UX gap. Clustering was a fun experiment I retired once I hit the cross-architecture wall. If your nodes share an architecture, keep the cluster. If they don’t, don’t.

So I wanted one rule, stated simply and annoying to enforce:

Trusted devices can reach IoT devices. IoT devices cannot reach trusted devices.

And I added a constraint that made it interesting: use only the hardware I already own. No new access point, no paid firewall, no subscription. This is the story of how I got there, including the dead ends, because the dead ends are where I actually learned something.

The starting point

My edge is a Calix Gigaspire from the ISP. It gives me a trusted Wi-Fi on 10.10.1.0/24 and a guest Wi-Fi on 192.168.0.0/24, isolated from trusted. What it does not give me is the ability to add static routes. The two LANs are fully isolated at layer 3, and the gateway (10.10.1.1) lives inside the trusted subnet.

On the IoT side I had an OpenWrt box acting as the IoT gateway and AP: its WAN on the guest LAN (192.168.0.x), its LAN on 172.16.1.0/24 with its own IoT Wi-Fi. The homelab runs on an Incus host on the trusted network.

That’s the whole inventory. The puzzle was to make trusted reach IoT one way, with exactly that pile.

Dead end: OPNsense as the trusted gateway

My first instinct was the textbook one. Stand up OPNsense in a VM, make it the router between trusted and IoT, write a zone firewall: trusted to IoT allow, IoT to trusted deny.

It failed, and the reasons are worth sitting with:

1. The ISP router won’t take a static route. Trusted devices use the Calix as their gateway, and the Calix has no route to 172.16.1.0/24, nor will it accept one. So trusted traffic to IoT dies at the Calix.
2. The subnet topology fights you. The Calix gateway (10.10.1.1) sits inside the trusted LAN. For OPNsense to be the gateway, its WAN and LAN would both have to be 10.10.1.0/24, the same subnet on two interfaces, which OPNsense simply won’t allow.
3. Kea DHCP won’t serve a WAN interface. Even when I forced the trusted network onto OPNsense’s WAN, the DHCP server refused to run there.

The lesson stuck with me: when the ISP router is the gateway and won’t do static routes, you can’t make a downstream box the trusted gateway without fighting the subnet topology the whole way down.

The pivot: stop routing, start tunneling

The thing that broke the logjam was realizing I didn’t need to route trusted to IoT at layer 3. I needed trusted devices to reach IoT. A tunnel does that without touching the Calix’s routing table.

Raw WireGuard would work, but I’d be hand-managing keys per device, and anything off-LAN means port forwarding. Tailscale is WireGuard underneath with automatic key exchange, NAT traversal, and magic DNS, so it won. Then the elegant bit: OpenWrt can run Tailscale. I turned the OpenWrt IoT box into a Tailscale subnet router for 172.16.1.0/24.

The architecture that worked

Trusted devices join my Tailscale tailnet and get a route to 172.16.1.0/24 through the OpenWrt subnet router. IoT devices stay behind OpenWrt, which itself sits on the isolated guest LAN.

How the one-way rule actually holds

This is the subtle part, and it’s easy to get wrong by assuming one control point does all the work. There are two, and both matter:

The real insight is that IoT to trusted is already blocked by the topology (IoT lives on the isolated guest LAN behind OpenWrt, with no route to 10.10.1.0/24). The Tailscale ACL is a second layer. Defense in depth, not a single gate.

What it costs you

This is good enough for home, not enterprise, and I’ll be honest about the trade-offs:

1. Third-party metadata. Tailscale sees your node graph and relay metadata. The traffic itself is end-to-end WireGuard. If that bugs you, raw WireGuard is the alternative.
2. Control-plane dependency. If Tailscale’s coordination server is down, new connections can’t be negotiated. Tunnels already up keep running.
3. OpenWrt storage. The Tailscale package needs flash and RAM. On a tiny router it may not fit, in which case run the subnet router as a container on the homelab host instead.
4. No layer 2 segmentation. This is layer 3 plus a tunnel. Turn on client isolation on the IoT SSID for real device-to-device isolation.
5. mDNS doesn’t cross. HomeKit, AirPlay, Chromecast from trusted to IoT won’t work without an mDNS reflector.
6. The Calix is still the perimeter. Its guest/trusted isolation is doing real work. Verify it every so often.

What I’d do differently

If I were building from scratch I’d buy a VLAN-capable router, UniFi or Omada or OPNsense on a mini-PC, and do this with real VLANs and a zone firewall. But given the “use what I have” constraint, the Tailscale-on-OpenWrt approach got me the security property I wanted with zero new hardware and an afternoon of tinkering.

The takeaway is less “buy a fancy firewall” and more “know what your ISP router will and won’t let you do, then tunnel around the parts it won’t.”

First, see the bill

You can’t optimize what you can’t attribute. Before touching anything:

• Turn on tagging and enforce it. Every resource needs owner, env, service, cost-center. No tag, no deploy.
• Use cost allocation reports per team or service so each team sees its own number.
• Set a budget and an anomaly alert. A weekly “this service cost $X” email beats a monthly shock.

Visibility is most of the win. Most teams find 10 to 20 percent of spend on idle or orphaned resources in the first week of actually looking.

The levers that actually move the number

Not all optimizations are equal, so rank by impact.

Compute shape and utilization is the biggest leak. Oversized instances are the number one culprit. Right-size from real CPU and memory metrics, not guesses. A service averaging 12 percent CPU on a 4-vCPU box is paying for headroom it never uses.

Stop paying for idle time. Dev and test environments don’t need to run at 2am or on weekends. Scheduled shutdown and start saves around 65 percent on non-prod.

Commitment discounts, reserved instances, savings plans, matter if a baseline workload runs 24/7. A one to three year commitment typically cuts 30 to 60 percent versus on-demand. Buy commitments for the stable baseline only and keep the spiky part on-demand.

Storage lifecycle. Move cold data down tiers, standard to infrequent to archive. Delete orphaned volumes and snapshots. Egress is the sneaky one, so watch cross-region and NAT gateway traffic.

Networking egress. Data out is expensive and easy to ignore. Cache at the edge, keep chatter in-region, and scrutinize any architecture that moves bytes between regions for no reason.

A workflow, not a one-off

Cost optimization decays. The team that ran a “cost cleanup sprint” last quarter is over budget again next quarter unless the discipline sticks. Make it continuous.

Architecture review includes a cost estimate. New service? Estimate the monthly run cost before approval. “Cheap to build, four grand a month to run” should be a conscious decision.

Right-size on a schedule, a quarterly utilization review per service.

Automate the easy wins. Auto-stop non-prod on a schedule, auto-delete orphaned volumes after N days, alert on untagged resources.

Track cost per transaction or per user, not just total. A total can hide a service that got three times more expensive per request.

Common mistakes

Over-buying commitments to hit a discount percentage, then not using them. Commit the stable baseline and let the rest float.

Chasing pennies, missing dollars. Tuning a Lambda to save five bucks a month while a forgotten oversized database costs nine hundred. Fix the biggest line item first.

No ownership. When cost is nobody’s job, it only improves during crises. Give each team its number.

Ignoring data transfer. Compute gets the attention, while egress and inter-AZ traffic quietly dominate.

The point

Cloud cost engineering isn’t about being cheap, it’s about spending deliberately. The goal is that every dollar on the bill corresponds to a decision someone made, not a default someone forgot. Build the visibility, pull the big levers, and bake a cost estimate into the review gate. The savings compound, and the discipline is what makes them stick.

What problem does it solve?

At its core, Kubernetes is a scheduler and reconciler for containers. You hand it a desired state (“run 3 copies of this image, expose port 8080”), and a control loop keeps reality matched to that state. A node dies, it reschedules. Traffic spikes, it scales, if you asked it to. A config changes, it rolls out without downtime.

That reconciliation loop, observe current state, compare to desired, act to close the gap, is the one idea underneath everything else.

The control plane

The control plane is the brain, and these are its parts:

• API server is the front door. Every kubectl command is an HTTP call to it, and it’s the only component that talks to etcd.
• etcd is the strongly-consistent datastore. The entire cluster state lives here. Back it up, seriously.
• Scheduler watches for unscheduled pods and picks a node for each, based on resources, affinity, and taints or tolerations.
• Controller managers are the reconcilers: deployment controller, node controller, endpoint controller, and friends. Each owns one resource type’s “make reality match spec” loop.
• kubelet is the agent on every node. It talks to the API server and runs pods through the container runtime.
• kube-proxy keeps node networking rules in place so Services route to the right pods.

The objects you’ll actually use

You don’t need to memorize the whole API. The daily drivers:

• Pod is one or more containers that share network and storage. It’s the smallest thing you deploy, but you rarely create pods directly. You create something that creates pods.
• Deployment declares how many replicas of a pod and which image. Rolling updates and rollback come free with it.
• Service gives stable networking and load balancing to a set of pods, by label selector. ClusterIP is internal, NodePort and LoadBalancer are external.
• ConfigMap and Secret hold config and credentials, mounted as files or env vars. Don’t bake secrets into images.
• Ingress is HTTP routing: path /api goes to Service X, host app.example goes to Service Y. It sits in front of Services.
• Namespace is a scoping boundary for names and RBAC. It is not a network boundary; that’s NetworkPolicy.

A useful mental model

Requests flow down, Ingress to Service to Pod. Desired state flows up: you write a Deployment, the controller makes pods, the scheduler places them, the kubelet runs them.

Where people get stuck

Confusing Deployment and Pod. Edit the Deployment’s image, don’t kubectl edit pod, it’ll get replaced.

Forgetting probes. Liveness and readiness probes are how Kubernetes knows a pod is actually serving, not just that it started. Without them, traffic lands on broken pods.

Secret hygiene. Turn on encryption at rest for etcd, and use an external secret store, Sealed Secrets, External Secrets Operator, or your cloud’s KMS, rather than committing YAML.

Resource requests and limits. Set requests so the scheduler places pods sanely, and set limits so one bad pod can’t eat the node.

When not to use it

Kubernetes is a platform, not a product. If you have one small service with steady traffic, a single container on a managed service, or even a PaaS, is simpler and cheaper. Reaching for Kubernetes because it’s fashionable is how teams end up operating a distributed system they never needed. Adopt it when you have multiple services, multiple teams, or real scaling and orchestration needs.

Closing thought

Learn it from the reconciliation loop outward. Once “observe, compare, act” clicks, every other object, HorizontalPodAutoscaler, CronJob, Operator, is just another specialized reconciler. The rest really is YAML.