The overlay that pinged but wouldn't carry TCP

The peer showed up in the overlay's device list. tailscale ping came back with a pong in 23 milliseconds, over a direct path.

Then I pointed a browser at it. Nothing. nc to port 80: timed out. nc to a port I knew the daemon was listening on: timed out. tailscale serve, which exists to make exactly this easy: timed out.

That went on for hours, and every signal saying "the network is up" stayed green, because in a narrow sense it was.

I spent most of those hours reading green signals as progress.

What we were building

We'd stood up a self-hosted secrets manager for AcmeCo (posts 4–5 are about filling it). It runs off-cluster in an LXC container on the Proxmox host: Docker inside the container, Compose, the app, its Postgres, its Redis. Call the container secrets-host.

The cluster reaches it over the LAN, which is correct and boring. The humans needed the web UI from their laptops, over an overlay network, with real HTTPS. That part ate the afternoon.

Step one was getting a VPN client to run in there at all. An LXC container has no TUN device unless you hand it one, and Docker inside it fights AppArmor. The config ended up:

# /etc/pve/lxc/<ctid>.conf (excerpt) -- privileged container
features: nesting=1
lxc.apparmor.profile: unconfined
lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file

10:200 is the major and minor number of /dev/net/tun. The two lxc. lines are the ones Tailscale's own docs give for Proxmox 7 and later. The unconfined line is the one the Proxmox docs say they don't recommend. They're right. We turned AppArmor off on the box that holds the secrets, and I'd rather say that plainly than have you find it in the config. Every Compose service needed security_opt: ["apparmor=unconfined"] as well. Nested confinement is a whole topic, and this is not the post for it.

If you can avoid all of that, avoid it. Tailscale documents a userspace-networking mode that doesn't need /dev/net/tun at all. We didn't try it, which will matter later.

What was actually happening

A tunnel is a stack of claims, and each check only tests the layers it passes through:

• Peer discovery means the coordination plane knows both nodes exist.
• tailscale ping (the default kind) is a disco ping: Tailscale's own path-discovery protocol, daemon to daemon. It doesn't even use the WireGuard data plane. The --tsmp variant does, and the CLI reference says plainly that it still skips both hosts' OS network stacks.
• A TCP connection needs the packet to leave the tunnel, go through the peer's kernel, its netfilter rules, the container's view of the interface, and reach a listening socket. Then the SYN-ACK has to make the whole trip back.

Every check I had was answering at the first two layers. The failure was at the third.

That's post 2 in a smaller box. The tunnel was vouching for the tunnel. A pong from Tailscale is Tailscale's opinion of Tailscale.

The proof

Here's the L3-versus-L4 split, the way I should have run it in the first ten minutes instead of the fourth hour. Overlay addresses below are illustrative.

1. Test the protocol, not the path.

$ tailscale ping secrets-host
pong from secrets-host (100.64.0.12) via 203.0.113.7:41641 in 23ms

$ nc -vz -w 5 100.64.0.12 80
nc: connectx to 100.64.0.12 port 80 (tcp) failed: Operation timed out

$ curl -v --connect-timeout 5 http://100.64.0.12/
*   Trying 100.64.0.12:80...
* Connection timed out after 5002 milliseconds

Read the error text. It's diagnostic. Operation timed out means the SYN went somewhere and nothing answered. Connection refused means a kernel got the packet and sent back a RST, so routing works and nobody is listening. No route to host means something refused to route it, usually your own machine. Three different bugs, and I'd been filing all of them under "network's flaky."

2. Watch both ends at once.

# laptop: are SYNs actually entering the tunnel?
sudo tcpdump -ni utun4 'tcp port 80'

# inside secrets-host: do they come out the other side?
tcpdump -ni tailscale0 'tcp port 80'

Interface names vary, so check ifconfig or ip link first. Then read the result as a decision tree:

• SYNs leave the laptop, never show up in the container: the problem is in the tunnel or the container's interface plumbing.
• SYNs arrive, no SYN-ACK goes back: something local is dropping or not listening. Check netfilter, ss -ltnp, and whether Docker bound the port where you think it did.
• SYN-ACKs go back but never reach the laptop: the return path.
• No SYNs leave the laptop at all: the client never put them in the tunnel.

3. Swap one variable at a time.

We had a second overlay available, ZeroTier. Same laptop, same container, same Docker stack, same port:

$ nc -vz -w 5 10.0.0.21 80
Connection to 10.0.0.21 port 80 [tcp/http] succeeded!

First try. Inbound TCP through ZeroTier into the LXC just worked.

Now, fairness, because this is a war story and not a review. What failed was one combination: a privileged Proxmox LXC running Docker with AppArmor unconfined, and the Mac's App Store variant of Tailscale, a sandboxed Network Extension. Tailscale ships two other macOS variants and a userspace mode. We tried none of them, and never ran tcpdump on both ends while it failed. ZeroTier worked, and we had a UI to reach.

So I have an observation, not a root cause. My best guess is the in-container TUN or netfilter path. Anyone who turns this into "Tailscale doesn't work in LXC" is doing the thing this series exists to stop.

One more trap from that detour: tailscale serve --http=80 claimed port 80 inside the container, so Docker's published port 80 lost the fight. Remove the serve config before you put a reverse proxy on that port.

The second trap: it only worked when someone was watching

With the overlay carrying TCP, the next job was making it durable. The laptop reaches the Kubernetes API through an SSH port-forward over the overlay. The obvious home for that is a launchd agent running autossh, so it comes back after sleep and reboot.

The command worked in a terminal. It still worked with the environment stripped:

env -i /opt/homebrew/bin/autossh -M 0 -N -L 6443:localhost:6443 ops@10.0.0.30

The same command under launchd:

ssh: connect to host 10.0.0.30 port 22: No route to host

Same binary, same arguments, same user. The only difference was who launched it.

At the time, I concluded the overlay's route was only usable from an interactive shell context, wrote that into the ops notes, and we moved autostart into the login profile:

# ~/.zprofile: bring the tunnel up when a terminal opens
if ! lsof -nP -iTCP:6443 -sTCP:LISTEN >/dev/null 2>&1; then
  AUTOSSH_GATETIME=0 autossh -M 0 -f -N \
    -o ServerAliveInterval=15 -o ServerAliveCountMax=3 \
    -L 6443:localhost:6443 ops@10.0.0.30
fi

-M 0 drops autossh's monitor port in favour of SSH keepalives; AUTOSSH_GATETIME=0 retries even an immediate failure. It's been stable since.

Writing this post, I checked my explanation against Apple's docs, and I'm no longer confident in it.

Apple's technote on local network privacy (TN3179) says macOS automatically allows local network access for launchd daemons, for anything running as root, and for command-line tools run from Terminal or over SSH. It says that exception doesn't apply to launchd agents. On macOS, ZeroTier presents its network as a pair of feth interfaces, which are Ethernet-type. That plausibly makes the overlay subnet "local network" as far as the OS is concerned. And when local network privacy blocks a connection, the error people report is EHOSTUNREACH: No route to host.

That fits every observation better than my routing theory, including why env -i changed nothing: the environment was never the problem, the launcher was. I haven't re-tested it, so it's the leading hypothesis, not a finding. What I'm sure of is that "only works from a shell" was a description I'd filed as a cause. I was running inside your machine's rules and never checked which ones.

Last mile: HTTPS on an address the internet can't reach

Let's Encrypt can't reach an overlay IP, so HTTP-01 is out. DNS-01 works, so Caddy needed a DNS provider plugin. Building one inside the LXC failed (AppArmor again, on the build step's containers), so we used a prebuilt image. Then every request 502'd:

dial tcp: lookup backend: no such host

Caddy wasn't on the app's Compose network. One networks: entry. (The DNS API token got pasted into chat during setup. Post 4 applies; it went on the rotation list.)

Why it fooled us

Three green signals, all true: discovered, pinged, container up. I counted each as progress, when all three answered the same question at the same layer.

Then I tested the tunnel from the context I happened to be in, a terminal, which is the most forgiving place on a Mac to test networking from. Nothing that runs in the background lives there.

The checks that actually prove it

1. Test the protocol you'll use. nc -vz for TCP, curl -v for HTTP. A ping, of any kind, only proves ping.
2. Read the failure verb. Timed out, refused, and no route to host are three different bugs.
3. tcpdump both ends at the same time, and locate the last point where the SYN is still visible.
4. Test from the origin context it will run in. For launchd, that means a real agent: launchctl bootstrap gui/$(id -u) <plist>, then read its StandardErrorPath. Use the exact binary. Apple's own tools and ad-hoc-signed Homebrew binaries have been reported to behave differently here.
5. Change one variable at a time, and write down which combination you tested before you blame a product.
6. Record the observation separately from the explanation. The observation in my notes was right. The explanation was a guess written up like a fact.

Rule to steal

Test the protocol and the origin context you'll actually use. Ping is not connectivity: a pong proves the tunnel can talk to itself, and a terminal proves only what a terminal can do. Before you call the network up, run nc and curl from the same launcher, with the same binary, that production will use.

Next: The control plane was flapping because of a spinning disk — the nodes were healthy, the network was fine, and etcd was on rust.