muse-reverse-ssh
Use when exposing a machine without a public IP (cloud VM, container, home server) as a publicly reachable SSH server via a reverse SSH tunnel through a VPS; setting up a persistent keepalive supervisor for the tunnel; or diagnosing dropped tunnels, GatewayPorts binding failures,
Install
npx skills add https://github.com/dengyie/awesome-skills/tree/main/muse-reverse-ssh
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install dengyie-awesome-skills@llmmart
git clone https://github.com/dengyie/awesome-skills.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole dengyie/awesome-skills collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
Muse Reverse SSH
Core Principle
The inner machine dials out; the VPS only forwards.
A reverse tunnel (ssh -R <REMOTE_PORT>:localhost:22) makes the VPS listen on a public port and forward everything to the inner machine's sshd. The inner network needs no inbound firewall rules and no public IP. Treat the tunnel itself as the server: if the tunnel process dies, the "server" is unreachable even though both machines are healthy. Monitor and keepalive the tunnel, not just the machines.
Decision Tree
Inner machine already has a public IP, or you control inbound port forwarding on its router?
-> Run sshd directly with firewall rules. Do not add a reverse tunnel.
You need HTTPS/browser access to inner services, not just SSH?
-> Prefer Cloudflare Tunnel or an nginx reverse proxy on the VPS.
Only your own devices need access (no public exposure)?
-> Prefer Tailscale or WireGuard over a publicly forwarded port.
Inner machine cannot accept inbound connections, but must be SSH-reachable from the internet?
-> Use this skill.
Roles and Keypairs
Two machines, two keypairs. Never reuse one keypair for both directions:
| Keypair | Private key lives on | Public key goes to | Purpose |
|---|---|---|---|
| Tunnel | inner machine (~/.ssh/tunnel-key) |
VPS user's ~/.ssh/authorized_keys |
inner machine authenticates TO the VPS to establish the tunnel |
| Access | operator's laptop (access-key.pem) |
inner machine user's ~/.ssh/authorized_keys |
operator authenticates THROUGH the forwarded port to the inner machine |
Generate both with ssh-keygen -t ed25519 -f <name> -N "" and keep private keys at mode 600.
Preflight
Run before changing anything; keep the output for the final report.
On the VPS (root or sudo):
ss -tlnp | grep <REMOTE_PORT> || echo "port <REMOTE_PORT> free"
grep -E "^GatewayPorts" /etc/ssh/sshd_config || echo "GatewayPorts not set"
On the inner machine:
systemctl is-active sshd || service ssh status
ss -tlnp | grep ':22 ' || echo "nothing listening on 22"
id <INNER_USER> 2>/dev/null || echo "user <INNER_USER> missing"
Interpretation:
- Port already listening on the VPS: pick another
<REMOTE_PORT>or stop the occupying process. GatewayPortsnot set (orno): remote forwards bind to127.0.0.1only — the port will NOT be publicly reachable until fixed (see VPS Setup).- Nothing on inner port 22: install and start openssh-server first.
VPS Setup (one time)
Create the tunnel user (skip if it exists):
id <VPS_USER> 2>/dev/null || useradd -m -s /bin/bash <VPS_USER> mkdir -p /home/<VPS_USER>/.ssh && chmod 700 /home/<VPS_USER>/.sshInstall the tunnel public key:
cat tunnel-key.pub >> /home/<VPS_USER>/.ssh/authorized_keys chmod 600 /home/<VPS_USER>/.ssh/authorized_keys chown -R <VPS_USER>:<VPS_USER> /home/<VPS_USER>/.sshAllow public binding of forwarded ports. In
/etc/ssh/sshd_configset:GatewayPorts yesthen
systemctl reload sshd. (Safer alternative:GatewayPorts clientspecified— see Safety Boundaries.)Open
<REMOTE_PORT>/tcp in the VPS firewall (ufw / cloud security group).
Inner Machine Setup (one time)
Install and start openssh-server:
sudo apt-get install -y openssh-server sudo systemctl enable --now sshCreate the login user and install the access public key:
id <INNER_USER> 2>/dev/null || sudo useradd -m -s /bin/bash <INNER_USER> sudo -u <INNER_USER> mkdir -p /home/<INNER_USER>/.ssh cat access-key.pub | sudo tee -a /home/<INNER_USER>/.ssh/authorized_keys >/dev/null sudo chmod 700 /home/<INNER_USER>/.ssh sudo chmod 600 /home/<INNER_USER>/.ssh/authorized_keys sudo chown -R <INNER_USER>:<INNER_USER> /home/<INNER_USER>/.sshHarden sshd (
/etc/ssh/sshd_config):PermitRootLogin prohibit-password PasswordAuthentication nothen
sudo systemctl reload ssh.Place the tunnel private key:
install -m 600 tunnel-key ~/.ssh/tunnel-key
Keepalive
The tunnel must survive network drops, not just machine reboots. Run a user-level supervisor (template: scripts/reverse-ssh-keepalive.sh):
nohup ./reverse-ssh-keepalive.sh >/dev/null 2>&1 &
Key options and why they matter:
-N: no remote command, forward only.-R <REMOTE_PORT>:localhost:22: the reverse forward itself.-o ServerAliveInterval=20 -o ServerAliveCountMax=3: detect a dead TCP connection within ~60s instead of hanging forever on a half-open socket.-o ExitOnForwardFailure=yes: fail fast if the VPS port cannot be bound (e.g. already taken) instead of sitting on a tunnel that forwards nowhere.-o BatchMode=yes -o ConnectTimeout=20: never prompt for input, never hang on connect.Atomic
mkdirlockdir for single instance (see the template script). Do not useflock -non a lock file here: the lock fd is inherited by the foregroundsshchild, so killing the supervisor leaves an orphaned ssh holding the lock forever and no new supervisor instance can ever start (observed in production). A directory cannot be inherited — with PID/cmdline stale-lock reclaim (the claim is verified via/proc/<pid>/cmdline; reclaim takes the stale lock aside with an atomic rename and re-verifies it after the rename, moving it back if it turned out to be alive — so two contenders can never both win. If the move-back itself loses to a fresh claimant, the displaced live owner is a ghost (alive but directory-less, possibly blocked in foreground ssh and never reaching its ownership check) and is terminated after a cmdline re-verify; its orphaned ssh child is reaped by the new holder's stale-tunnel cleanup), takeover is always kill-and-restart via the pidfile ($STATE_DIR/keepalive.pid, verify/proc/<pid>/cmdlinebefore killing).Write the operator-facing endpoint to a file (e.g.
current-endpoint.txt) so the access command stays discoverable:ssh -p <REMOTE_PORT> -i access-key.pem <INNER_USER>@<VPS_IP>
Boot Persistence
The keepalive supervisor only survives network drops. A machine reboot kills it silently — nohup ... & does not persist across boots, and without autostart the "server" stays down after every reboot with no alert. This is the most common cause of a tunnel that "just stops working" days later. Protect three layers independently:
Layer 1 — process supervision (ssh dies, machine stays up)
Covered by scripts/reverse-ssh-keepalive.sh (atomic-mkdir-lock-guarded restart loop). Alternatively, autossh is a drop-in replacement:
autossh -M 0 -N -o "ServerAliveInterval 30" -o "ServerAliveCountMax 3" \
-o ExitOnForwardFailure=yes -i tunnel-key.pem \
-R 0.0.0.0:<REMOTE_PORT>:localhost:22 <TUNNEL_USER>@<VPS_IP>
-M 0 disables autossh's legacy monitor port and relies on SSH keepalives. Either way, something must (re)start the supervisor itself after a reboot — that is layer 2.
Layer 2 — boot autostart (machine reboots, disk persists)
Pick the mechanism the platform actually persists (template: scripts/reverse-ssh-boot.service):
| Platform | Mechanism |
|---|---|
| systemd (most Linux servers / VPS) | unit file + systemctl enable, Restart=always |
| Linux without systemd | @reboot cron entry, or executable /etc/rc.local |
| macOS | launchd plist in ~/Library/LaunchAgents with RunAtLoad |
| Windows | Task Scheduler task with an "At startup" trigger |
Pitfalls:
HOMEmust resolve to the home holding the script, keys, and state dir. A unit running as root getsHOME=/rootby default, which silently breaks every$HOME-relative path. SetEnvironment=HOME=...explicitly.- The keepalive's atomic
mkdirsingle-instance lock makes boot restarts safe: a directory lock cannot be inherited by orphaned children, and stale locks (dead PID, or a PID whose cmdline is no longer the keepalive script) are reclaimed by taking the stale lockdir aside with an atomic rename and re-verifying it after the rename (moved back if it was actually alive) — so a fresh instance after an unclean shutdown always takes over cleanly instead of exiting as "another instance running". There is deliberately no "older than N seconds" rule for the keepalive: a healthy supervisor runs for weeks, so its timestamp is always old — an age rule would let any second instance steal the lock from a healthy first one and leave two supervisors fighting over the VPS port. (The restore script in Layer 3 is different: restores are time-bounded operations, so an age cap there is valid.)
Layer 3 — ephemeral machines (disk resets on reboot)
Containers, reset-on-boot VMs, and spot instances wipe the root filesystem on reboot: no unit file, no cron entry, no installed package survives — only a designated data disk persists, if any. Layer 2 cannot work there because there is nothing durable left to trigger it. Use instead:
- Idempotent restore script on the persistent disk. One script that reinstalls packages, recreates users and keys, and restarts the keepalive — and is safe to run repeatedly (
id <user> || useradd ...,apt-get install -yis idempotent, kill-and-restart supervisors rather than start-if-absent). - Offline package cache for the restore path. On reset-prone machines,
apt-get update/installis the slowest and flakiest part of a restore (network, mirror issues, the platform's own apt reconciliation holding the lock). Pre-download the needed.debs plus their dependency closure onto the persistent disk (apt-cache depends --recurse --no-recommends ... <pkg> | apt-get download), refresh on a schedule; the restore script installs withdpkg -i <cachedir>/*.debfirst and only falls back to apt when the cache is missing or incomplete. Measured on a real reset-prone VM: ~7s offline vs 2–4 min via apt — recovery is then dominated by detection latency alone. - External watchdog, two-tier (resident loop + supervisor). When the machine itself can run shell, do not make an external poller do the checking: put the minute-level checks in a zero-cost resident loop on the machine (
health-watch-loop.sh: every ~60–90s it runs the check script, which writes a JSON status file and self-heals), and demote the external watcher to a sparse supervisor whose only job is to verify the resident loop process lives and read its status file. This split matters most when the supervisor is an LLM/agent-driven scheduled task: every poll costs real tokens, and a 1-minute agent poll on an idle machine burned ~7% of a weekly free quota in production before this redesign. The resident loop does the same checks for free; a 5-minute supervisor cut the agent-side usage to ~1/5 of the 1-minute era (≈3–6M input tokens/day measured, vs ≈26M/day). Detection latency for a dead loop equals the supervisor interval — state that honestly instead of promising instant recovery. Five minutes is a calm default; production later tightened to 2.5 minutes after a churny platform kept killing the loop several times a night, accepting roughly double the supervisor cost (≈11M input/day) to halve the worst-case window to ≤2.5 min. Service-level faults are still caught by the loop itself in ~1 min regardless. Polling the supervisor faster barely shortens recovery while multiplying agent cost.- Supervisor liveness check is one fixed command — pidfile and
/proc/<pid>/cmdlinematch (neverkill -0alone; PIDs are reused after reboots and a stale pidfile can point at an innocent process):pid=$(cat <LOOP_PIDFILE> 2>/dev/null); if [ -n "$pid" ] && ps -o cmd= -p "$pid" 2>/dev/null | grep -q health-watch-loop.sh; then echo ALIVE; else echo DEAD; fi - If DEAD, relaunch fully detached (
setsid nohup bash <LOOP> >/dev/null 2>&1 < /dev/null &), sleep 2s, re-verify with the same check. Then do not run the check script yourself — the loop runs a check immediately on start, and a second concurrent run only collides with the check script'smkdirlock and exits as a no-op. Instead, wait (up to ~2 min) until the status file's timestamp refreshes; that refresh is the proof the relaunched loop covered this interval. - Read the status with one fixed command, never an improvised parser (an agent that re-derives JSON parsing each run wastes tokens and intermittently writes buggy one-offs that throw and retry):
python3 -c "import json;d=json.load(open('<STATUS_FILE>'));print(d['overall'],d['time'],d.get('restored'));[print(k,v['status']) for k,v in d['checks'].items() if v['status']!='ok']" - Reporting discipline: healthy means silent — the supervisor reports only on degraded/failed status, on its own execution failure, or when the loop resists relaunch. A watchdog that posts "all green" every poll trains the operator to ignore it and, if agent-driven, burns budget to say nothing. Keep the supervisor's own instructions short for the same reason: they are re-read on every poll.
- Logging discipline: append to the ops log only on anomalies (loop relaunched, any non-
okcheck, degraded). An all-green run writes nothing; a log that grows on healthy runs stops being read. - The loop's check script should check more than the tunnel: the tunnel ssh process (
pgrep -f "[s]sh.*-R <REMOTE_PORT>"plus an ESTABLISHED-socket verification — see the pitfalls),sshd, every keepalive supervisor, any other long-lived agents (with a log-freshness check for agents that can hang silently: restart if the log has not grown in N minutes), and any local origins/proxies the tunnel exists to serve. Each check gets a targeted repair first; only items that resist repair escalate to the full restore script. Template loop:scripts/health-watch-loop.sh.
- Supervisor liveness check is one fixed command — pidfile and
- VPS-side detection. The VPS is usually a normal persistent machine. A tiny check there —
ss -tlnp | grep <REMOTE_PORT>or a TCP connect attempt against the forwarded port — notices the tunnel disappearing before any human does, and is the cheapest layer-3 signal.
Watchdog and restore pitfalls (learned the hard way)
- Never guard the restore script with an outer
flock -n <restore.lock>. This was the old advice in this very skill — and it silently disabled a production watchdog for ~2.5 hours before anyone noticed. The flock file descriptor is inherited by every long-lived daemon the restore script starts (keepalive, agent, ssh), so the advisory lock is held forever by those background processes and never released; every later watchdog run then believes "a restore is already in progress" and quietly skips recovery. The watchdog keeps polling, logging healthy runs, but can never recover anything again. Fix: put the mutual exclusion inside the restore script as an atomicmkdirlockdir — a directory cannot be inherited by children, so it can only be held by the script itself — with PID/timestamp stale-lock reclaim (see the Layer 3 bullet above). The keepalive supervisor needs the same treatment: use the atomicmkdirlockdir there too.flock -n 9plus9>&-looks correct on paper, but in production the foregroundsshchild inherited the lock fd anyway; after the supervisor was killed, the orphaned ssh held the lock forever and permanently disabled recovery. - Never
set -ein the resident loop. The check script exits 1 exactly when it finds degraded state — that is its job. Underset -e, the first degraded run kills the watchdog loop itself, silently ending all checks until the supervisor happens by. Wrap each run intimeout 300as well: a check that hangs would otherwise serialize the loop forever (one stuckcurl= no more checks, ever). Discard the check's stdout (the status JSON is the source of truth) and trim its stderr file to the last ~200 lines each cycle so logs cannot grow unboundedly. - After a restore pass, the re-verify must clear first-pass failures. A common aggregation bug: the verify pass skips or keeps the first pass's
failedverdict for an item ([ "$ST" != failed ] && ST=ok-style guards), so an item the restore actually fixed still reports failed → the run endsdegradedwith exit 1 even though everything is healthy again. When the post-restore verify passes, the item's final state isfixed("recovered after restore"), never the stale first-pass failure. pgrep -fmatches the checker itself.pgrep -f "ssh.*-R <REMOTE_PORT>"also matches the very command running the check, because the pattern text appears in the checker's own command line — a dead tunnel then looks healthy forever. Use the bracket trick:pgrep -f "[s]sh.*-R <REMOTE_PORT>". The regex[s]shmatchessshin the target but not the literal[s]shin your own command line. (Same reason supervisor self-checks use[/], as inpgrep -f "[/]keepalive.sh".)- PIDs get reused after a reboot. A pidfile alone can lie: after a reset, an unrelated process may hold the recorded PID. When validating a pidfile, also compare
/proc/<pid>/cmdlineagainst the expected program name before trusting it — and before killing it. - Never
pkill -f <supervisor-name>to stop supervisors. The pattern matches your own management shell when its command line contains the name (e.g. you launched the restore from a shell whose command includes it), killing your own session mid-restore. Stop via the pidfile: read the PID, verify/proc/<pid>/cmdline, thenkillexactly that PID. (Same self-match hazard as thepgreppitfall above — the bracket trick works forpkilltoo.) dpkg -ican still ask questions. Reinstalling a package whose config file you modified (e.g.sshd_config) makesucfprompt interactively about which version to keep — under a non-interactive restore this hangs forever (observed: stuck 7+ min inopenssh-server.postinst). Always run restore installs asDEBIAN_FRONTEND=noninteractive dpkg --force-confdef --force-confold -i ....- After a platform reboot,
apt/dpkgmay be locked by the platform's own reconciliation for several minutes. A restore script that fails on the lock turns one outage into two. Loop-wait on the lock (e.g. up to ~10 min, checking every 30s) before giving up. - Supervisors should also watch
sshd. If the tunnel supervisor noticessshdgone but thesshdbinary still exists, recreate/run/sshdif needed and restart sshd directly — a seconds-level fix, no full restore needed. If the binary is gone too (ephemeral rootfs, see below), reinstall from the offline deb cache first. - On ephemeral machines, the
sshdbinary itself is gone after a reboot.openssh-serveris apt-installed, so on overlay / reset-on-boot root filesystems/usr/sbin/sshdvanishes along with everything else. A watchdog sshd-repair that only recreates/run/sshdand host keys, then runs[ -x /usr/sbin/sshd ] && /usr/sbin/sshd, silently does nothing — the-xguard skips the start and the check fails again, forcing a full restore every single time. Fix it at the root: in the watchdog's sshd repair, if the binary is missing, reinstall from the offline deb cache first (DEBIAN_FRONTEND=noninteractive dpkg --force-confdef --force-confold -i <cachedir>/openssh-*.deb, no network needed), then recreate/run/sshd, regenerate host keys (ssh-keygen -A), alignsshd_config, start, and re-check in a short retry loop (e.g. 3 × 2s) before declaring failure — a single check immediately after boot is racy under load. - A live tunnel process does not mean a live tunnel. A wedged
ssh(dead TCP, live process) passespgrep -f "[s]sh.*-R <REMOTE_PORT>"— pgrep only proves the process exists. Tunnel health checks must also confirm a real socket: iterate the candidate pids and requiress -tnpto show an ESTABLISHED socket owned by that pid. - Never silence repair diagnostics. Redirecting every repair step to
/dev/nullturns "still failed" into a mystery — one sshd outage was only diagnosable viadpkg.logbecausesshd's stderr had been discarded. Append repair output to the watchdog log instead; a quiet success costs nothing, a silent failure costs hours.
Verify each layer separately: kill the tunnel ssh (layer 1), reboot the machine (layer 2), and for ephemeral setups simulate a full reset and confirm the watchdog restores service within one poll interval (layer 3).
Verification
From a third machine (the operator's laptop):
ssh -p <REMOTE_PORT> -i access-key.pem <INNER_USER>@<VPS_IP> 'hostname; whoami'
On the VPS, confirm the public listener:
ss -tlnp | grep <REMOTE_PORT> # expect 0.0.0.0:<REMOTE_PORT>
On the inner machine, confirm exactly one tunnel process with a live socket (a wedged tunnel passes a process check but forwards nothing):
pgrep -f "[s]sh.*-R <REMOTE_PORT>:localhost:22" | wc -l # expect 1 (bracket trick: plain "ssh" would also match the checker itself)
for pid in $(pgrep -f "[s]sh.*-R <REMOTE_PORT>" 2>/dev/null); do
ss -tnp 2>/dev/null | grep -q "pid=$pid" && echo "tunnel alive (pid $pid)"
done
Then kill the tunnel ssh once and confirm it reconnects within ~10s and the endpoint works again.
Finally, reboot the inner machine (or restart the boot unit) and confirm the tunnel and endpoint recover without manual intervention — this is the test that catches missing boot persistence.
## Safety Boundaries
- `GatewayPorts yes` is global on the VPS: any user with ssh access can bind public ports. Prefer `GatewayPorts clientspecified` with an explicit `-R 0.0.0.0:<REMOTE_PORT>:...`, or restrict the port with firewall source-IP rules.
- The forwarded port is public: anyone on the internet can attempt SSH. Key-only auth (`PasswordAuthentication no`) is mandatory, not optional. Consider restricting source IPs at the VPS firewall.
- `StrictHostKeyChecking=no` with `UserKnownHostsFile=/dev/null` (used when the VPS is frequently reimaged) disables MITM protection for the tunnel leg. Accept it only for the tunnel leg, never for the operator access leg.
- Never store either private key in a repo, doc, or chat. Reference paths only.
- If the VPS is reimaged, the tunnel user's `authorized_keys` must be restored before the tunnel can reconnect — keep the tunnel public key somewhere durable.
## Failure Modes
| Symptom | Likely cause | Fix |
| --- | --- | --- |
| `ssh -p <PORT>` times out from outside | VPS firewall blocks the port, or `GatewayPorts` is not `yes` (bound to 127.0.0.1) | open the firewall; set `GatewayPorts yes` and reload sshd |
| Tunnel connects but forwards nowhere | port already bound on the VPS; without `ExitOnForwardFailure` ssh stays up silently | add `ExitOnForwardFailure=yes`; pick a free port |
| Tunnel hangs after a network blip, never recovers | no liveness probing on a half-open TCP connection | `ServerAliveInterval` / `ServerAliveCountMax` plus the supervisor loop |
| Two tunnel processes fight over the port | duplicate keepalive instances | atomic `mkdir` single-instance guard; kill-and-restart for takeover |
| Tunnel never comes back after a reboot | no boot autostart; `nohup &` does not survive reboots | systemd unit or `@reboot` cron (see Boot Persistence); verify with a real reboot |
| Watchdog never fires even though the tunnel is dead | `pgrep -f "ssh.*-R <PORT>"` matches the checker's own command line | use the bracket trick: `pgrep -f "[s]sh.*-R <PORT>"` |
| Restore fails right after a reboot with apt/dpkg lock errors | platform reconciliation holds the package lock for minutes after boot | loop-wait on the lock (up to ~10 min) instead of failing immediately |
| Restore kills your own SSH session mid-run | `pkill -f <supervisor-name>` matched your management shell's command line | stop supervisors via pidfile + `/proc/<pid>/cmdline` verification, never broad `pkill -f` |
| `Permission denied (publickey)` on the tunnel leg | tunnel public key missing from the VPS user's `authorized_keys` | reinstall `tunnel-key.pub` |
| `Permission denied (publickey)` on the access leg | access public key missing from the inner user's `authorized_keys` | reinstall `access-key.pub` |
| Watchdog sshd repair does nothing after a reboot; always escalates to full restore | `openssh-server` binary wiped with the ephemeral rootfs; `[ -x /usr/sbin/sshd ] &&` guard silently skips the start | reinstall openssh from the offline deb cache inside the repair path, before recreating `/run/sshd` and host keys |
| Tunnel checks green but the forwarded port is dead | wedged tunnel: dead TCP with a live ssh process fools `pgrep` | verify an ESTABLISHED socket per pid with `ss -tnp`, not just the process |
Files (awesome-skills)
-
agents
-
openai.yaml 212 B
Loading…
-
-
scripts
-
health-watch-loop.sh 1.9 KB
Loading… -
reverse-ssh-boot.service 613 B · in bundle
-
reverse-ssh-keepalive.sh 13.5 KB
Loading…
-
-
tests
-
test_skill_package.py 20.2 KB
Loading…
-
-
SKILL.md 24.6 KB
--- name: muse-reverse-ssh description: Use when exposing a machine without a public IP (cloud VM, container, home server) as a publicly reachable SSH server via a reverse SSH tunnel through a VPS; setting up a persistent keepalive supervisor for the tunnel; or diagnosing dropped tunnels, GatewayPorts binding failures, and forwarded-port access problems. --- # Muse Reverse SSH ## Core Principle The inner machine dials out; the VPS only forwards. A reverse tunnel (`ssh -R <REMOTE_PORT>:localhost:22`) makes the VPS listen on a public port and forward everything to the inner machine's sshd. The inner network needs no inbound firewall rules and no public IP. Treat the tunnel itself as the server: if the tunnel process dies, the "server" is unreachable even though both machines are healthy. Monitor and keepalive the tunnel, not just the machines. ## Decision Tree ```text Inner machine already has a public IP, or you control inbound port forwarding on its router? -> Run sshd directly with firewall rules. Do not add a reverse tunnel. You need HTTPS/browser access to inner services, not just SSH? -> Prefer Cloudflare Tunnel or an nginx reverse proxy on the VPS. Only your own devices need access (no public exposure)? -> Prefer Tailscale or WireGuard over a publicly forwarded port. Inner machine cannot accept inbound connections, but must be SSH-reachable from the internet? -> Use this skill. ``` ## Roles and Keypairs Two machines, two keypairs. Never reuse one keypair for both directions: | Keypair | Private key lives on | Public key goes to | Purpose | | --- | --- | --- | --- | | Tunnel | inner machine (`~/.ssh/tunnel-key`) | VPS user's `~/.ssh/authorized_keys` | inner machine authenticates TO the VPS to establish the tunnel | | Access | operator's laptop (`access-key.pem`) | inner machine user's `~/.ssh/authorized_keys` | operator authenticates THROUGH the forwarded port to the inner machine | Generate both with `ssh-keygen -t ed25519 -f <name> -N ""` and keep private keys at mode `600`. ## Preflight Run before changing anything; keep the output for the final report. On the VPS (root or sudo): ```bash ss -tlnp | grep <REMOTE_PORT> || echo "port <REMOTE_PORT> free" grep -E "^GatewayPorts" /etc/ssh/sshd_config || echo "GatewayPorts not set" ``` On the inner machine: ```bash systemctl is-active sshd || service ssh status ss -tlnp | grep ':22 ' || echo "nothing listening on 22" id <INNER_USER> 2>/dev/null || echo "user <INNER_USER> missing" ``` Interpretation: - Port already listening on the VPS: pick another `<REMOTE_PORT>` or stop the occupying process. - `GatewayPorts` not set (or `no`): remote forwards bind to `127.0.0.1` only — the port will NOT be publicly reachable until fixed (see VPS Setup). - Nothing on inner port 22: install and start openssh-server first. ## VPS Setup (one time) 1. Create the tunnel user (skip if it exists): ```bash id <VPS_USER> 2>/dev/null || useradd -m -s /bin/bash <VPS_USER> mkdir -p /home/<VPS_USER>/.ssh && chmod 700 /home/<VPS_USER>/.ssh ``` 2. Install the tunnel public key: ```bash cat tunnel-key.pub >> /home/<VPS_USER>/.ssh/authorized_keys chmod 600 /home/<VPS_USER>/.ssh/authorized_keys chown -R <VPS_USER>:<VPS_USER> /home/<VPS_USER>/.ssh ``` 3. Allow public binding of forwarded ports. In `/etc/ssh/sshd_config` set: ```text GatewayPorts yes ``` then `systemctl reload sshd`. (Safer alternative: `GatewayPorts clientspecified` — see Safety Boundaries.) 4. Open `<REMOTE_PORT>`/tcp in the VPS firewall (ufw / cloud security group). ## Inner Machine Setup (one time) 1. Install and start openssh-server: ```bash sudo apt-get install -y openssh-server sudo systemctl enable --now ssh ``` 2. Create the login user and install the access public key: ```bash id <INNER_USER> 2>/dev/null || sudo useradd -m -s /bin/bash <INNER_USER> sudo -u <INNER_USER> mkdir -p /home/<INNER_USER>/.ssh cat access-key.pub | sudo tee -a /home/<INNER_USER>/.ssh/authorized_keys >/dev/null sudo chmod 700 /home/<INNER_USER>/.ssh sudo chmod 600 /home/<INNER_USER>/.ssh/authorized_keys sudo chown -R <INNER_USER>:<INNER_USER> /home/<INNER_USER>/.ssh ``` 3. Harden sshd (`/etc/ssh/sshd_config`): ```text PermitRootLogin prohibit-password PasswordAuthentication no ``` then `sudo systemctl reload ssh`. 4. Place the tunnel private key: ```bash install -m 600 tunnel-key ~/.ssh/tunnel-key ``` ## Keepalive The tunnel must survive network drops, not just machine reboots. Run a user-level supervisor (template: `scripts/reverse-ssh-keepalive.sh`): ```bash nohup ./reverse-ssh-keepalive.sh >/dev/null 2>&1 & ``` Key options and why they matter: - `-N`: no remote command, forward only. - `-R <REMOTE_PORT>:localhost:22`: the reverse forward itself. - `-o ServerAliveInterval=20 -o ServerAliveCountMax=3`: detect a dead TCP connection within ~60s instead of hanging forever on a half-open socket. - `-o ExitOnForwardFailure=yes`: fail fast if the VPS port cannot be bound (e.g. already taken) instead of sitting on a tunnel that forwards nowhere. - `-o BatchMode=yes -o ConnectTimeout=20`: never prompt for input, never hang on connect. - Atomic `mkdir` lockdir for single instance (see the template script). Do **not** use `flock -n` on a lock file here: the lock fd is inherited by the foreground `ssh` child, so killing the supervisor leaves an orphaned ssh holding the lock forever and no new supervisor instance can ever start (observed in production). A directory cannot be inherited — with PID/cmdline stale-lock reclaim (the claim is verified via `/proc/<pid>/cmdline`; reclaim takes the stale lock aside with an atomic rename and re-verifies it *after* the rename, moving it back if it turned out to be alive — so two contenders can never both win. If the move-back itself loses to a fresh claimant, the displaced live owner is a ghost (alive but directory-less, possibly blocked in foreground ssh and never reaching its ownership check) and is terminated after a cmdline re-verify; its orphaned ssh child is reaped by the new holder's stale-tunnel cleanup), takeover is always kill-and-restart via the pidfile (`$STATE_DIR/keepalive.pid`, verify `/proc/<pid>/cmdline` before killing). - Write the operator-facing endpoint to a file (e.g. `current-endpoint.txt`) so the access command stays discoverable: ```text ssh -p <REMOTE_PORT> -i access-key.pem <INNER_USER>@<VPS_IP> ``` ## Boot Persistence The keepalive supervisor only survives network drops. A machine reboot kills it silently — `nohup ... &` does not persist across boots, and without autostart the "server" stays down after every reboot with no alert. This is the most common cause of a tunnel that "just stops working" days later. Protect three layers independently: ### Layer 1 — process supervision (ssh dies, machine stays up) Covered by `scripts/reverse-ssh-keepalive.sh` (atomic-`mkdir`-lock-guarded restart loop). Alternatively, `autossh` is a drop-in replacement: ```bash autossh -M 0 -N -o "ServerAliveInterval 30" -o "ServerAliveCountMax 3" \ -o ExitOnForwardFailure=yes -i tunnel-key.pem \ -R 0.0.0.0:<REMOTE_PORT>:localhost:22 <TUNNEL_USER>@<VPS_IP> ``` `-M 0` disables autossh's legacy monitor port and relies on SSH keepalives. Either way, something must (re)start the supervisor itself after a reboot — that is layer 2. ### Layer 2 — boot autostart (machine reboots, disk persists) Pick the mechanism the platform actually persists (template: `scripts/reverse-ssh-boot.service`): | Platform | Mechanism | |---|---| | systemd (most Linux servers / VPS) | unit file + `systemctl enable`, `Restart=always` | | Linux without systemd | `@reboot` cron entry, or executable `/etc/rc.local` | | macOS | `launchd` plist in `~/Library/LaunchAgents` with `RunAtLoad` | | Windows | Task Scheduler task with an "At startup" trigger | Pitfalls: - `HOME` must resolve to the home holding the script, keys, and state dir. A unit running as root gets `HOME=/root` by default, which silently breaks every `$HOME`-relative path. Set `Environment=HOME=...` explicitly. - The keepalive's atomic `mkdir` single-instance lock makes boot restarts safe: a directory lock cannot be inherited by orphaned children, and stale locks (dead PID, or a PID whose cmdline is no longer the keepalive script) are reclaimed by taking the stale lockdir aside with an atomic rename and re-verifying it *after* the rename (moved back if it was actually alive) — so a fresh instance after an unclean shutdown always takes over cleanly instead of exiting as "another instance running". There is deliberately no "older than N seconds" rule for the keepalive: a healthy supervisor runs for weeks, so its timestamp is always old — an age rule would let any second instance steal the lock from a healthy first one and leave two supervisors fighting over the VPS port. (The restore script in Layer 3 is different: restores are time-bounded operations, so an age cap there is valid.) ### Layer 3 — ephemeral machines (disk resets on reboot) Containers, reset-on-boot VMs, and spot instances wipe the root filesystem on reboot: no unit file, no cron entry, no installed package survives — only a designated data disk persists, if any. Layer 2 cannot work there because there is nothing durable left to trigger it. Use instead: - **Idempotent restore script on the persistent disk.** One script that reinstalls packages, recreates users and keys, and restarts the keepalive — and is safe to run repeatedly (`id <user> || useradd ...`, `apt-get install -y` is idempotent, kill-and-restart supervisors rather than start-if-absent). - **Offline package cache for the restore path.** On reset-prone machines, `apt-get update/install` is the slowest and flakiest part of a restore (network, mirror issues, the platform's own apt reconciliation holding the lock). Pre-download the needed `.deb`s plus their dependency closure onto the persistent disk (`apt-cache depends --recurse --no-recommends ... <pkg> | apt-get download`), refresh on a schedule; the restore script installs with `dpkg -i <cachedir>/*.deb` first and only falls back to apt when the cache is missing or incomplete. Measured on a real reset-prone VM: ~7s offline vs 2–4 min via apt — recovery is then dominated by detection latency alone. - **External watchdog, two-tier (resident loop + supervisor).** When the machine itself can run shell, do not make an external poller do the checking: put the minute-level checks in a **zero-cost resident loop on the machine** (`health-watch-loop.sh`: every ~60–90s it runs the check script, which writes a JSON status file and self-heals), and demote the external watcher to a **sparse supervisor** whose only job is to verify the resident loop *process* lives and read its status file. This split matters most when the supervisor is an LLM/agent-driven scheduled task: every poll costs real tokens, and a 1-minute agent poll on an idle machine burned ~7% of a weekly free quota in production before this redesign. The resident loop does the same checks for free; a 5-minute supervisor cut the agent-side usage to ~1/5 of the 1-minute era (≈3–6M input tokens/day measured, vs ≈26M/day). Detection latency for a dead loop equals the supervisor interval — state that honestly instead of promising instant recovery. Five minutes is a calm default; production later tightened to 2.5 minutes after a churny platform kept killing the loop several times a night, accepting roughly double the supervisor cost (≈11M input/day) to halve the worst-case window to ≤2.5 min. Service-level faults are still caught by the loop itself in ~1 min regardless. Polling the supervisor faster barely shortens recovery while multiplying agent cost. - Supervisor liveness check is one fixed command — pidfile **and** `/proc/<pid>/cmdline` match (never `kill -0` alone; PIDs are reused after reboots and a stale pidfile can point at an innocent process): ```bash pid=$(cat <LOOP_PIDFILE> 2>/dev/null); if [ -n "$pid" ] && ps -o cmd= -p "$pid" 2>/dev/null | grep -q health-watch-loop.sh; then echo ALIVE; else echo DEAD; fi ``` - If DEAD, relaunch fully detached (`setsid nohup bash <LOOP> >/dev/null 2>&1 < /dev/null &`), sleep 2s, re-verify with the same check. Then **do not run the check script yourself** — the loop runs a check immediately on start, and a second concurrent run only collides with the check script's `mkdir` lock and exits as a no-op. Instead, wait (up to ~2 min) until the status file's timestamp refreshes; that refresh *is* the proof the relaunched loop covered this interval. - Read the status with one fixed command, never an improvised parser (an agent that re-derives JSON parsing each run wastes tokens and intermittently writes buggy one-offs that throw and retry): ```bash python3 -c "import json;d=json.load(open('<STATUS_FILE>'));print(d['overall'],d['time'],d.get('restored'));[print(k,v['status']) for k,v in d['checks'].items() if v['status']!='ok']" ``` - Reporting discipline: healthy means **silent** — the supervisor reports only on degraded/failed status, on its own execution failure, or when the loop resists relaunch. A watchdog that posts "all green" every poll trains the operator to ignore it and, if agent-driven, burns budget to say nothing. Keep the supervisor's own instructions short for the same reason: they are re-read on every poll. - Logging discipline: append to the ops log only on anomalies (loop relaunched, any non-`ok` check, degraded). An all-green run writes nothing; a log that grows on healthy runs stops being read. - The loop's check script should check more than the tunnel: the tunnel ssh process (`pgrep -f "[s]sh.*-R <REMOTE_PORT>"` plus an ESTABLISHED-socket verification — see the pitfalls), `sshd`, every keepalive supervisor, any other long-lived agents (with a log-freshness check for agents that can hang silently: restart if the log has not grown in N minutes), and any local origins/proxies the tunnel exists to serve. Each check gets a targeted repair first; only items that resist repair escalate to the full restore script. Template loop: `scripts/health-watch-loop.sh`. - **VPS-side detection.** The VPS is usually a normal persistent machine. A tiny check there — `ss -tlnp | grep <REMOTE_PORT>` or a TCP connect attempt against the forwarded port — notices the tunnel disappearing before any human does, and is the cheapest layer-3 signal. ### Watchdog and restore pitfalls (learned the hard way) - **Never guard the restore script with an outer `flock -n <restore.lock>`.** This was the old advice in this very skill — and it silently disabled a production watchdog for ~2.5 hours before anyone noticed. The flock file descriptor is inherited by every long-lived daemon the restore script starts (keepalive, agent, ssh), so the advisory lock is held forever by those background processes and never released; every later watchdog run then believes "a restore is already in progress" and quietly skips recovery. The watchdog keeps polling, logging healthy runs, but can never recover anything again. Fix: put the mutual exclusion *inside* the restore script as an atomic `mkdir` lockdir — a directory cannot be inherited by children, so it can only be held by the script itself — with PID/timestamp stale-lock reclaim (see the Layer 3 bullet above). The keepalive supervisor needs the same treatment: use the atomic `mkdir` lockdir there too. `flock -n 9` plus `9>&-` looks correct on paper, but in production the foreground `ssh` child inherited the lock fd anyway; after the supervisor was killed, the orphaned ssh held the lock forever and permanently disabled recovery. - **Never `set -e` in the resident loop.** The check script exits 1 exactly when it finds degraded state — that is its job. Under `set -e`, the first degraded run kills the watchdog loop itself, silently ending all checks until the supervisor happens by. Wrap each run in `timeout 300` as well: a check that hangs would otherwise serialize the loop forever (one stuck `curl` = no more checks, ever). Discard the check's stdout (the status JSON is the source of truth) and trim its stderr file to the last ~200 lines each cycle so logs cannot grow unboundedly. - **After a restore pass, the re-verify must clear first-pass failures.** A common aggregation bug: the verify pass skips or keeps the first pass's `failed` verdict for an item (`[ "$ST" != failed ] && ST=ok`-style guards), so an item the restore actually fixed still reports failed → the run ends `degraded` with exit 1 even though everything is healthy again. When the post-restore verify passes, the item's final state is `fixed` ("recovered after restore"), never the stale first-pass failure. - **`pgrep -f` matches the checker itself.** `pgrep -f "ssh.*-R <REMOTE_PORT>"` also matches the very command running the check, because the pattern text appears in the checker's own command line — a dead tunnel then looks healthy forever. Use the bracket trick: `pgrep -f "[s]sh.*-R <REMOTE_PORT>"`. The regex `[s]sh` matches `ssh` in the target but not the literal `[s]sh` in your own command line. (Same reason supervisor self-checks use `[/]`, as in `pgrep -f "[/]keepalive.sh"`.) - **PIDs get reused after a reboot.** A pidfile alone can lie: after a reset, an unrelated process may hold the recorded PID. When validating a pidfile, also compare `/proc/<pid>/cmdline` against the expected program name before trusting it — and before killing it. - **Never `pkill -f <supervisor-name>` to stop supervisors.** The pattern matches your own management shell when its command line contains the name (e.g. you launched the restore from a shell whose command includes it), killing your own session mid-restore. Stop via the pidfile: read the PID, verify `/proc/<pid>/cmdline`, then `kill` exactly that PID. (Same self-match hazard as the `pgrep` pitfall above — the bracket trick works for `pkill` too.) - **`dpkg -i` can still ask questions.** Reinstalling a package whose config file you modified (e.g. `sshd_config`) makes `ucf` prompt interactively about which version to keep — under a non-interactive restore this hangs forever (observed: stuck 7+ min in `openssh-server.postinst`). Always run restore installs as `DEBIAN_FRONTEND=noninteractive dpkg --force-confdef --force-confold -i ...`. - **After a platform reboot, `apt`/`dpkg` may be locked** by the platform's own reconciliation for several minutes. A restore script that fails on the lock turns one outage into two. Loop-wait on the lock (e.g. up to ~10 min, checking every 30s) before giving up. - **Supervisors should also watch `sshd`.** If the tunnel supervisor notices `sshd` gone but the `sshd` binary still exists, recreate `/run/sshd` if needed and restart sshd directly — a seconds-level fix, no full restore needed. If the binary is gone too (ephemeral rootfs, see below), reinstall from the offline deb cache first. - **On ephemeral machines, the `sshd` binary itself is gone after a reboot.** `openssh-server` is apt-installed, so on overlay / reset-on-boot root filesystems `/usr/sbin/sshd` vanishes along with everything else. A watchdog sshd-repair that only recreates `/run/sshd` and host keys, then runs `[ -x /usr/sbin/sshd ] && /usr/sbin/sshd`, silently does nothing — the `-x` guard skips the start and the check fails again, forcing a full restore every single time. Fix it at the root: in the watchdog's sshd repair, if the binary is missing, reinstall from the offline deb cache first (`DEBIAN_FRONTEND=noninteractive dpkg --force-confdef --force-confold -i <cachedir>/openssh-*.deb`, no network needed), then recreate `/run/sshd`, regenerate host keys (`ssh-keygen -A`), align `sshd_config`, start, and re-check in a short retry loop (e.g. 3 × 2s) before declaring failure — a single check immediately after boot is racy under load. - **A live tunnel process does not mean a live tunnel.** A wedged `ssh` (dead TCP, live process) passes `pgrep -f "[s]sh.*-R <REMOTE_PORT>"` — pgrep only proves the process exists. Tunnel health checks must also confirm a real socket: iterate the candidate pids and require `ss -tnp` to show an ESTABLISHED socket owned by that pid. - **Never silence repair diagnostics.** Redirecting every repair step to `/dev/null` turns "still failed" into a mystery — one sshd outage was only diagnosable via `dpkg.log` because `sshd`'s stderr had been discarded. Append repair output to the watchdog log instead; a quiet success costs nothing, a silent failure costs hours. Verify each layer separately: kill the tunnel ssh (layer 1), reboot the machine (layer 2), and for ephemeral setups simulate a full reset and confirm the watchdog restores service within one poll interval (layer 3). ## Verification From a third machine (the operator's laptop): ```bash ssh -p <REMOTE_PORT> -i access-key.pem <INNER_USER>@<VPS_IP> 'hostname; whoami' ``` On the VPS, confirm the public listener: ```bash ss -tlnp | grep <REMOTE_PORT> # expect 0.0.0.0:<REMOTE_PORT> ``` On the inner machine, confirm exactly one tunnel process *with a live socket* (a wedged tunnel passes a process check but forwards nothing): ```bash pgrep -f "[s]sh.*-R <REMOTE_PORT>:localhost:22" | wc -l # expect 1 (bracket trick: plain "ssh" would also match the checker itself) for pid in $(pgrep -f "[s]sh.*-R <REMOTE_PORT>" 2>/dev/null); do ss -tnp 2>/dev/null | grep -q "pid=$pid" && echo "tunnel alive (pid $pid)" done Then kill the tunnel ssh once and confirm it reconnects within ~10s and the endpoint works again. Finally, reboot the inner machine (or restart the boot unit) and confirm the tunnel and endpoint recover without manual intervention — this is the test that catches missing boot persistence. ## Safety Boundaries - `GatewayPorts yes` is global on the VPS: any user with ssh access can bind public ports. Prefer `GatewayPorts clientspecified` with an explicit `-R 0.0.0.0:<REMOTE_PORT>:...`, or restrict the port with firewall source-IP rules. - The forwarded port is public: anyone on the internet can attempt SSH. Key-only auth (`PasswordAuthentication no`) is mandatory, not optional. Consider restricting source IPs at the VPS firewall. - `StrictHostKeyChecking=no` with `UserKnownHostsFile=/dev/null` (used when the VPS is frequently reimaged) disables MITM protection for the tunnel leg. Accept it only for the tunnel leg, never for the operator access leg. - Never store either private key in a repo, doc, or chat. Reference paths only. - If the VPS is reimaged, the tunnel user's `authorized_keys` must be restored before the tunnel can reconnect — keep the tunnel public key somewhere durable. ## Failure Modes | Symptom | Likely cause | Fix | | --- | --- | --- | | `ssh -p <PORT>` times out from outside | VPS firewall blocks the port, or `GatewayPorts` is not `yes` (bound to 127.0.0.1) | open the firewall; set `GatewayPorts yes` and reload sshd | | Tunnel connects but forwards nowhere | port already bound on the VPS; without `ExitOnForwardFailure` ssh stays up silently | add `ExitOnForwardFailure=yes`; pick a free port | | Tunnel hangs after a network blip, never recovers | no liveness probing on a half-open TCP connection | `ServerAliveInterval` / `ServerAliveCountMax` plus the supervisor loop | | Two tunnel processes fight over the port | duplicate keepalive instances | atomic `mkdir` single-instance guard; kill-and-restart for takeover | | Tunnel never comes back after a reboot | no boot autostart; `nohup &` does not survive reboots | systemd unit or `@reboot` cron (see Boot Persistence); verify with a real reboot | | Watchdog never fires even though the tunnel is dead | `pgrep -f "ssh.*-R <PORT>"` matches the checker's own command line | use the bracket trick: `pgrep -f "[s]sh.*-R <PORT>"` | | Restore fails right after a reboot with apt/dpkg lock errors | platform reconciliation holds the package lock for minutes after boot | loop-wait on the lock (up to ~10 min) instead of failing immediately | | Restore kills your own SSH session mid-run | `pkill -f <supervisor-name>` matched your management shell's command line | stop supervisors via pidfile + `/proc/<pid>/cmdline` verification, never broad `pkill -f` | | `Permission denied (publickey)` on the tunnel leg | tunnel public key missing from the VPS user's `authorized_keys` | reinstall `tunnel-key.pub` | | `Permission denied (publickey)` on the access leg | access public key missing from the inner user's `authorized_keys` | reinstall `access-key.pub` | | Watchdog sshd repair does nothing after a reboot; always escalates to full restore | `openssh-server` binary wiped with the ephemeral rootfs; `[ -x /usr/sbin/sshd ] &&` guard silently skips the start | reinstall openssh from the offline deb cache inside the repair path, before recreating `/run/sshd` and host keys | | Tunnel checks green but the forwarded port is dead | wedged tunnel: dead TCP with a live ssh process fools `pgrep` | verify an ESTABLISHED socket per pid with `ss -tnp`, not just the process |
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.