156 lines
11 KiB
Markdown
156 lines
11 KiB
Markdown
# How etrelay works
|
||
|
||
> Diagrams: PlantUML sources in [`diagrams/`](diagrams), PNGs regenerated with `make diagrams`.
|
||
|
||
## The problem
|
||
|
||
A server shows up in the in-game browser only if a **master server** lists it. Registration is a 3-step handshake over UDP:
|
||
|
||
1. The game server sends `heartbeat EnemyTerritory-1` to the master.
|
||
2. The master does not trust it: it queries the **source IP:port of the heartbeat**. `etmaster.net` sends `getChallenge`, then `getInfo` (classic Quake 3 masters send `getinfo <challenge>` directly).
|
||
3. The game server answers (`challengeResponse`, `infoResponse`). The master lists the **source IP:port of those answers**.
|
||
|
||
Our ET server sits at home, reachable from the Internet only through the front server and the WireGuard tunnel. Players went in through the front (DNAT, then nginx `stream`), but the ET server's own heartbeats went **out through the home box**: the master challenged the home IP, where nothing listens for it. Result: server reachable by IP, invisible in the browser.
|
||
|
||
|
||
|
||
## The fix
|
||
|
||
`etrelay` runs on the front and owns the public UDP ports, one per ET server (27960 for the first). For each server it does two things on **the same socket**:
|
||
|
||
- **Proxy**: forwards every client (players, server browsers, masters) to the ET server, one upstream socket per client address. Packets a client has no business sending (`rcon`...) are dropped on the way.
|
||
- **Heartbeat**: sends the heartbeats itself, so the master challenges the **front** address. That challenge is just another client packet: it goes through the proxy, the ET server answers it, the answer leaves from the front address, and the master lists the front.
|
||
|
||
The ET server's own heartbeats are disabled (`sv_master1..5 ""`).
|
||
|
||
The ET server is its own WireGuard peer of the front, with no proxy in between (the DNAT rule and the home nginx `stream` are gone). Each client reaches it from its own port of the front's tunnel address: the **control API** maps that port back to the real address, and bans addresses (see [Control API and bans](#control-api-and-bans)).
|
||
|
||
Masters are independent: a server is only listed on the masters it heartbeats, and each client browses its own master (ET 2.60b: `etmaster.idsoftware.com`, ET: Legacy: `etmaster.net`). The relay heartbeats both by default; diagrams show `etmaster.net`, the id Software master works the same way.
|
||
|
||
|
||
|
||
## Master registration sequence
|
||
|
||
|
||
|
||
What it looks like in production (`--debug`):
|
||
|
||
```
|
||
heartbeat EnemyTerritory-1 -> etmaster.net:27950 (198.51.100.10:27950)
|
||
session + 198.51.100.10:27950 (2 active)
|
||
198.51.100.10:27950 -> ET getChallenge (27 B)
|
||
198.51.100.10:27950 <- ET challengeResponse (32 B)
|
||
198.51.100.10:27950 -> ET getInfo (22 B)
|
||
198.51.100.10:27950 <- ET infoResponse (303 B)
|
||
session + 203.0.113.7:45488 (3 active)
|
||
203.0.113.7:45488 -> ET getstatus (14 B)
|
||
203.0.113.7:45488 <- ET statusResponse (993 B)
|
||
```
|
||
|
||
The last two lines are a server tracker that got the address from the master.
|
||
|
||
## Player sequence
|
||
|
||
|
||
|
||
The ET server sees every client as `<front-wg>:<port>`: one source IP (the front's tunnel address), one port per session, the port of the session's upstream socket. Its own IP bans can't tell players apart: bans are etrelay's (below). A player coming back after the session expired gets a new port.
|
||
|
||
## Several servers: registering
|
||
|
||
The relay holds several ET servers. The ET protocol has no host name to tell them apart: each one gets its own public port (`--ports`), with its own socket, sessions, heartbeats and bans. Masters list servers by address and port, so several on the front's address are fine.
|
||
|
||
A server is relayed once its admin tool registered it, with `PUT /server` on the control API (`--control <addr>`, on the front's tunnel address, unreachable from the internet):
|
||
|
||
|
||
|
||
- **The relay's token** only lets a tool register: it keeps strangers of the tunnel off the public ports.
|
||
- **The key** is picked by the tool, and is what every later call carries: it names the server, so a tool only reaches its own. etrelay keeps its SHA-256 (`servers.json`), never the key.
|
||
- **The proof**: before taking a key, etrelay checks the tool runs the ET server it registers. The tool makes the server show the key's fingerprint (the first 128 bits of its SHA-256) in the serverinfo cvar `etrelay_proof`, etrelay reads it there with a `getstatus`. Only who holds that server's console can do it, wherever the request comes from (a NAT between the two changes nothing), and the fingerprint, public for a moment, doesn't give the key away. An empty cvar is left out of serverinfo: the tool sets it to `""` afterwards (`unset` isn't in 2.60b).
|
||
|
||
What can't happen:
|
||
|
||
- **two servers on one port**: the second one gets a 409, the first one is untouched;
|
||
- **another tool on a registered server**: it has neither the key, nor the server's console to prove another one.
|
||
|
||
One limit: a server that is gone for good without `DELETE /server` keeps its port (the masters drop it, nothing answers there), until its entry is removed from `servers.json` and etrelay restarted.
|
||
|
||
An ET server registered again is taken over by who proves to run it (a tool reinstalled, with a new key): the former key is refused from then on. On the same port the relay goes on (sessions, bans); on another port the former one is freed. A relay that lost its state answers 401 to every key: tools register again by themselves.
|
||
|
||
The registered servers are relayed again after a restart. `DELETE /server` leaves the relay: the masters get a flatline, the port is freed, the bans dropped.
|
||
|
||
`etrelay <UPSTREAM>` still relays one server without any of this, on `--listen`: it has no key, so no control API over it.
|
||
|
||
## Control API and bans
|
||
|
||
Once registered, the admin tool asks about its server, with its key:
|
||
|
||
- `GET /server`: the registration, and whether the ET server answered the last probe.
|
||
- `GET /sessions`: the open sessions, real client address and port the ET server sees, so the tool can put a real address on each player.
|
||
- `GET /bans`, `PUT /bans/{ip}`, `DELETE /bans/{ip}`: bans by source IP, with a reason and an optional duration.
|
||
|
||
|
||
|
||
A ban cuts the address's sessions at once, and the relay loop drops its packets before any session lookup: the ET server gets nothing more from the player and times them out, and they can't come back. Each server has its own bans, saved to `<state-dir>/bans-<port>.json` on each change (written next to it, then renamed), expired ones forgotten.
|
||
|
||
## Filter
|
||
|
||
Only what clients and masters send a server gets through, checked on the first word as the ET server tokenizes it:
|
||
|
||
- in-game packets (no `\xff\xff\xff\xff` prefix): always;
|
||
- `getstatus`, `getinfo`, `getchallenge`, `connect`, PunkBuster's `PB_...`: allowed;
|
||
- anything else, `rcon` above all, dropped: it travels in clear, and a stream of it blocks the local admin's own rcon (2.60b runs one rcon per 500 ms, whoever sends it).
|
||
|
||
`getstatus` and `getinfo` are limited per source IP (token bucket: a burst of 5, then 1/s): their answers are much bigger than the query, and from a spoofed source they would make the server a DDoS reflector. The limiter remembers at most 65 536 IPs, idle ones forgotten after 60 s.
|
||
|
||
## Internals
|
||
|
||
Single-threaded tokio runtime, four kinds of tasks. Each server has its relay loop, its heartbeat task and its state (sessions, bans), shared with the control API behind a mutex. The registry (the servers by public port, the hash of their key) is behind another one, which the packets never take. Neither is held across an `await`:
|
||
|
||
|
||
|
||
- **relay loop** (one per server): reads the server's public socket, runs the filter and the ban check, finds the client's session and pushes the packet into its queue. No session, or an expired one: opens a new one (and purges expired entries), unless `MAX_SESSIONS` are open, all servers together. A full queue drops the packet, like any congested UDP hop.
|
||
- **session task**: owns one ephemeral socket connected to the ET server. Forwards the queue to the ET server and the ET server answers back to the client **through the public socket** (so they come from `:27960`). Ends after `SESSION_TIMEOUT` without a packet either way, or at once when a ban drops its queue's sender.
|
||
- **heartbeat task** (one per server): probes the ET server with `getinfo` on its own socket (up to 3 attempts, 3 s each), then heartbeats the masters through the public socket. Masters are resolved on every heartbeat (DNS may change).
|
||
- **control API task** (with `--control`): axum on the tunnel address. Binding is retried every 10 s: the tunnel may come up after etrelay. Registering starts a server's two tasks, leaving aborts them and drops its sessions.
|
||
|
||
The code follows that split: `server` (one relayed server), `registry` (which servers, `servers.json`), `proof` (keys and their proof), `control` (the API), `et` (the probes: `getinfo`, `getstatus`), `filter`, `bans`.
|
||
|
||
### Heartbeat states
|
||
|
||
|
||
|
||
The master drops a server that stops heartbeating after a while anyway; the flatline only makes it faster.
|
||
|
||
### Constants
|
||
|
||
| Constant | Value | Why |
|
||
|---|---|---|
|
||
| `HEARTBEAT_INTERVAL` | 5 min | Same period as the ET server itself |
|
||
| `PROBE_ATTEMPTS` × `PROBE_TIMEOUT` | 3 × 3 s | One lost UDP packet must not flatline the server |
|
||
| `SESSION_TIMEOUT` | 2 min | Idle clients (server browsers) freed quickly; players send packets constantly |
|
||
| `SESSION_QUEUE` | 64 packets | Burst absorption per client before dropping |
|
||
| `MAX_PACKET` | 65 535 B | Any UDP datagram: `statusResponse` easily exceeds nginx's old 512 B buffer |
|
||
| `MAX_SESSIONS` | 2048 | One socket each, all servers together: spoofed sources must not use up the file descriptors (`LimitNOFILE=4096` in the unit) |
|
||
| `QUERY_BURST`, `QUERY_RATE` | 5, then 1/s | `getstatus`/`getinfo` per source IP: a browser refresh or a master's check passes, a reflection attack doesn't |
|
||
| `MAX_QUERY_SOURCES`, `QUERY_FORGET` | 65 536 IPs, 60 s | The limiter's memory stays bounded under a flood of spoofed sources |
|
||
| `MIN_TOKEN`, `MIN_KEY` | 32 characters | Shortest relay token and server key accepted |
|
||
| `PROOF_BYTES` | 16 | Bytes of the key's SHA-256 shown as its fingerprint: serverinfo holds 1024 characters for every cvar |
|
||
|
||
### Protocol cheat sheet
|
||
|
||
Connectionless packets start with `\xff\xff\xff\xff`, then a command:
|
||
|
||
| Command | From → to | Meaning |
|
||
|---|---|---|
|
||
| `heartbeat EnemyTerritory-1` | server → master | "I'm alive, come check" |
|
||
| `heartbeat ETFlatline-1` | server → master | "I'm going down" |
|
||
| `getChallenge` / `challengeResponse` | master ↔ server | Master check before `getInfo` (etmaster.net) |
|
||
| `getinfo` / `getInfo` | master/browser → server | Short info request (optional challenge) |
|
||
| `infoResponse` | server → master/browser | Hostname, map, player count, challenge |
|
||
| `getstatus` / `statusResponse` | browser ↔ server | Full cvars + player list |
|
||
| `getchallenge` / `challengeResponse` / `connect` | player ↔ server | Connection handshake |
|
||
| `PB_...` | PunkBuster ↔ server | PunkBuster's own packets |
|
||
| `rcon <password> <command>` | admin → server | Remote console: dropped by the relay |
|
||
|
||
Packets with no prefix are in-game traffic: forwarded as-is, only counted in `--debug` logs.
|