feat: relay several servers, registered through the control API
This commit is contained in:
rmanach
@@ -1,26 +1,34 @@
|
||||
# etrelay
|
||||
|
||||
UDP relay + master heartbeat for a Wolfenstein: Enemy Territory server reachable only through a tunnel: runs on the front server (public IP) and makes the server show up in the in-game browser.
|
||||
UDP relay + master heartbeat for Wolfenstein: Enemy Territory servers reachable only through a tunnel: runs on the front server (public IP) and makes the servers show up in the in-game browser. Each server gets its own public port.
|
||||
|
||||
```
|
||||
players / masters ──> front :27960 (etrelay) ══wg══> ET <et-server>:27960
|
||||
front <front-wg>:8090 (control API) <══wg══ admin tool
|
||||
players / masters ──> front :27960 (etrelay) ══wg══> ET <et-server-1>:27960
|
||||
front :27961 (etrelay) ══wg══> ET <et-server-2>:27960
|
||||
front <front-wg>:8090 (control API) <══wg══ admin tools
|
||||
```
|
||||
|
||||
A server is relayed because its admin tool (etadm) **registered** it through the control API, or because it was given on the command line (one server, no control API over it).
|
||||
|
||||
- **Proxy**: one upstream socket per client address, packets forwarded as-is both ways, session dropped after 2 min of silence.
|
||||
- **Filter**: connectionless packets pass only for what clients and masters send a server (`getstatus`, `getinfo`, `getchallenge`, `connect`, PunkBuster's `PB_`). `rcon` never reaches the server from the internet: it travels in clear, and a stream of it blocks the local admin tool (2.60b runs one rcon per 500 ms, whoever sends it). `getstatus`/`getinfo` are limited per source IP (burst of 5, then 1/s): their answers are much bigger than the query, a spoofed source would make the server a DDoS reflector. At most 2048 sessions (one socket each).
|
||||
- **Heartbeat**: every 5 min, `getinfo` to the ET server; if it answers, `heartbeat EnemyTerritory-1` to the masters **from the public socket**. When it stops answering, one `heartbeat ETFlatline-1`.
|
||||
- **Heartbeat**: every 5 min, `getinfo` to the ET server; if it answers, `heartbeat EnemyTerritory-1` to the masters **from the server's public socket**. When it stops answering, or when the server leaves the relay, one `heartbeat ETFlatline-1`.
|
||||
|
||||
Masters check (`getChallenge`, `getInfo`) the address the heartbeat came from: the public `:27960`. Those packets go through the relay like any client, the ET server answers them, and the master lists the front public address.
|
||||
Masters list servers by address and port, several on one address are fine. They check (`getChallenge`, `getInfo`) the address the heartbeat came from: the server's public port. Those packets go through the relay like any client, the ET server answers them, and the master lists the front public address.
|
||||
|
||||
How it works in detail, with diagrams: [ENGINE.md](ENGINE.md).
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
etrelay <UPSTREAM> [--listen 0.0.0.0:27960] [--master host:port]... [--control <addr>] [--state-dir /var/lib/etrelay] [--debug]
|
||||
etrelay [UPSTREAM] [--listen 0.0.0.0:27960] [--ports 27960-27969] [--master host:port]... [--control <addr>] [--state-dir /var/lib/etrelay] [--debug]
|
||||
```
|
||||
|
||||
- `--control <addr>`: servers register through the control API, on a port of `--ports` bound on `--listen`'s address. They are kept in `<state-dir>/servers.json` and relayed again after a restart.
|
||||
- `UPSTREAM` (an ET server's `ip:port`): relayed on `--listen` from the start, without registering. Nothing reaches it through the control API: no real addresses, no bans. It can't be registered, nor its port taken.
|
||||
|
||||
One of the two at least. The ports of `--ports` must be open (UDP) on the front's firewall.
|
||||
|
||||
Default masters: `etmaster.idsoftware.com:27950` (id Software, the one ET 2.60b clients query) and `etmaster.net:27950` (ET: Legacy community). Passing `--master` replaces both.
|
||||
|
||||
`--debug` logs sessions (open/close, packet counts), connectionless packets by command name only (`getinfo`, `getstatus`, `connect`...: no arguments, `connect` carries the userinfo), probes and heartbeats. In-game packets are only counted.
|
||||
@@ -29,7 +37,7 @@ Default masters: `etmaster.idsoftware.com:27950` (id Software, the one ET 2.60b
|
||||
probe <wg-peer>:27960: up
|
||||
ET server <wg-peer>:27960 is up
|
||||
heartbeat EnemyTerritory-1 -> etmaster.net:27950 (198.51.100.10:27950)
|
||||
session + 198.51.100.10:27950 as port 51234 (2 active)
|
||||
session + 198.51.100.10:27950 as port 51234 (2 active on port 27960)
|
||||
198.51.100.10:27950 -> ET getChallenge (27 B)
|
||||
198.51.100.10:27950 <- ET challengeResponse (32 B)
|
||||
198.51.100.10:27950 -> ET getInfo (22 B)
|
||||
@@ -43,18 +51,44 @@ Without `--debug`, each player joining is logged with its real address and the p
|
||||
|
||||
## Control API
|
||||
|
||||
`--control <addr>` serves a small HTTP API for the admin tool (etadm), on the tunnel address only (e.g. `10.66.0.1:8090`: unreachable from the internet), with `Authorization: Bearer $ETRELAY_CONTROL_TOKEN` (32 characters at least). Binding is retried every 10 s: the tunnel may come up after etrelay.
|
||||
`--control <addr>` serves a small HTTP API for admin tools (etadm), on the tunnel address only (e.g. `10.66.0.1:8090`: unreachable from the internet). Binding is retried every 10 s: the tunnel may come up after etrelay.
|
||||
|
||||
The ET server sees every player as `<tunnel address>:<port>`, the port of the player's session socket; the API maps it back to the real address.
|
||||
### Registering a server
|
||||
|
||||
| Method | URL | |
|
||||
|---|---|---|
|
||||
| GET | `/sessions` | `[{"client": "203.0.113.7:27960", "port": 51234, "since": <unix>}]`: open sessions |
|
||||
| GET | `/bans` | `[{"ip", "reason", "at", "until"}]` (`until` null: until lifted) |
|
||||
| PUT | `/bans/{ip}` | `{"reason": "...", "minutes": N}` (`minutes` left out: until lifted); cuts the address's sessions at once |
|
||||
| DELETE | `/bans/{ip}` | 204, 404 when not banned |
|
||||
An admin tool registers the ET server it runs, with two things:
|
||||
|
||||
A banned address's packets are dropped before reaching the server: the player times out, and can't come back. Bans are kept in `<state-dir>/bans.json` (written atomically), expired ones forgotten.
|
||||
- **the relay's token** (`$ETRELAY_CONTROL_TOKEN`, 32 characters at least): it only lets a server register;
|
||||
- **a key it picks** (32 characters at least), and the proof it runs that server: it makes the server show the key's fingerprint in the serverinfo cvar `etrelay_proof` (`sets etrelay_proof <fingerprint>` on its console), where etrelay reads it with a `getstatus` before taking the key. The fingerprint is the first 32 hex digits of the key's SHA-256: public while registering, it doesn't give the key away. The tool removes the cvar afterwards (`set etrelay_proof ""`).
|
||||
|
||||
Then every other call carries the key (`Authorization: Bearer <key>`), which names the server: a tool only sees and changes its own server. etrelay keeps the key's SHA-256, never the key.
|
||||
|
||||
| Method | URL | Bearer | |
|
||||
|---|---|---|---|
|
||||
| PUT | `/server` | token | `{"name": "...", "port": 27960, "upstream_ip": "10.66.0.20", "upstream_port": 27960, "key": "..."}`: register the ET server `upstream_ip:upstream_port` (as seen from the relay; `upstream_ip` left out: the caller's address) on the public `port`. Answers the registration (below) |
|
||||
| GET | `/server` | key | `{"port", "upstream", "name", "since", "alive"}`: `alive`, the ET server answered the last probe, the masters are told |
|
||||
| DELETE | `/server` | key | leave the relay: the port is freed, the masters get a flatline, the server's bans are dropped (204) |
|
||||
| GET | `/sessions` | key | `[{"client": "203.0.113.7:27960", "port": 51234, "since": <unix>}]`: open sessions |
|
||||
| GET | `/bans` | key | `[{"ip", "reason", "at", "until"}]` (`until` null: until lifted) |
|
||||
| PUT | `/bans/{ip}` | key | `{"reason": "...", "minutes": N}` (`minutes` left out: until lifted); cuts the address's sessions at once |
|
||||
| DELETE | `/bans/{ip}` | key | 204, 404 when not banned |
|
||||
|
||||
`PUT /server` answers:
|
||||
|
||||
| Status | |
|
||||
|---|---|
|
||||
| 200 | registered |
|
||||
| 400 | port outside `--ports`, key too short |
|
||||
| 401 | wrong token |
|
||||
| 403 | the ET server doesn't answer, or doesn't show the key's fingerprint |
|
||||
| 409 | another server holds the port, or the ET server is the command line's |
|
||||
|
||||
One server per public port, one port per server. Registering a server again (same ET server, another key: an admin tool reinstalled) takes it over: who proves to run it holds it, the former key is refused from then on. On the same port its sessions and bans go on; on another port the former one is freed. Unknown keys get 401: a tool whose relay lost its state registers again.
|
||||
|
||||
### Sessions and bans
|
||||
|
||||
The ET server sees every player as `<tunnel address>:<port>`, the port of the player's session socket; `/sessions` maps it back to the real address.
|
||||
|
||||
A banned address's packets are dropped before reaching the server: the player times out, and can't come back. Each server has its own bans, kept in `<state-dir>/bans-<port>.json` (written atomically), expired ones forgotten (`bans.json`: the command line's server).
|
||||
|
||||
## Build
|
||||
|
||||
@@ -67,20 +101,26 @@ make release # both
|
||||
Artifacts land in `dist/`, named `<name>-v<version>-<target triple>` with a `.sha256` checksum:
|
||||
|
||||
```
|
||||
dist/etrelay-v0.1.0-x86_64-unknown-linux-gnu # dynamic
|
||||
dist/etrelay-v0.1.0-x86_64-unknown-linux-musl # static
|
||||
dist/etrelay-v0.3.0-x86_64-unknown-linux-gnu # dynamic
|
||||
dist/etrelay-v0.3.0-x86_64-unknown-linux-musl # static
|
||||
```
|
||||
|
||||
## Run as a service
|
||||
|
||||
1. Binary and unit on the front server ([etrelay.service](etrelay.service) is a template: replace `@UPSTREAM@` with the ET server `ip:port` as seen from the front, `@ARGS@` with extra flags or nothing):
|
||||
1. The token servers register with, on the front server (root only, systemd hands it to the service):
|
||||
```bash
|
||||
install -d -m 700 /etc/etrelay
|
||||
echo "ETRELAY_CONTROL_TOKEN=$(head -c 32 /dev/urandom | od -An -tx1 | tr -d ' \n')" > /etc/etrelay/env
|
||||
```
|
||||
2. Binary and unit ([etrelay.service](etrelay.service) is a template: `@UPSTREAM@` is nothing when servers register, or one ET server's `ip:port` as seen from the front; `@ARGS@` the flags):
|
||||
```bash
|
||||
install -m 755 dist/etrelay-v<version>-x86_64-unknown-linux-musl /usr/local/bin/etrelay
|
||||
sed -e 's#@UPSTREAM@#<ip:port>#' -e 's#@ARGS@##' etrelay.service > /etc/systemd/system/etrelay.service
|
||||
sed -e 's#@UPSTREAM@##' -e 's#@ARGS@#--control <tunnel address>:8090 --ports 27960-27969#' etrelay.service > /etc/systemd/system/etrelay.service
|
||||
systemctl daemon-reload && systemctl enable --now etrelay
|
||||
```
|
||||
2. Nothing else on the front may capture UDP 27960 (DNAT rule, nginx `stream`...), or players bypass the relay: `iptables -t nat -S | grep 27960` must print nothing.
|
||||
3. ET server `server.cfg`: stop its own heartbeats, they leave through the home connection with the wrong IP:
|
||||
3. The UDP ports of `--ports` open on the front. Nothing else may capture them (DNAT rule, nginx `stream`...), or players bypass the relay: `iptables -t nat -S | grep 27960` must print nothing.
|
||||
4. Each admin tool gets the relay's address, the token and the public port it asks for (etadm: `--relay-url`, `--relay-token-file`, `--relay-port`, or its Relay page).
|
||||
5. ET server `server.cfg`: stop its own heartbeats, they leave through the home connection with the wrong IP:
|
||||
```
|
||||
set sv_master1 ""
|
||||
set sv_master2 ""
|
||||
@@ -92,7 +132,7 @@ dist/etrelay-v0.1.0-x86_64-unknown-linux-musl # static
|
||||
## Check
|
||||
|
||||
```bash
|
||||
journalctl -u etrelay -n 50 # "ET server ... is up"
|
||||
journalctl -u etrelay -n 50 # "registered ... on port 27960", "ET server ... is up"
|
||||
tcpdump -ni any udp port 27950 # heartbeat out, getChallenge/getInfo in, answers out
|
||||
```
|
||||
|
||||
|
||||
Reference in New Issue
Block a user