fix: update doc arch
@@ -18,11 +18,13 @@ Our ET server sits at home, reachable from the Internet only through the front s
|
|||||||
|
|
||||||
`etrelay` runs on the front and owns the public UDP port 27960. It does two things on **the same socket**:
|
`etrelay` runs on the front and owns the public UDP port 27960. It does two things on **the same socket**:
|
||||||
|
|
||||||
- **Proxy**: forwards every client (players, server browsers, masters) to the ET server, like nginx `stream`.
|
- **Proxy**: forwards every client (players, server browsers, masters) to the ET server, one upstream socket per client address. Packets a client has no business sending (`rcon`...) are dropped on the way.
|
||||||
- **Heartbeat**: sends the heartbeats itself, so the master challenges the **front** address. That challenge is just another client packet: it goes through the proxy, the ET server answers it, the answer leaves from the front address, and the master lists the front.
|
- **Heartbeat**: sends the heartbeats itself, so the master challenges the **front** address. That challenge is just another client packet: it goes through the proxy, the ET server answers it, the answer leaves from the front address, and the master lists the front.
|
||||||
|
|
||||||
The ET server's own heartbeats are disabled (`sv_master1..5 ""`).
|
The ET server's own heartbeats are disabled (`sv_master1..5 ""`).
|
||||||
|
|
||||||
|
The ET server is its own WireGuard peer of the front, with no proxy in between (the DNAT rule and the home nginx `stream` are gone). Each client reaches it from its own port of the front's tunnel address: the **control API** maps that port back to the real address, and bans addresses (see [Control API and bans](#control-api-and-bans)).
|
||||||
|
|
||||||
Masters are independent: a server is only listed on the masters it heartbeats, and each client browses its own master (ET 2.60b: `etmaster.idsoftware.com`, ET: Legacy: `etmaster.net`). The relay heartbeats both by default; diagrams show `etmaster.net`, the id Software master works the same way.
|
Masters are independent: a server is only listed on the masters it heartbeats, and each client browses its own master (ET 2.60b: `etmaster.idsoftware.com`, ET: Legacy: `etmaster.net`). The relay heartbeats both by default; diagrams show `etmaster.net`, the id Software master works the same way.
|
||||||
|
|
||||||

|

|
||||||
@@ -51,17 +53,39 @@ The last two lines are a server tracker that got the address from the master.
|
|||||||
|
|
||||||

|

|
||||||
|
|
||||||
The ET server sees every client as the nginx `stream` host (one source IP, one port per session): IP bans don't work, same as before the relay.
|
The ET server sees every client as `<front-wg>:<port>`: one source IP (the front's tunnel address), one port per session, the port of the session's upstream socket. Its own IP bans can't tell players apart: bans are etrelay's (below). A player coming back after the session expired gets a new port.
|
||||||
|
|
||||||
|
## Control API and bans
|
||||||
|
|
||||||
|
With `--control <addr>`, an HTTP API listens on the front's tunnel address (unreachable from the internet), bearer token on top. It serves an admin tool on the tunnel side:
|
||||||
|
|
||||||
|
- `GET /sessions`: the open sessions, real client address and port the ET server sees, so the tool can put a real address on each player.
|
||||||
|
- `GET /bans`, `PUT /bans/{ip}`, `DELETE /bans/{ip}`: bans by source IP, with a reason and an optional duration.
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
A ban cuts the address's sessions at once, and the relay loop drops its packets before any session lookup: the ET server gets nothing more from the player and times them out, and they can't come back. Bans are saved to `<state-dir>/bans.json` on each change (written next to it, then renamed), expired ones forgotten.
|
||||||
|
|
||||||
|
## Filter
|
||||||
|
|
||||||
|
Only what clients and masters send a server gets through, checked on the first word as the ET server tokenizes it:
|
||||||
|
|
||||||
|
- in-game packets (no `\xff\xff\xff\xff` prefix): always;
|
||||||
|
- `getstatus`, `getinfo`, `getchallenge`, `connect`, PunkBuster's `PB_...`: allowed;
|
||||||
|
- anything else, `rcon` above all, dropped: it travels in clear, and a stream of it blocks the local admin's own rcon (2.60b runs one rcon per 500 ms, whoever sends it).
|
||||||
|
|
||||||
|
`getstatus` and `getinfo` are limited per source IP (token bucket: a burst of 5, then 1/s): their answers are much bigger than the query, and from a spoofed source they would make the server a DDoS reflector. The limiter remembers at most 65 536 IPs, idle ones forgotten after 60 s.
|
||||||
|
|
||||||
## Internals
|
## Internals
|
||||||
|
|
||||||
Single-threaded tokio runtime, three kinds of tasks:
|
Single-threaded tokio runtime, four kinds of tasks. The sessions and the bans are shared by the relay loop and the control API behind a mutex, never held across an `await`:
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
- **relay loop**: reads the public socket, finds the client's session and pushes the packet into its queue. No session, or an expired one: opens a new one (and purges expired entries). A full queue drops the packet, like any congested UDP hop.
|
- **relay loop**: reads the public socket, runs the filter and the ban check, finds the client's session and pushes the packet into its queue. No session, or an expired one: opens a new one (and purges expired entries), unless `MAX_SESSIONS` are open. A full queue drops the packet, like any congested UDP hop.
|
||||||
- **session task**: owns one ephemeral socket connected to the ET server. Forwards the queue to the ET server and the ET server answers back to the client **through the public socket** (so they come from `:27960`). Ends after `SESSION_TIMEOUT` without a packet either way.
|
- **session task**: owns one ephemeral socket connected to the ET server. Forwards the queue to the ET server and the ET server answers back to the client **through the public socket** (so they come from `:27960`). Ends after `SESSION_TIMEOUT` without a packet either way, or at once when a ban drops its queue's sender.
|
||||||
- **heartbeat task**: probes the ET server with `getinfo` on its own socket (up to 3 attempts, 3 s each), then heartbeats the masters through the public socket. Masters are resolved on every heartbeat (DNS may change).
|
- **heartbeat task**: probes the ET server with `getinfo` on its own socket (up to 3 attempts, 3 s each), then heartbeats the masters through the public socket. Masters are resolved on every heartbeat (DNS may change).
|
||||||
|
- **control API task** (with `--control`): axum on the tunnel address. Binding is retried every 10 s: the tunnel may come up after etrelay.
|
||||||
|
|
||||||
### Heartbeat states
|
### Heartbeat states
|
||||||
|
|
||||||
@@ -78,6 +102,10 @@ The master drops a server that stops heartbeating after a while anyway; the flat
|
|||||||
| `SESSION_TIMEOUT` | 2 min | Idle clients (server browsers) freed quickly; players send packets constantly |
|
| `SESSION_TIMEOUT` | 2 min | Idle clients (server browsers) freed quickly; players send packets constantly |
|
||||||
| `SESSION_QUEUE` | 64 packets | Burst absorption per client before dropping |
|
| `SESSION_QUEUE` | 64 packets | Burst absorption per client before dropping |
|
||||||
| `MAX_PACKET` | 65 535 B | Any UDP datagram: `statusResponse` easily exceeds nginx's old 512 B buffer |
|
| `MAX_PACKET` | 65 535 B | Any UDP datagram: `statusResponse` easily exceeds nginx's old 512 B buffer |
|
||||||
|
| `MAX_SESSIONS` | 2048 | One socket each: spoofed sources must not use up the file descriptors (`LimitNOFILE=4096` in the unit) |
|
||||||
|
| `QUERY_BURST`, `QUERY_RATE` | 5, then 1/s | `getstatus`/`getinfo` per source IP: a browser refresh or a master's check passes, a reflection attack doesn't |
|
||||||
|
| `MAX_QUERY_SOURCES`, `QUERY_FORGET` | 65 536 IPs, 60 s | The limiter's memory stays bounded under a flood of spoofed sources |
|
||||||
|
| `MIN_TOKEN` | 32 characters | Shortest control API token accepted |
|
||||||
|
|
||||||
### Protocol cheat sheet
|
### Protocol cheat sheet
|
||||||
|
|
||||||
@@ -92,5 +120,7 @@ Connectionless packets start with `\xff\xff\xff\xff`, then a command:
|
|||||||
| `infoResponse` | server → master/browser | Hostname, map, player count, challenge |
|
| `infoResponse` | server → master/browser | Hostname, map, player count, challenge |
|
||||||
| `getstatus` / `statusResponse` | browser ↔ server | Full cvars + player list |
|
| `getstatus` / `statusResponse` | browser ↔ server | Full cvars + player list |
|
||||||
| `getchallenge` / `challengeResponse` / `connect` | player ↔ server | Connection handshake |
|
| `getchallenge` / `challengeResponse` / `connect` | player ↔ server | Connection handshake |
|
||||||
|
| `PB_...` | PunkBuster ↔ server | PunkBuster's own packets |
|
||||||
|
| `rcon <password> <command>` | admin → server | Remote console: dropped by the relay |
|
||||||
|
|
||||||
Anything else (no prefix) is in-game traffic: forwarded as-is, only counted in `--debug` logs.
|
Packets with no prefix are in-game traffic: forwarded as-is, only counted in `--debug` logs.
|
||||||
|
|||||||
@@ -3,7 +3,8 @@
|
|||||||
UDP relay + master heartbeat for a Wolfenstein: Enemy Territory server reachable only through a tunnel: runs on the front server (public IP) and makes the server show up in the in-game browser.
|
UDP relay + master heartbeat for a Wolfenstein: Enemy Territory server reachable only through a tunnel: runs on the front server (public IP) and makes the server show up in the in-game browser.
|
||||||
|
|
||||||
```
|
```
|
||||||
players / masters ──> front :27960 (etrelay) ══wg══> <wg-peer>:27960 (nginx stream) ──> ET <et-server>:27960
|
players / masters ──> front :27960 (etrelay) ══wg══> ET <et-server>:27960
|
||||||
|
front <front-wg>:8090 (control API) <══wg══ admin tool
|
||||||
```
|
```
|
||||||
|
|
||||||
- **Proxy**: one upstream socket per client address, packets forwarded as-is both ways, session dropped after 2 min of silence.
|
- **Proxy**: one upstream socket per client address, packets forwarded as-is both ways, session dropped after 2 min of silence.
|
||||||
|
|||||||
|
Before Width: | Height: | Size: 24 KiB After Width: | Height: | Size: 30 KiB |
@@ -4,18 +4,19 @@ title After: everything goes through the front
|
|||||||
|
|
||||||
actor Player as player
|
actor Player as player
|
||||||
node "Master\netmaster.net:27950" as master
|
node "Master\netmaster.net:27950" as master
|
||||||
|
actor "Admin tool\n(tunnel side)" as admin
|
||||||
|
|
||||||
node "Front (public IP)" {
|
node "Front (public IP)" {
|
||||||
component "etrelay\n0.0.0.0:27960" as relay
|
component "etrelay\n0.0.0.0:27960" as relay
|
||||||
|
component "control API\n<front-wg>:8090" as control
|
||||||
}
|
}
|
||||||
|
|
||||||
node "Home" {
|
node "Home" {
|
||||||
component "nginx stream\n<wg-peer>:27960" as nginx
|
component "ET server\n<et-server>:27960\n(own WireGuard peer)" as et
|
||||||
component "ET server\n<et-server>:27960" as et
|
|
||||||
}
|
}
|
||||||
|
|
||||||
player <--> relay : game traffic
|
player <--> relay : game traffic
|
||||||
master <--> relay : heartbeat / getChallenge / getInfo
|
master <--> relay : heartbeat / getChallenge / getInfo
|
||||||
relay <--> nginx : WireGuard (wg0)
|
relay <--> et : WireGuard (wg0), direct:\none source port per client
|
||||||
nginx <--> et : LAN
|
admin --> control : WireGuard: sessions, bans\n(bearer token)
|
||||||
@enduml
|
@enduml
|
||||||
|
|||||||
|
After Width: | Height: | Size: 40 KiB |
@@ -0,0 +1,22 @@
|
|||||||
|
@startuml
|
||||||
|
skinparam shadowing false
|
||||||
|
title Control API: real address and ban of a player
|
||||||
|
|
||||||
|
participant "Admin tool" as admin
|
||||||
|
participant "etrelay control API\n<front-wg>:8090" as control
|
||||||
|
participant "etrelay\nrelay loop" as relay
|
||||||
|
participant "ET server" as et
|
||||||
|
actor "Player\n203.0.113.7" as player
|
||||||
|
|
||||||
|
note over admin, et : the ET server shows the player as <front-wg>:51234
|
||||||
|
admin -> control : GET /sessions
|
||||||
|
control --> admin : [{client: 203.0.113.7:27960, port: 51234}]
|
||||||
|
note right of admin : port 51234 = 203.0.113.7
|
||||||
|
|
||||||
|
== Ban ==
|
||||||
|
admin -> control : PUT /bans/203.0.113.7\n{reason, minutes}
|
||||||
|
control -> control : bans.json (atomic write)\ncut 203.0.113.7's sessions
|
||||||
|
player -> relay : game packets
|
||||||
|
relay -> relay : banned: dropped
|
||||||
|
note over et : no packet: times the player out
|
||||||
|
@enduml
|
||||||
|
Before Width: | Height: | Size: 35 KiB After Width: | Height: | Size: 44 KiB |
@@ -2,18 +2,20 @@
|
|||||||
skinparam shadowing false
|
skinparam shadowing false
|
||||||
title Player: one session per client address
|
title Player: one session per client address
|
||||||
|
|
||||||
actor "Player\n1.2.3.4:5555" as player
|
actor "Player\n203.0.113.7:27960" as player
|
||||||
participant "etrelay\nfront :27960" as relay
|
participant "etrelay\nfront :27960" as relay
|
||||||
participant "session task\n(ephemeral socket)" as session
|
participant "session task\n(socket :51234)" as session
|
||||||
participant "ET server" as et
|
participant "ET server" as et
|
||||||
|
|
||||||
player -> relay : getchallenge / connect / game packets
|
player -> relay : getchallenge / connect / game packets
|
||||||
|
relay -> relay : filter (command, query rate)\nban check
|
||||||
relay -> session : first packet: open session
|
relay -> session : first packet: open session
|
||||||
session -> et : forward (source = ephemeral port)
|
session -> et : forward (source = <front-wg>:51234)
|
||||||
|
note right of et : sees the player as\n<front-wg>:51234
|
||||||
et --> session : reply
|
et --> session : reply
|
||||||
session --> player : reply sent from front:27960
|
session --> player : reply sent from front:27960
|
||||||
...packets both ways, deadline pushed back on each packet...
|
...packets both ways, deadline pushed back on each packet...
|
||||||
...2 min without any packet...
|
...2 min without any packet...
|
||||||
session -> session : close (socket dropped)
|
session -> session : close (socket dropped)
|
||||||
note over relay : next packet from 1.2.3.4:5555\nopens a new session
|
note over relay : next packet from 203.0.113.7:27960\nopens a new session (new port)
|
||||||
@enduml
|
@enduml
|
||||||
|
|||||||
|
Before Width: | Height: | Size: 48 KiB After Width: | Height: | Size: 47 KiB |
@@ -4,7 +4,7 @@ title Master registration (every 5 min)
|
|||||||
|
|
||||||
participant "Master\netmaster.net:27950" as master
|
participant "Master\netmaster.net:27950" as master
|
||||||
participant "etrelay\nfront :27960" as relay
|
participant "etrelay\nfront :27960" as relay
|
||||||
participant "ET server\n(via wg + nginx)" as et
|
participant "ET server\n(via wg)" as et
|
||||||
|
|
||||||
== Liveness probe (dedicated ephemeral socket) ==
|
== Liveness probe (dedicated ephemeral socket) ==
|
||||||
relay -> et : getinfo etrelay
|
relay -> et : getinfo etrelay
|
||||||
@@ -16,7 +16,7 @@ relay -> master : heartbeat EnemyTerritory-1\nsource = front:27960
|
|||||||
|
|
||||||
== Master checks: handled like any client ==
|
== Master checks: handled like any client ==
|
||||||
master -> relay : getChallenge\nto front:27960
|
master -> relay : getChallenge\nto front:27960
|
||||||
relay -> relay : new session for master:27950
|
relay -> relay : filter: allowed, rate ok\nnew session for master:27950
|
||||||
relay -> et : getChallenge
|
relay -> et : getChallenge
|
||||||
et --> relay : challengeResponse
|
et --> relay : challengeResponse
|
||||||
relay --> master : challengeResponse\nsource = front:27960
|
relay --> master : challengeResponse\nsource = front:27960
|
||||||
|
|||||||
|
Before Width: | Height: | Size: 35 KiB After Width: | Height: | Size: 56 KiB |
@@ -4,7 +4,12 @@ title Tasks and data flow
|
|||||||
|
|
||||||
rectangle "relay loop (main task)" as loop {
|
rectangle "relay loop (main task)" as loop {
|
||||||
card "public socket\nrecv_from" as recv
|
card "public socket\nrecv_from" as recv
|
||||||
card "sessions:\nHashMap<client, Sender>" as map
|
card "filter\n(commands, query rate)" as filter
|
||||||
|
}
|
||||||
|
|
||||||
|
rectangle "shared state (Mutex)" as shared {
|
||||||
|
card "sessions:\nHashMap<client, Session>" as map
|
||||||
|
card "bans\n(bans.json)" as bans
|
||||||
}
|
}
|
||||||
|
|
||||||
rectangle "session task (one per client)" as st {
|
rectangle "session task (one per client)" as st {
|
||||||
@@ -17,10 +22,18 @@ rectangle "heartbeat task" as hb {
|
|||||||
card "probe socket" as probe
|
card "probe socket" as probe
|
||||||
}
|
}
|
||||||
|
|
||||||
recv --> map : lookup client
|
rectangle "control API task (axum)" as ctl {
|
||||||
|
card "GET /sessions\nGET/PUT/DELETE /bans" as routes
|
||||||
|
}
|
||||||
|
|
||||||
|
recv --> filter
|
||||||
|
filter --> bans : banned?
|
||||||
|
filter --> map : lookup client
|
||||||
map --> rx : try_send(packet)\n(full: drop)
|
map --> rx : try_send(packet)\n(full: drop)
|
||||||
rx --> up : send
|
rx --> up : send
|
||||||
up --> recv : recv, then public.send_to(client)
|
up --> recv : recv, then public.send_to(client)
|
||||||
tick --> probe : getinfo
|
tick --> probe : getinfo
|
||||||
tick --> recv : heartbeat via public socket
|
tick --> recv : heartbeat via public socket
|
||||||
|
routes --> map : list ports,\ndrop a banned IP's sessions
|
||||||
|
routes --> bans : set / lift
|
||||||
@enduml
|
@enduml
|
||||||
|
|||||||