etrelay/ENGINE.md

How etrelay works

Diagrams: PlantUML sources in diagrams/, PNGs regenerated with make diagrams.

The problem

A server shows up in the in-game browser only if a master server lists it. Registration is a 3-step handshake over UDP:

1. The game server sends heartbeat EnemyTerritory-1 to the master.
2. The master does not trust it: it queries the source IP:port of the heartbeat. etmaster.net sends getChallenge, then getInfo (classic Quake 3 masters send getinfo <challenge> directly).
3. The game server answers (challengeResponse, infoResponse). The master lists the source IP:port of those answers.

Our ET server sits at home, reachable from the Internet only through the front server and the WireGuard tunnel. Players went in through the front (DNAT, then nginx stream), but the ET server's own heartbeats went out through the home box: the master challenged the home IP, where nothing listens for it. Result: server reachable by IP, invisible in the browser.

The fix

etrelay runs on the front and owns the public UDP ports, one per ET server (27960 for the first). For each server it does two things on the same socket:

• Proxy: forwards every client (players, server browsers, masters) to the ET server, one upstream socket per client address. Packets a client has no business sending (rcon...) are dropped on the way.
• Heartbeat: sends the heartbeats itself, so the master challenges the front address. That challenge is just another client packet: it goes through the proxy, the ET server answers it, the answer leaves from the front address, and the master lists the front.

The ET server's own heartbeats are disabled (sv_master1..5 "").

The ET server is its own WireGuard peer of the front, with no proxy in between (the DNAT rule and the home nginx stream are gone). Each client reaches it from its own port of the front's tunnel address: the control API maps that port back to the real address, and bans addresses (see Control API and bans).

Masters are independent: a server is only listed on the masters it heartbeats, and each client browses its own master (ET 2.60b: etmaster.idsoftware.com, ET: Legacy: etmaster.net). The relay heartbeats both by default; diagrams show etmaster.net, the id Software master works the same way.

Master registration sequence

What it looks like in production (--debug):

heartbeat EnemyTerritory-1 -> etmaster.net:27950 (198.51.100.10:27950)
session + 198.51.100.10:27950 (2 active)
198.51.100.10:27950 -> ET  getChallenge (27 B)
198.51.100.10:27950 <- ET  challengeResponse (32 B)
198.51.100.10:27950 -> ET  getInfo (22 B)
198.51.100.10:27950 <- ET  infoResponse (303 B)
session + 203.0.113.7:45488 (3 active)
203.0.113.7:45488 -> ET  getstatus (14 B)
203.0.113.7:45488 <- ET  statusResponse (993 B)

The last two lines are a server tracker that got the address from the master.

Player sequence

The ET server sees every client as <front-wg>:<port>: one source IP (the front's tunnel address), one port per session, the port of the session's upstream socket. Its own IP bans can't tell players apart: bans are etrelay's (below). A player coming back after the session expired gets a new port.

Several servers: registering

The relay holds several ET servers. The ET protocol has no host name to tell them apart: each one gets its own public port (--ports), with its own socket, sessions, heartbeats and bans. Masters list servers by address and port, so several on the front's address are fine.

A server is relayed once its admin tool registered it, with PUT /server on the control API (--control <addr>, on the front's tunnel address, unreachable from the internet):

• The relay's token only lets a tool register: it keeps strangers of the tunnel off the public ports.
• The key is picked by the tool, and is what every later call carries: it names the server, so a tool only reaches its own. etrelay keeps its SHA-256 (servers.json), never the key.
• The proof: before taking a key, etrelay checks the tool runs the ET server it registers. The tool makes the server show the key's fingerprint (the first 128 bits of its SHA-256) in the serverinfo cvar etrelay_proof, etrelay reads it there with a getstatus. Only who holds that server's console can do it, wherever the request comes from (a NAT between the two changes nothing), and the fingerprint, public for a moment, doesn't give the key away. An empty cvar is left out of serverinfo: the tool sets it to "" afterwards (unset isn't in 2.60b).

What can't happen:

• two servers on one port: the second one gets a 409, the first one is untouched;
• another tool on a registered server: it has neither the key, nor the server's console to prove another one.

One limit: a server that is gone for good without DELETE /server keeps its port (the masters drop it, nothing answers there), until its entry is removed from servers.json and etrelay restarted.

An ET server registered again is taken over by who proves to run it (a tool reinstalled, with a new key): the former key is refused from then on. On the same port the relay goes on (sessions, bans); on another port the former one is freed. A relay that lost its state answers 401 to every key: tools register again by themselves.

The registered servers are relayed again after a restart. DELETE /server leaves the relay: the masters get a flatline, the port is freed, the bans dropped.

etrelay <UPSTREAM> still relays one server without any of this, on --listen: it has no key, so no control API over it.

Control API and bans

Once registered, the admin tool asks about its server, with its key:

• GET /server: the registration, and whether the ET server answered the last probe.
• GET /sessions: the open sessions, real client address and port the ET server sees, so the tool can put a real address on each player.
• GET /bans, PUT /bans/{ip}, DELETE /bans/{ip}: bans by source IP, with a reason and an optional duration.

A ban cuts the address's sessions at once, and the relay loop drops its packets before any session lookup: the ET server gets nothing more from the player and times them out, and they can't come back. Each server has its own bans, saved to <state-dir>/bans-<port>.json on each change (written next to it, then renamed), expired ones forgotten.

Filter

Only what clients and masters send a server gets through, checked on the first word as the ET server tokenizes it:

• in-game packets (no \xff\xff\xff\xff prefix): always;
• getstatus, getinfo, getchallenge, connect, PunkBuster's PB_...: allowed;
• anything else, rcon above all, dropped: it travels in clear, and a stream of it blocks the local admin's own rcon (2.60b runs one rcon per 500 ms, whoever sends it).

getstatus and getinfo are limited per source IP (token bucket: a burst of 5, then 1/s): their answers are much bigger than the query, and from a spoofed source they would make the server a DDoS reflector. The limiter remembers at most 65 536 IPs, idle ones forgotten after 60 s.

Internals

Single-threaded tokio runtime, four kinds of tasks. Each server has its relay loop, its heartbeat task and its state (sessions, bans), shared with the control API behind a mutex. The registry (the servers by public port, the hash of their key) is behind another one, which the packets never take. Neither is held across an await:

• relay loop (one per server): reads the server's public socket, runs the filter and the ban check, finds the client's session and pushes the packet into its queue. No session, or an expired one: opens a new one (and purges expired entries), unless MAX_SESSIONS are open, all servers together. A full queue drops the packet, like any congested UDP hop.
• session task: owns one ephemeral socket connected to the ET server. Forwards the queue to the ET server and the ET server answers back to the client through the public socket (so they come from :27960). Ends after SESSION_TIMEOUT without a packet either way, or at once when a ban drops its queue's sender.
• heartbeat task (one per server): probes the ET server with getinfo on its own socket (up to 3 attempts, 3 s each), then heartbeats the masters through the public socket. Masters are resolved on every heartbeat (DNS may change).
• control API task (with --control): axum on the tunnel address. Binding is retried every 10 s: the tunnel may come up after etrelay. Registering starts a server's two tasks, leaving aborts them and drops its sessions.

The code follows that split: server (one relayed server), registry (which servers, servers.json), proof (keys and their proof), control (the API), et (the probes: getinfo, getstatus), filter, bans.

Heartbeat states

The master drops a server that stops heartbeating after a while anyway; the flatline only makes it faster.

Constants

Constant	Value	Why
HEARTBEAT_INTERVAL	5 min	Same period as the ET server itself
PROBE_ATTEMPTS × PROBE_TIMEOUT	3 × 3 s	One lost UDP packet must not flatline the server
SESSION_TIMEOUT	2 min	Idle clients (server browsers) freed quickly; players send packets constantly
SESSION_QUEUE	64 packets	Burst absorption per client before dropping
MAX_PACKET	65 535 B	Any UDP datagram: statusResponse easily exceeds nginx's old 512 B buffer
MAX_SESSIONS	2048	One socket each, all servers together: spoofed sources must not use up the file descriptors (LimitNOFILE=4096 in the unit)
QUERY_BURST, QUERY_RATE	5, then 1/s	getstatus/getinfo per source IP: a browser refresh or a master's check passes, a reflection attack doesn't
MAX_QUERY_SOURCES, QUERY_FORGET	65 536 IPs, 60 s	The limiter's memory stays bounded under a flood of spoofed sources
MIN_TOKEN, MIN_KEY	32 characters	Shortest relay token and server key accepted
PROOF_BYTES	16	Bytes of the key's SHA-256 shown as its fingerprint: serverinfo holds 1024 characters for every cvar

Protocol cheat sheet

Connectionless packets start with \xff\xff\xff\xff, then a command:

Command	From → to	Meaning
heartbeat EnemyTerritory-1	server → master	"I'm alive, come check"
heartbeat ETFlatline-1	server → master	"I'm going down"
getChallenge / challengeResponse	master ↔ server	Master check before getInfo (etmaster.net)
getinfo / getInfo	master/browser → server	Short info request (optional challenge)
infoResponse	server → master/browser	Hostname, map, player count, challenge
getstatus / statusResponse	browser ↔ server	Full cvars + player list
getchallenge / challengeResponse / connect	player ↔ server	Connection handshake
PB_...	PunkBuster ↔ server	PunkBuster's own packets
rcon <password> <command>	admin → server	Remote console: dropped by the relay

Packets with no prefix are in-game traffic: forwarded as-is, only counted in --debug logs.