diff --git a/docker/compose.yaml b/docker/compose.yaml index 99df2733..f33a6cda 100644 --- a/docker/compose.yaml +++ b/docker/compose.yaml @@ -26,6 +26,8 @@ services: mem_limit: 512m environment: NODE_OPTIONS: --max-old-space-size=384 + # mem_limit, which the relay sheds load short of (docs/RELAY.md). + MEMORY_MAX_BYTES: "536870912" # The room page's origin (the Pages site, say). Unset, the relay # has no accounts: the account API is not there, and a session in # a hello is ignored. diff --git a/docs/JAM_BACKLOG.md b/docs/JAM_BACKLOG.md index 9ce6a795..852b0104 100644 --- a/docs/JAM_BACKLOG.md +++ b/docs/JAM_BACKLOG.md @@ -355,7 +355,56 @@ is many relays with a rule for which one. One Node process carries thousands of musician rooms, so this is for tens of thousands of concurrent sessions. -## 8. Decisions to make early +## 8. What the relay's memory model misses + +**What.** The relay charges each room a model of what its document +holds, and bounds the rooms by those charges. One thing the model does +not see: Yjs splits a text item by slicing its string, and V8 keeps a +slice of 13 characters or more as a view of the parent string, so a +`ContentString` cut down to a few characters keeps its whole original +string alive. `watch` credits the deleted parts as freed, so the heap +can grow past what the rooms are charged. + +**Measured** with 20 rooms, each sent a text of a million characters +in one update and then a deletion of all but 41 characters in its +middle. After a full collection the relay held about 21 MB more, and +the rooms were charged under 1 MB. + +**Today** the memory backstop covers it: past its share of memory the +relay sheds load whatever the rooms are charged. But it picks the room +to shed by charge, with its sockets' queues: a room holding what the +model misses is charged less than it holds, so the largest rooms go +first, and the one holding the memory may go last. + +**Candidate fixes:** + +- Charge a sliced text by the length of the parent string it retains, + not by its own. +- Copy a sliced string when Yjs splits an item, so the parent can be + collected. This needs a change to Yjs. + +**Done when** a room's charge bounds what its texts keep alive, and the +measurement above holds the heap to about what the rooms are charged. + +## 9. Smaller relay gaps + +- **A second catch-up refused within a second waits 15 s.** The relay + tells a refusal once a second for each type, so the page hears of the + first and its wait for the second runs out (room.js, `CATCHUP_WAIT`). + Telling every catch-up refusal, as for counted chat lines, fixes it. +- **Every update is read twice.** `refusal()` decodes it to price it + and the apply decodes it again, about 85% more on a large update. + Pricing from the apply's own read, or applying what was priced, would + read it once. +- **Empty rooms go for an update that is then not taken.** `refusal()` + lets go of the empty rooms an update needs before it is applied; one + that then fails to decode has cost them for nothing. +- **`Queues.trim` may count a fan-out Buffer once per socket.** A line + sent to a room is one Buffer in every recipient's `bufferedAmount`, + so the total can be several times what is held, and sockets cut past + `QUEUED_TOTAL_MAX_BYTES` sooner than they need be. + +## 10. Decisions to make early The short list of things that are free now and costly later, gathered from above: diff --git a/docs/RELAY.md b/docs/RELAY.md index a5f67be0..43831698 100644 --- a/docs/RELAY.md +++ b/docs/RELAY.md @@ -245,17 +245,136 @@ relay looks at the sessions behind its open rooms once a minute and closes those that have ended, so a ban or a revoke empties the account out of every room within the minute. +## Limits + +What one client can cost the relay is bounded, by constants in +`wasm/web/relay.mjs`. A client past one is refused or cut, and told why +on its room socket where that can reach it, which the page shows as the +room closing. A refusal that passes -- too many new rooms, a full room +or relay, a page that fell behind or sent too much, a relay short of +memory -- the page waits +out and joins again by itself, as after a relay restart, no sooner +than the relay's `retryMs` when it gives one; the rest it leaves to +Rejoin. + +- Send queues: 16 MiB queued for one socket, 128 MiB for all of them, + looked at every quarter second. A page that stops reading is cut + ("the connection fell too far behind the room"). The reason reaches + it when its document socket fell behind, on the room socket; when + the room socket did, the reason is queued behind its backlog, and a + page that has not read that within 5 s sees the room close without + it. Over the total, the socket holding the most is cut at once, with + no time to say why. A catch-up or a sync of the whole document is held apart + from its socket's cap. A gesture or a signal is not queued for a + peer with half its cap queued, rather than cutting it. +- What a socket's messages queue for others: 8 MiB at once and 4 MiB a + second, each message's size times its recipients; a `to` list counts + each peer once, and as many as a room holds. Past it a message is + dropped, as one past its rate. A transport command -- a start, a + stop, an edit -- has 64 MiB at once and 8 MiB a second of its own, so + the longest line fits a full room. A start, a stop or a tempo dropped, + for this or for its rate, is still the run the relay keeps, since the + mesh carried it to every peer; an edit, which goes by the relay alone, + dropped changes nothing. +- What a socket sends, counted before a frame is read: a room socket + 4 MiB at once and 1 MiB a second, a peer's document sockets between + them 16 MiB and 2 MiB. Past it a frame is dropped unread, as one past + its rate, and counts as one drop for every 64 KiB of it, so that a + flood of long lines is cut too. The document updates of one address's + peers between them have the same, and an IPv6 /48's four times it: + reading an update is most of what it costs, up to a quarter second of + the loop for a frame of one-byte structs, and an address has as many + peers as it has joins. Past it the document socket is cut. +- Rooms: a room is made by the first hello the relay welcomes into it, + at most 4096 of them, holding at most 128 MiB between them by what + each is charged: 8 KiB, and what its document holds of the heap, + measured after every update -- 320 B a struct, whole or split from + another, 512 B a client, 1 KiB a type, 80 B a value or a key, and 2 B + a character of a string or of an XML element's name; one room at most + 16 MiB, and a document inside the document, which no page makes, is + past any room's most. An update that could + take a room past that, read before it is applied, is refused ("the + room's document is as large as the relay keeps one"): the page is told + on its room socket, which stays, and its document socket is closed; + the page stops its document, which holds the edit, until it is + reloaded. One that adds nothing the room has not, or lets go of as + much as it adds, is taken, so a room at its most is still joined, + read and cut down. A room past its most, or a relay + past its budget, is switched to no other piece. An update that builds + on structs the room does not have is held by Yjs until they arrive, + and charged what it could add, or twice its size, while it is; past 64 + KiB held, it goes, and the document socket whose frames added the most + of it is cut. A frame Yjs cannot read closes its document socket. The + commands a playing run keeps for late joiners, 8 MiB of them at most, + are charged to its room at their size, and the copy of the document + its start named at two bytes a character, and as JSON once a catch-up + has been asked for; a command that does not fit, or an edit that fits + only without them, makes the run one a joiner cannot catch up with, + and it is handed the document alone. Past either cap, the rooms empty + longest go, as many as make room for a new room or an edit; with too + few to, the room is refused ("the relay has as many rooms as it can + hold"), and the edit ("the relay holds as much as it can"), which the + page sends again ten seconds later rather than stopping its document. An empty + room is otherwise kept for ten minutes. One address may make 20 rooms + at once and one more every 3 s, and an IPv6 /48 60 and one a second, + read through `TRUST_PROXY` as the account API's limits are; a room + refused for the caps before its piece is read does not count, and one + seeded and then refused does. +- People: 64 in a room. Past that a hello is refused ("the room is + full"), unless it is a page joining again with a live ticket, whose + old peer goes. One address may join rooms 128 times at once and once + more a second, and an IPv6 /48 512 times and four a second; past that + a hello is refused ("too many joins from your address"), with when to + try again, which a page waits for. A page waiting on that, or on too + many new rooms, spends none of its eight tries, and tries again when + told, a second apart at random: a class of 300 behind one address, + all joining again after a relay restart, is back within three + minutes, 128 at once and the rest one a second, the last of them + having asked about a hundred times. The steady rate stays at one a second rather than more because + every join is told to everyone in its room; a client joining and + leaving as fast as it could otherwise did so 2300 times a second. + A room socket has 5 s to say hello, and one address may hold 128 + open that have not been welcomed; past that the upgrade is refused + (429). +- Message rates: every type a room socket sends has a burst and a rate + a second (`RATES`), set well above what a page sends; chat's is 5 at + once and 5 a second. Past one a message is dropped and the page told + so, once a second for each type, and for each chat line, so the page + can mark it; a socket that has had 1000 dropped, at 100 more a + second, is cut ("this page sent more than the relay takes"). A peer's document sockets' frames are limited the same way, + between them, and their sync step 1s, each answered with the whole + document, to 4 and then one every 10 s; past that the socket is + closed, and its provider connects again. + +- Memory: what the rooms are charged is a model of what they hold, and + apart from it the relay looks at its heap four times a second. Past + 75% of the most V8 gives it, or, with `MEMORY_MAX_BYTES` set to the + container's limit, past 75% of that in heap and Buffers together, it + collects its garbage, and while it is still past 60% sheds the empty + rooms and one thing more a look, twice as many at each look it is + still over, what costs nobody their session first: the largest + runs' logs, overflowed, so that a late joiner has the document alone. + Only then a room: the one whose charge grew the most since the last + look, or with none grown the one charged the most with what its + sockets have queued. Everyone in it is told "the relay is short of + memory; joining again shortly" (`why: 'memory'`, with a `retryMs` of + 10 s), and their pages join again by themselves. Each is logged, and counted in the metrics' `shed`. What + grows faster than a look -- many rooms each sent a megabyte at once, + in the same turn of the loop -- it cannot catch, and the charges are + what bound. + ## Metrics - `METRICS_PORT`: a port to serve the relay's metrics on, for a load test (`wasm/web/relayload.mjs`): `GET /` answers JSON with the event loop's delay, the worst lag of a 10 ms timer and the CPU used, all over the window since the last reset; memory; counts of rooms, peers - and sockets; and messages and bytes each way by type, since the start. - `GET /?reset` answers the same and starts a new window; `window` - counts the resets. Have one scraper reset -- relayload's first shard - does -- and anyone else read without it. Unset, there is no such port, - and nothing is counted. + and sockets; messages and bytes each way by type, since the start; + what the rooms are charged, and what has been shed for memory + (Limits). `GET /?reset` answers the + same and starts a new window; `window` counts the resets. Have one + scraper reset -- relayload's first shard does -- and anyone else read + without it. Unset, there is no such port, and nothing is counted. It listens on 127.0.0.1 alone, and never on the relay's own port: nginx reaches that port from loopback, so a loopback check there would let @@ -270,5 +389,3 @@ with `METRICS_PORT: "9100"` in the service's `environment`. - TURN. Peers whose NATs defeat STUN fall back to the relay forwarding their commands (`mesh.js`), which works but adds a hop. -- Limits on rooms. Anyone who can reach the relay can open one; the - account API is the part that is rate-limited. diff --git a/wasm/web/accounts.mjs b/wasm/web/accounts.mjs index 55d9fa17..d343ebc9 100644 --- a/wasm/web/accounts.mjs +++ b/wasm/web/accounts.mjs @@ -569,65 +569,86 @@ export const taken = (handle) => export const passkeyTaken = () => new ApiError(409, 'passkey_taken', 'that passkey is registered already'); -/* One token bucket per key, least recently seen first. */ -class RateLimiter +/* A token bucket: `burst' at once, then one back every `refillMs', by + the clock `now'. */ +export class Bucket { constructor ({ burst, refillMs }, now) { this.burst = burst; this.refillMs = refillMs; this.now = now; - this.buckets = new Map(); + this.tokens = burst; + this.at = now(); } - refilled (b, now) + refill () { + const now = this.now(); + /* A wall clock stepped back must not drain the bucket. */ - return Math.min(this.burst, b.tokens + - Math.max(0, now - b.at) / this.refillMs); + this.tokens = Math.min(this.burst, this.tokens + + Math.max(0, now - this.at) / this.refillMs); + this.at = now; } - take (key) + take (n = 1) + { + this.refill(); + + if (this.tokens < n) + return false; + + this.tokens -= n; + return true; + } + + waitMs () + { + this.refill(); + + return this.tokens >= 1 + ? 0 : Math.ceil((1 - this.tokens) * this.refillMs); + } +} + +/* One token bucket per key, least recently seen first. */ +class RateLimiter +{ + constructor (limit, now) + { + this.limit = limit; + this.now = now; + this.buckets = new Map(); + } + + take (key, n = 1) { - const now = this.now(); let b = this.buckets.get(key); if (b === undefined) { - b = { tokens: this.burst, at: now }; + b = new Bucket(this.limit, this.now); if (this.buckets.size >= TRACKED_MAX) this.buckets.delete(this.buckets.keys().next().value); } else - { - b.tokens = this.refilled(b, now); - b.at = now; this.buckets.delete(key); - } this.buckets.set(key, b); - - if (b.tokens < 1) - return false; - - b.tokens--; - return true; + return b.take(n); } waitMs (key) { - const b = this.buckets.get(key); - const tokens = b === undefined ? this.burst - : this.refilled(b, this.now()); - - return tokens >= 1 ? 0 : Math.ceil((1 - tokens) * this.refillMs); + return this.buckets.get(key)?.waitMs() ?? 0; } } /* A request's limits, narrowest first, so that a client already over its own spends nothing of the shared ones. */ -class TieredLimit +export class TieredLimit { constructor (tiers, now, onShared) { @@ -636,7 +657,7 @@ class TieredLimit this.onShared = onShared; } - take (client) + take (client, n = 1) { const site = siteKey(client); @@ -644,7 +665,7 @@ class TieredLimit [site === null ? null : this.site, site], [this.everyone, '*']]) { - if (limiter === null || limiter.take(key)) + if (limiter === null || limiter.take(key, n)) continue; if (key === '*') @@ -656,6 +677,17 @@ class TieredLimit limiter.waitMs(key) / 1000))) }); } } + + /* How long until `client' may take one, with nothing taken: 0 if it + may now. */ + waitMs (client) + { + const site = siteKey(client); + + return Math.max(this.own?.waitMs(client) ?? 0, + site === null ? 0 : this.site?.waitMs(site) ?? 0, + this.everyone?.waitMs('*') ?? 0); + } } /* The key a client is limited by: an IPv4 address (an IPv4-mapped one @@ -1022,6 +1054,32 @@ function bearer (authorization) return m[1]; } +/* What a request is limited as (clientKey), behind `trustProxy' proxies + that append to X-Forwarded-For. Behind a proxy that sends none, every + client is the proxy's address and shares its buckets: a few + registrations, or rooms, would hold off everyone's. Said once, + loudly, with `log'. */ +export function clientOf (trustProxy, log) +{ + let unforwarded = false; + + return (req) => + { + const header = req.headers['x-forwarded-for']; + + if (trustProxy > 0 && header === undefined && !unforwarded) + { + unforwarded = true; + log('relay: TRUST_PROXY is set but the proxy sends no ' + + 'X-Forwarded-For; every client shares one rate limit ' + + 'until it does (docs/RELAY.md)'); + } + + return clientKey(forwardedAddress(header, trustProxy) ?? + req.socket.remoteAddress ?? 'unknown'); + }; +} + /* ---- the routes ---- */ /* A request listener for everything under ACCOUNT_API. `corsOrigin' is @@ -1030,17 +1088,16 @@ function bearer (authorization) serves none of this without one (relay.mjs); `trustProxy' how many proxies in front append to X-Forwarded-For. Only count proxies that set it, or a client picks its own rate limit. - `passkeys' is a Passkeys (passkeys.mjs), or null for none. */ + `passkeys' is a Passkeys (passkeys.mjs), or null for none. `client' is + clientOf's, for one shared with the relay's rooms. */ export function accountRoutes (accounts, { corsOrigin = null, trustProxy = 0, log = accounts.log, - passkeys = null } = {}) + passkeys = null, + client: keyOf = + clientOf(trustProxy, log) } + = {}) { - /* Behind a proxy that sends no X-Forwarded-For, every client is the - proxy's address and shares its buckets: a few registrations would - hold off everyone's. Said once, loudly. */ - let unforwarded = false; - const routes = { '/register': ['POST', (c, a, body) => accounts.register(c, body)], '/login': ['POST', (c, a, body) => accounts.login(c, body)], @@ -1123,20 +1180,7 @@ export function accountRoutes (accounts, { corsOrigin = null, 'the body must be application/json', { Connection: 'close' }); - const forwarded = forwardedAddress(req.headers['x-forwarded-for'], - trustProxy); - - if (trustProxy > 0 && req.headers['x-forwarded-for'] === - undefined && !unforwarded) - { - unforwarded = true; - log('accounts: TRUST_PROXY is set but the proxy sends no ' + - 'X-Forwarded-For; every client shares one rate limit ' + - 'until it does (docs/RELAY.md)'); - } - - const client = clientKey(forwarded ?? req.socket.remoteAddress ?? - 'unknown'); + const client = keyOf(req); const body = method === 'POST' ? await readJson(req, large ? RESPONSE_MAX_BYTES : BODY_MAX_BYTES) diff --git a/wasm/web/jam.js b/wasm/web/jam.js index a672f3b0..9f4c44c7 100644 --- a/wasm/web/jam.js +++ b/wasm/web/jam.js @@ -2014,6 +2014,19 @@ function openRoom (url, roomName, name, opts) status(`The relay did not switch to ${m.piece}: ${m.why}.`); showPiece(); } + /* A Play, a Stop or a tempo went over the mesh as well, and + missed only the peers the relay carries for; an edit goes + by the relay alone. Said, for whoever pressed it to press + it again. */ + else if (m.of === 'transport') + status(`The relay did not pass your ` + + `${{ start: 'Play', stop: 'Stop' }[m.op] ?? + m.op ?? 'command'} on` + + (['start', 'stop', 'tempo'].includes(m.op) + ? ', so only peers on the mesh had it' + : ' to the room') + `: ${m.why}.`); + else if (m.of === 'edit') + editRefused(m); }) .on('switched', switched) .on('clock', () => { showNumbers(); enable(); }) @@ -2129,6 +2142,42 @@ function followTickets () room.on('ticket', (ticket) => { provider.params = { ticket }; }); } +/* Whether the relay refused an edit of this page's, which this document + holds and the room's does not: the provider would send it again at + every reconnect, and be refused again. The document stops here, as + when the room is lost, and stays stopped through a rejoin; a reload + brings the room's back. The room plays on. One refused for the relay + being full stops only until it is sent again, no more often than the + relay answers a document socket's sync (relay.mjs, SYNC_ASKS). */ +let docRefused = false; +let editRetry = null; +const EDIT_RETRY_MS = 10 * 1000; + +function editRefused (m) +{ + provider?.disconnect(); + $('editor').inert = true; + clearTimeout(editRetry); + + if (m.why === 'rooms') + { + status('The relay holds as much as it can; sending your edit ' + + `again in ${EDIT_RETRY_MS / 1000} s...`); + editRetry = setTimeout(() => + { + provider.connect(); + $('editor').inert = false; + }, EDIT_RETRY_MS); + return; + } + + docRefused = true; + status((m.why === 'big' + ? 'Your edit was too large for the room and was not kept.' + : `Your edit was not kept: ${m.text}.`) + + ' Reload the page to go on editing.'); +} + /* One mesh to a room socket: a room lost while the join awaited the document is joined again, and given its mesh, before the join goes on to open one. */ @@ -2150,20 +2199,38 @@ const REJOIN_FIRST_MS = 1000; const REJOIN_MAX_MS = 30 * 1000; const REJOIN_TRIES = 8; +/* How long a room joined again stays up before the tries start over: a + page the relay cuts again sooner -- one that keeps falling behind, or + keeps sending too much -- goes on backing off, and stops. */ +const REJOIN_STEADY_MS = 30 * 1000; + +/* What the relay closes a room for that may pass, so the page tries + again as for a relay gone away: too many new rooms, or a full room or + relay; a page that fell behind, or sent too much; an accounts file + busy; a relay short of memory. The rest stand until something + changes here: a session ended, a name taken, a page too old, a + document too large. */ +const PASSING = new Set(['rooms', 'full', 'slow', 'flood', 'accounts', + 'memory']); + let rejoinTimer = null; let rejoinTries = 0; +let rejoinSteady = null; /* The room socket `r' closed. Its tickets went with it, so the document socket would be refused at every retry and the editor would type into a document nobody else sees: the document stops, and stays as it is on this page, the editor read only, until the room is joined again -- by - itself, after a relay restart or a dropped network, or with Rejoin - when the relay said no or would not answer. */ + itself, after a relay restart, a dropped network or a refusal that + passes, no sooner than the relay says; or with Rejoin when the relay + said no for good or would not answer. */ function lost (r, refused) { if (r !== room) return; + clearTimeout(rejoinSteady); + clearTimeout(editRetry); provider?.disconnect(); mesh?.close(); $('editor').inert = true; @@ -2172,7 +2239,19 @@ function lost (r, refused) if (refused?.why === 'session') accounts.ended(r.session); - if (refused !== null || rejoinTries >= REJOIN_TRIES) + /* A refusal that says when to try again -- too many joins or new + rooms from this address -- is the relay pacing it, and spends no + try: a class behind one address, joining again after a relay + restart, is let in one a second past the burst, and waits its turn + however long that takes, a second apart at random so that the + waiting do not all ask at once. A cut for memory is everyone in + the rooms cut joining again at once, to a relay that may cut them + again: it spends a try, doubling the wait to REJOIN_MAX_MS, and + waits up to as long again at random. */ + const retryMs = Number(refused?.retryMs) || 0; + + if ((refused !== null && !PASSING.has(refused.why)) || + (retryMs === 0 && rejoinTries >= REJOIN_TRIES)) { status(refused !== null ? `The relay closed the room: ${refused.text}.` @@ -2180,11 +2259,15 @@ function lost (r, refused) return; } - const wait = Math.min(REJOIN_FIRST_MS * 2 ** rejoinTries++, - REJOIN_MAX_MS); + const wait = refused?.why === 'memory' + ? Math.min(retryMs * 2 ** rejoinTries++, REJOIN_MAX_MS) * + (1 + Math.random()) + : retryMs > 0 ? retryMs + Math.random() * 1000 + : Math.min(REJOIN_FIRST_MS * 2 ** rejoinTries++, REJOIN_MAX_MS); - status(`The relay went away; joining again in ${Math.round( - wait / 1000)} s...`); + status(`${refused === null ? 'The relay went away' + : `The relay closed the room: ${refused.text}`}` + + `; joining again in ${Math.round(wait / 1000)} s...`); clearTimeout(rejoinTimer); rejoinTimer = setTimeout(rejoin, wait); } @@ -2221,13 +2304,15 @@ async function rejoin () if (e.why === 'session') accounts.ended(session); - /* Refused: said, and left to Rejoin. Unreachable: tried again. */ + /* Refused: said, and tried again if it passes, or left to + Rejoin. Unreachable: tried again. */ lost(was, e.why === undefined ? null : { text: e.message, - why: e.why }); + why: e.why, + retryMs: e.retryMs }); return; } - rejoinTries = 0; + rejoinSteady = setTimeout(() => { rejoinTries = 0; }, REJOIN_STEADY_MS); room = next; maker = new Maker(room.peer, transportNow, @@ -2235,9 +2320,14 @@ async function rejoin () transportLead: maker.transportLead }); provider.params = room.ticket === null ? {} : { ticket: room.ticket }; followTickets(); - provider.connect(); + + if (!docRefused) + { + provider.connect(); + $('editor').inert = false; + } + openMesh(); - $('editor').inert = false; if (seat !== null) room.claim(seat); diff --git a/wasm/web/jamtest.mjs b/wasm/web/jamtest.mjs index ae5dd68f..c2874dfb 100644 --- a/wasm/web/jamtest.mjs +++ b/wasm/web/jamtest.mjs @@ -103,6 +103,7 @@ import { execFileSync } from 'node:child_process'; import { fileURLToPath } from 'node:url'; import { chromium, firefox } from 'playwright'; +import { WebSocketServer } from 'ws'; import { tapeBefore } from '../tape.mjs'; import { firstDifference, reference } from './piececheck.mjs'; @@ -1969,6 +1970,132 @@ async function accountsTogether (pages) + 'rejoined'}, the edit ${synced ? 'synced' : 'not synced'} -- ` + await why(A.page)); + /* Cut for falling behind, the room socket's reason said: that passes, + and the page joins again by itself too. */ + const r = relayServer.rooms.get('jamaccounts'); + + for (const p of r.peers.values()) + if (p.name === 'Ann') + r.drop(p.ws, 'slow', + 'the connection fell too far behind the room'); + + const retrying = await A.page.waitForFunction( + () => /behind the room; joining again in/.test( + document.getElementById('status').textContent), + null, { timeout: 10000 }).then(() => true, () => false); + const waitS = () => A.page.evaluate(() => Number(/joining again in (\d+) s/ + .exec(document.getElementById('status').textContent)?.[1])); + const firstS = await waitS(); + const backAgain = retrying && await A.page.waitForFunction( + () => /^Back in jamaccounts/.test( + document.getElementById('status').textContent), + null, { timeout: 15000 }).then(() => true, () => false); + + if (backAgain) + ok('a page cut for falling behind says so, and joins again'); + else + fail(`a page cut for falling behind: ${retrying ? 'retried' : 'not ' + + 'retried'}, ${backAgain ? 'back' : 'not back'} -- ` + + await why(A.page)); + + /* Cut again as soon as it is back: it waits longer this time. */ + for (const p of r.peers.values()) + if (p.name === 'Ann') + r.drop(p.ws, 'slow', + 'the connection fell too far behind the room'); + + const again = await A.page.waitForFunction( + () => /behind the room; joining again in/.test( + document.getElementById('status').textContent), + null, { timeout: 5000 }).then(() => true, () => false); + const againS = await waitS(); + + if (again && againS > firstS) + ok(`a page cut again as soon as it is back waits longer to join ` + + `again: ${againS} s, after ${firstS} s`); + else + fail(`a page cut again as soon as it is back: ${firstS} s, then ` + + `${againS} s -- ${await why(A.page)}`); + + await A.page.waitForFunction( + () => /^Back in jamaccounts/.test( + document.getElementById('status').textContent), + null, { timeout: 15000 }).catch(() => {}); + + /* A transport command the relay did not pass on is said. */ + for (const p of r.peers.values()) + if (p.name === 'Ann') + p.ws.send(JSON.stringify({ type: 'refused', of: 'transport', + op: 'start', + why: 'too fast; slow down' })); + + if (await A.page.waitForFunction( + () => /Play on, so only peers on the mesh had it: too fast/.test( + document.getElementById('status').textContent), + null, { timeout: 5000 }).then(() => true, () => false)) + ok('a Play the relay did not pass on is said'); + else + fail(`a Play the relay did not pass on: ${await why(A.page)}`); + + /* An edit refused for a full relay is sent again by itself. */ + for (const p of r.peers.values()) + if (p.name === 'Ann') + p.ws.send(JSON.stringify({ type: 'refused', of: 'edit', + why: 'rooms', + text: 'the relay holds as much as ' + + 'it can; try again later' })); + + const waiting = await A.page.waitForFunction( + () => /sending your edit again in 10 s/.test( + document.getElementById('status').textContent), + null, { timeout: 5000 }).then(() => true, () => false); + const resent = waiting && await A.page.waitForFunction( + () => !document.getElementById('editor').inert, null, + { timeout: 15000 }).then(() => true, () => false); + + await A.page.evaluate(() => window.jam.setFile( + 'hands.gen', `# sent again\n${window.jam.file('hands.gen')}`)); + + if (resent && await B.page.waitForFunction( + () => window.jam.file('hands.gen')?.startsWith('# sent again'), + null, { timeout: 10000 }).then(() => true, () => false)) + ok('an edit refused for a full relay is sent again by itself'); + else + fail(`an edit refused for a full relay: ${waiting ? '' : 'not '}` + + `said, ${resent ? '' : 'not '}sent again -- ` + + await why(A.page)); + + /* An edit too large for the room is refused: the page says so and + stops its document, which would send the edit again at every + reconnect, and stays in the room. */ + const peerOf = () => [...r.peers.values()].find((p) => p.name === 'Ann'); + const was = peerOf()?.ws; + const most = r.ctx.bytesMax; + + r.ctx.bytesMax = r.bytes + 1024; + await A.page.evaluate(() => window.jam.setFile( + 'hands.gen', + `# ${'x'.repeat(4096)}\n${window.jam.file('hands.gen')}`)); + + const told = await A.page.waitForFunction( + () => /too large for the room and was not kept/.test( + document.getElementById('status').textContent), + null, { timeout: 10000 }).then(() => true, () => false); + + await new Promise((resolve) => setTimeout(resolve, 3000)); + r.ctx.bytesMax = most; + + if (told && peerOf()?.ws === was && r.docConns.size === 1 && + !r.doc.getMap('files').get('hands.gen').toString() + .startsWith('# xxx') && + await A.page.evaluate(() => document.getElementById('editor').inert)) + ok('an edit too large for the room is said, and stops the page\'s ' + + 'document, and the page stays in the room'); + else + fail(`an edit too large for the room: ${told ? 'said' : 'not said'}` + + `, the room socket ${peerOf()?.ws === was ? '' : 'not '}kept, ` + + `${r.docConns.size} document sockets -- ${await why(A.page)}`); + await A.page.evaluate(() => localStorage.clear()); } @@ -1978,6 +2105,54 @@ async function accountsTogether (pages) * answers whatever asks while its presence is simulated, the login form's * autofill offer included; the button is tried with autofill taken away. */ +/* A page the relay keeps turning away for too many joins from its + * address -- a class behind one, after a relay restart -- tries again + * when it says, as long as it says, and spends none of its tries on it. + */ +async function joinsPaced (pages) +{ + const [A] = pages; + const first = await relay({ port: 0, host: '127.0.0.1', tree: top }); + const port = first.address().port; + let hellos = 0; + + await A.page.goto(`${url}&room=jampaced&name=Pat&relay=` + + `ws://127.0.0.1:${port}`); + await A.page.waitForFunction( + () => !document.getElementById('roompanel').hidden, null, + { timeout: 15000 }); + first.shutdown(); + await new Promise((r) => setTimeout(r, 100)); + + const pacer = new WebSocketServer({ port, host: '127.0.0.1' }); + + pacer.on('connection', (ws) => ws.on('message', () => + { + hellos++; + ws.send(JSON.stringify({ type: 'error', why: 'flood', retryMs: 300, + text: 'too many joins from your address; ' + + 'try again in 1 s' })); + ws.close(); + })); + + const t0 = performance.now(); + + while (hellos < 8 && performance.now() - t0 < 12000) + await new Promise((r) => setTimeout(r, 100)); + + const s = (performance.now() - t0) / 1000; + + if (hellos >= 8) + ok(`a page refused for too many joins tries again as the relay ` + + `says: 8 times in ${s.toFixed(1)} s`); + else + fail(`a page refused for too many joins tried ${hellos} times in ` + + `${s.toFixed(1)} s -- ${await why(A.page)}`); + + pacer.close(); + await A.page.goto('about:blank'); +} + async function passkeysTogether (browser) { const context = await browser.newContext(); @@ -2923,6 +3098,7 @@ try /* ---- as an account, and a guest ---- */ await accountsTogether(pages); + await joinsPaced(pages); await passkeysTogether(browsers[0]); for (const e of errors) diff --git a/wasm/web/relay.mjs b/wasm/web/relay.mjs index 9d38cb8f..1fb92726 100644 --- a/wasm/web/relay.mjs +++ b/wasm/web/relay.mjs @@ -62,9 +62,9 @@ * hands out a ticket for it, good for one room for a few minutes, and the * relay opens no document socket without one. * - * A room is made when the first peer arrives and seeded with a shipped - * piece -- the .gen, and every .dsp it names, from the tree -- and kept - * for an hour after the last one leaves. A peer can have it seeded again + * A room is made when the first peer's hello is welcomed and seeded with a + * shipped piece -- the .gen, and every .dsp it names, from the tree -- and + * kept for a while after the last one leaves. A peer can have it seeded again * with another (`switch'). Rooms are not persisted; accounts are, in one * SQLite file (DB=..., beside the relay by default). */ @@ -75,6 +75,8 @@ import http from 'node:http'; import path from 'node:path'; import { fileURLToPath, pathToFileURL } from 'node:url'; import { monitorEventLoopDelay } from 'node:perf_hooks'; +import v8 from 'node:v8'; +import vm from 'node:vm'; import { WebSocketServer } from 'ws'; import * as Y from 'yjs'; @@ -84,8 +86,9 @@ import * as decoding from 'lib0/decoding'; import * as encoding from 'lib0/encoding'; import { ACCOUNT_API, isOrigin, normalizeName, shownName } from './account.js'; -import { AccountStore, Accounts, ADMIN_USAGE, accountRoutes, - runAdmin } from './accounts.mjs'; +import { AccountStore, Accounts, ADMIN_USAGE, Bucket as TokenBucket, + TieredLimit, accountRoutes, clientOf, runAdmin } + from './accounts.mjs'; import { Passkeys, passkeyConfig } from './passkeys.mjs'; import { RELAY, TRANSPORT_LEAD } from './commands.js'; import { DEFAULT_PIECE, dspNames, files, hashOfFiles, hasSeen, meta, @@ -102,8 +105,10 @@ const MSG_AWARENESS = 1; const CURSOR_COLOR = /^hsl\(\d{1,3} 70% 45%( \/ 0\.25)?\)$/; /* How long a start waits for the relay's copy of the document to reach - the revision it names before keeping what is there. */ + the revision it names before keeping what is there, and how often at + most a room looks (snapshotAt). */ const SNAPSHOT_WAIT = 10 * 1000; +const SNAPSHOT_EVERY_MS = 100; /* How long a switch made while the room plays waits for another before the relay plays it: switches made together play the last of them once, @@ -117,19 +122,125 @@ const LOG_MAX = 200000; /* And the most it keeps in bytes, of JSON, and of one command: an edit carries a piece's texts, and nothing a page sends is longer than - that. */ -const LOG_BYTES_MAX = 32 * 1024 * 1024; + that. A run keeps its commands as their JSON, which is what a + catch-up sends, and is charged that: parsed, a command of empty + objects held twenty times its JSON. A command that would take the room or the rooms past what + they may be charged overflows the run, and so does an edit to the + document that would: the document is the one to keep. The most is + half what a room may be charged (ROOM_MAX_BYTES). */ +const LOG_BYTES_MAX = 8 * 1024 * 1024; const LOG_ENTRY_MAX = 512 * 1024; +/* How a catch-up ends, after the run's commands. */ +const CAUGHT_UP = Buffer.from('],"overflowed":false}'); +const OVERFLOWED = Buffer.from('],"overflowed":true}'); + +/* The most peers in a room. What the relay forwards for a room whose + mesh has failed grows as the square of its peers: 192 players at 10 + knobs a second, all through the relay, is where one core falls behind + (an event-loop p99 past 20 ms), and 64 is a ninth of that, which leaves + the core to the other rooms. There are 16 seats. */ +const PEERS_PER_ROOM_MAX = 64; + /* Awareness clients one document socket may speak for, and a room may hold: a page is one, and a reconnect briefly has the old socket's too. */ const CLIENTS_PER_SOCKET = 2; const CLIENTS_PER_ROOM = 256; -/* How long an empty room is kept, and how often that is looked at. */ -const EMPTY_FOR = 60 * 60 * 1000; -const SWEEP_EVERY = 60 * 1000; +/* And the longest state one may have, as JSON: a page's is a name, two + colors and a selection, about 400 bytes. Each is kept for the room as + those fields alone (vouched) and handed to every joiner, and is no + charge of the room's. */ +const CLIENT_STATE_MAX = 1024; + +/* How long an empty room is kept, and how often that is looked at. A + page that loses its room socket joins again by itself for about two + minutes (jam.js, REJOIN_TRIES), and Rejoin after that brings its own + document to a room the relay no longer has; an empty room is kept for + five times that. */ +const EMPTY_FOR_MS = 10 * 60 * 1000; +const SWEEP_EVERY_MS = 60 * 1000; + +/* Rooms: at most this many, and this much heap between them by what each + is charged -- a base, and what its document holds, measured as it is + after every transaction (Room.watch). Past either, the rooms empty + longest go, as many as make room for a new one or an edit, and with + too few to let go it is refused. Measured, an empty room holds 6 KB + of heap and is charged 8 KB, one seeded with airports.gen 18 KB and + 26 KB, and with sunrise.gen 96 KB and 165 KB. The budget is a third + of the 384 MiB heap (docker/compose.yaml): 4096 rooms of the default + piece, or about 800 of sunrise.gen's. One room may be charged an + eighth of it, and an update that could take it past that is refused + (refusal): twice the largest document frame (DOC_FRAME_MAX), twice + over. */ +const ROOMS_MAX = 4096; +const ROOMS_MAX_BYTES = 128 * 1024 * 1024; +const ROOM_MAX_BYTES = 16 * 1024 * 1024; +const ROOM_BYTES = 8 * 1024; + +/* What a document holds of the heap, as it is charged: every struct + Yjs keeps, whole or a piece split from one, and every client it keeps + them for; every root type and every type an item holds; every value + an item holds and every key of an object or of a map, and two bytes + for every character of a string or a key. Measured on Node 24, a + struct is 270 B (an item split by a deletion, or a keystroke and the + piece of the text it split), a client 490 B and a type 300 to 600 B, + besides the item holding it; a value in an array is 10 B (null) to + 66 B ({}), and a key in a map 190 B. Every one of these is a few + bytes of an update at least, and none is bounded by the update's size + alone: deleting every other character of a text splits it into one + struct for each, at 2 B of update apiece. */ +const STRUCT_BYTES = 320; +const CLIENT_BYTES = 512; +const TYPE_BYTES = 1024; +const VALUE_BYTES = 80; + +/* What Yjs may hold of a room's updates whose structs build on ones its + document does not have. Every struct a page has came through the + relay, or is its own and came on the socket it sends on, so a page + leaves next to nothing held, and only while a reconnect's sync catches + up with an update sent ahead of it. Past this what is held is a + socket's made to be held: it goes, and the socket whose frames added + the most of it is cut, whose provider, if it is a page's, syncs + again. Kept, it would be merged again with every update that adds to + it, and handed to every joiner; while it is not past this it is + charged what it could add to the document (growthOf), or twice its + size if that is more. */ +const PENDING_MAX_BYTES = 64 * 1024; + +/* New rooms one client may make: a burst, then one back every + `refillMs', per address and per IPv6 /48 (accounts.mjs, LIMITS). A + page makes one when it joins a room nobody is in; a class behind one + address making a room each is the burst. */ +const ROOM_LIMITS = [{ burst: 20, refillMs: 3000 }, + { burst: 60, refillMs: 1000 }, + null]; + +/* Hellos one client may have welcomed, into a room old or new: a burst, + then one back every `refillMs', per address and per IPv6 /48. A page + joins once, and again when it loses the room, a second at the soonest + and doubling (jam.js, REJOIN_FIRST_MS); the burst is the fullest room + joined from one address twice over. Each join and leave is told to + everyone in the room, and a client joining and leaving again as fast + as it could did so 2300 times a second. */ +const JOIN_LIMITS = [{ burst: 2 * PEERS_PER_ROOM_MAX, refillMs: 1000 }, + { burst: 8 * PEERS_PER_ROOM_MAX, refillMs: 250 }, + null]; + +/* The bytes of document updates the peers of one address may send + between them, at once and then a second, and of an IPv6 /48: reading + an update is most of what it costs, a 3.8 MB frame of 1.9 M one-byte + structs 260 ms of the loop, and an address has as many peers as it + has joins (JOIN_LIMITS). One peer's worth (DOC_IN_BYTES), and four + for a /48, as for its joins: a page sends its keystrokes, and a whole + document when it brings back a room the relay lost. Past it the + socket is cut, as one past its own. */ +const UPDATE_LIMITS = [{ burst: 16 * 1024 * 1024, + refillMs: 1000 / (2 * 1024 * 1024) }, + { burst: 64 * 1024 * 1024, + refillMs: 1000 / (8 * 1024 * 1024) }, + null]; /* Chat: the longest line, and how many a peer may send at once and then per second. Enough to talk in, and short of what a stuck key or a @@ -138,6 +249,96 @@ const CHAT_MAX = 500; const CHAT_BURST = 5; const CHAT_PER_SECOND = 5; +/* How many of each room message a socket may send at once, and then a + second: what a page sends at most (room.js, mesh.js, jam.js), with room + to spare. A ping goes once a second. A signal is an offer or an answer + and a few candidates for each peer -- twenty, on a host with many + interfaces -- and a page joining the fullest room links to 63 at once. + A gesture is one `relayed' line for every peer the mesh cannot reach, + and a copy for the log: a knob dragged is one per input event, 240 a + second on a fast display, and a MIDI keyboard's notes fewer. + `transport' is a start, a stop or an edit, or a tempo dragged. A seat, + a switch and a catch-up (at Start, and for a Play whose document is + late) are clicks. Anything else, a second hello among it, does nothing + here. Chat is CHAT_BURST's. + * + Past one, the message is dropped, and the page told so at most once a + second for each type; a socket that has had DROPS dropped is cut. A + chat line the page counts is told of each time: the page waits to + hear of every line it sent. */ +const RATES = { + ping: [10, 4], + signal: [2048, 128], + relayed: [1000, 500], + log: [1000, 500], + transport: [500, 250], + seat: [10, 2], + switch: [10, 2], + catchup: [5, 1], + chat: [CHAT_BURST, CHAT_PER_SECOND], + other: [10, 1], +}; +const DROPS = [1000, 100]; + +/* The bytes a socket's gestures and signals may have the relay queue for + others -- each one's size times its recipients -- at once, and then a + second. The most a page makes is its gestures through the relay to a + full room, 2.3 MB a second, and a join's signals, 600 KB; the burst + is half a socket's cap (QUEUED_MAX_BYTES). Past it a message is + dropped as one past its rate is. */ +const FANOUT_BYTES = [8 * 1024 * 1024, 4 * 1024 * 1024]; + +/* And for `transport', apart: a start, a stop and an edit change the + run every peer plays, and one refused for its size would leave the + room playing two. The longest line (ROOM_FRAME_MAX) to the fullest + room fits at once, and one every eight seconds; a tempo dragged to + it is 3 MB a second. The line goes to every peer as one Buffer. */ +const RUN_FANOUT_BYTES = [64 * 1024 * 1024, 8 * 1024 * 1024]; + +/* The bytes a socket may send, at once and then a second, counted before + a frame is read: reading it is most of what a frame costs. A room + socket's longest line is an edit, which carries the piece and its + files (ROOM_FRAME_MAX); a start carries a snapshot (doc.js, SEEN_MAX), + 64 KB; a join is an offer and twenty candidates for each of 63 peers, + about 600 KB. A knob dragged is a `relayed' line naming every peer the + mesh cannot reach and a copy for the log, 240 a second of 150 B each, + and a tempo dragged 250 `transport' lines a second: 125 KB a second + together. The burst is the longest line and a join, twice over; the + rate is the longest line once a second, eight times the drags. A + document socket's largest frames are a first sync, a resync after a + reconnect and a whole piece pasted (DOC_FRAME_MAX), and the cursors it + sends back are 2000 a second of 100 B (DOC_RATE): its burst is four of + the largest, and its rate ten times the cursors. Past it a frame is + dropped unread, as one past its rate, and counts as a drop for every + DROP_BYTES of it: at one apiece, a flood of the longest lines, fewer a + second than DROPS gives back, would never be cut. */ +const ROOM_IN_BYTES = [4 * 1024 * 1024, 1024 * 1024]; +const DOC_IN_BYTES = [16 * 1024 * 1024, 2 * 1024 * 1024]; +const DROP_BYTES = 64 * 1024; + +/* Document frames: a page's edits and cursor, and back again every + awareness update it hears, which in the fullest room of 64 cursors + moving 30 times a second is about 2000 a second. Past it a cursor is + dropped, and a document update cuts the socket, whose provider + reconnects and syncs again: dropped, the update would be missing until + then. */ +const DOC_RATE = [4096, 2048]; + +/* Sync step 1s a peer's document sockets may send between them, at once + and then a second: each is answered with the whole document. A + provider sends one when it connects; a few more is a page's resync or + reconnect, and past them the socket is closed, and its provider + connects again. A peer's and not a socket's, or a socket opened again + would bring as many again. */ +const SYNC_ASKS = [4, 0.1]; + +/* The rates above as relay() takes them: its `limits' over these, and + their `room' over RATES. */ +const LIMITS = { room: RATES, drops: DROPS, doc: DOC_RATE, + syncAsks: SYNC_ASKS, fanout: FANOUT_BYTES, + runFanout: RUN_FANOUT_BYTES, roomIn: ROOM_IN_BYTES, + docIn: DOC_IN_BYTES }; + /* A document ticket's life. A room socket is handed a new one when two fifths of it have gone, so the ticket a page reconnects its document with is never one about to lapse. */ @@ -159,6 +360,58 @@ const DOC_FRAME_MAX = 4 * 1024 * 1024; one, and a reconnect briefly two. */ const DOCS_PER_PEER = 4; +/* What the relay will queue for one socket, and for all of them: past + the first the socket is cut, and past the second the one holding the + most. A page that keeps up has next to nothing queued; 16 MiB is four + of the largest document frames, or sixteen of the longest room lines, + and half a minute of backlog on a 4 Mbit/s link. A catch-up or a sync + of the whole document can be larger than that by itself (LOG_BYTES_MAX), + so the one a socket asked for is held apart from its cap while it + drains, and counted relay-wide once however many sockets share it. The + total is a quarter of the 512 MiB the relay's container is given + (docker/compose.yaml): queued frames are Buffers, outside the heap but + inside that. A cut socket's close is waited for a few seconds, for the + page to read why. */ +const QUEUED_MAX_BYTES = 16 * 1024 * 1024; +const QUEUED_TOTAL_MAX_BYTES = 128 * 1024 * 1024; +const QUEUE_CHECK_MS = 250; +const CORK_MAX_BYTES = 64 * 1024; +const CUT_GRACE_MS = 5 * 1000; + +/* Past this share of the heap V8 gives the relay, or with + MEMORY_MAX_BYTES of the container's memory, heap and Buffers + together, it sheds load until it is under the lower one. What the + rooms are charged is a model of what they hold, and whatever that + misses would otherwise take the process down. The garbage is + collected first, so that only what is held counts. Then the empty + rooms go, and one thing more a look (QUEUE_CHECK_MS), twice as many + at each look it is still over, what costs nobody their session + first: what the largest runs keep for late joiners (overflow), who + then have the document alone. Only then a room, and everyone in + them, told to join again shortly: the one whose charge grew the most + since the last look, or with none grown, the one charged the most + with its sockets' queues. A model that misses what a room holds + charges it no more than the rest, so the count is what catches up + with memory growing faster than one a look frees. It doubles to + 2 ** SHED_DOUBLINGS_MAX, and only at a look that sheds something + with memory no lower than at the last. What a look frees is not seen + at the next, which the Buffers' sweep has not reached yet: the look + after one that shed sheds nothing. */ +const SHED_HIGH = 0.75; +const SHED_LOW = 0.6; +const SHED_DOUBLINGS_MAX = 8; +const SHORT_OF_MEMORY = { why: 'memory', retryMs: 10 * 1000, + text: 'the relay is short of memory; joining ' + + 'again shortly' }; + +/* Room sockets one address may hold open unwelcomed, and how long one + has to say hello. Until it is welcomed a socket is in no room, which + bounds the rest, and its pongs keep it open; a page says hello as its + socket opens. The most is the joins an address may make at once + (JOIN_LIMITS). */ +const UNWELCOMED_MAX = 2 * PEERS_PER_ROOM_MAX; +const HELLO_WITHIN_MS = 5 * 1000; + /* How often every socket is pinged; one that has not answered the last ping by the next is cut. A document socket says nothing while nobody types, and one whose page went away without a close would otherwise @@ -175,6 +428,16 @@ const SESSION_CHECK_MS = 60 * 1000; reported is the time past it. */ const DELAY_RESOLUTION_MS = 10; +/* What a socket cut for falling behind is told. */ +const SLOW = 'the connection fell too far behind the room'; + +/* What an edit refused for its room's size is told, and one refused for + all the rooms'. */ +const TOO_BIG = { why: 'big', text: 'the room\'s document is as large as ' + + 'the relay keeps one' }; +const RELAY_FULL = { why: 'rooms', text: 'the relay holds as much as it ' + + 'can; try again later' }; + /* The room socket's message types, each counted apart for the metrics port. Any other a client sends is counted as `other', so that a made-up type costs one counter and not one more each. */ @@ -185,6 +448,10 @@ const ROOM_TYPES = ['hello', 'welcome', 'joined', 'left', 'seat', 'seats', const here = path.dirname(fileURLToPath(import.meta.url)); +/* Room lines go as text frames, from Buffers: a string is copied into + every socket it is written to, and a Buffer is not. */ +const TEXT = { binary: false }; + /* What the relay has carried since it started, for the metrics port: messages and bytes each way, by room-message type and for document frames, and the catch-ups answered. Integers only, added to on the path @@ -211,11 +478,10 @@ class Traffic c.inBytes += bytes; } - /* `line' went to `n' sockets; its length in bytes. */ - roomOut (type, line, n = 1) + /* `bytes' went to `n' sockets. */ + roomOut (type, bytes, n = 1) { const c = this.room.get(type) ?? this.room.get('other'); - const bytes = Buffer.byteLength(line); c.out += n; c.outBytes += bytes * n; @@ -245,6 +511,20 @@ class Traffic } } +/* `m' as one line to every room socket in `to', counted: the line's + length, or 0 if it went to none. */ +function sendLine (queues, traffic, to, m) +{ + const line = Buffer.from(JSON.stringify(m)); + let n = 0; + + for (const ws of to) + n += queues.send(ws, line, TEXT); + + traffic?.roomOut(m.type, line.length, n); + return n > 0 ? line.length : 0; +} + /* The relay's clock: milliseconds as a double, from the monotonic clock and never from Date.now(), so a step of the system clock does not move a room's origin. */ @@ -311,35 +591,582 @@ export function runKeyOf (start) return start ? `${start.from}#${start.seq}` : null; } +/* A start as the relay keeps it for its run: the fields a page reads of + one (commands.js, Maker.start), each of its type, and none of the rest + a page may have put on it. Parsed, a megabyte of empty objects held + twenty. */ +function startOf ({ at, from, seq, origin, seed, seek, piece }) +{ + const n = (x) => (typeof x === 'number' ? x : undefined); + const s = (x) => (typeof x === 'string' ? x : undefined); + + return { type: 'transport', op: 'start', at: n(at), from, seq: n(seq), + origin: n(origin), seed: n(seed), seek: n(seek), + piece: { hash: s(piece?.hash), seen: s(piece?.seen) } }; +} + +/* A cursor as y-codemirror.next writes and reads one: an anchor and a + head, each a Yjs relative position, or null. */ +function cursorOf (c) +{ + const id = (x) => (Number.isSafeInteger(x?.client) && + Number.isSafeInteger(x?.clock) + ? { client: x.client, clock: x.clock } : null); + const at = (p) => ({ type: id(p.type), item: id(p.item), + tname: typeof p.tname === 'string' ? p.tname : null, + assoc: Number.isSafeInteger(p.assoc) ? p.assoc : 0 }); + + return typeof c?.anchor === 'object' && c.anchor !== null && + typeof c.head === 'object' && c.head !== null + ? { anchor: at(c.anchor), head: at(c.head) } : null; +} + +/* Every socket's send queue, bounded (QUEUED_MAX_BYTES). A socket's + `cut' is how it goes when it is over, and it is terminated without + one. + * + A socket written to is corked until the loop has run every callback + that was ready, so what one turn sends it goes in one write: a write a + frame, each a syscall, was a third of the relay's time in a busy room. + A pong waits as long, which only adds to the round trips a clock + discards (clock.js keeps the shortest). Past CORK_MAX_BYTES it is + written at once: the gain is in small frames, and what waits on a cork + is not yet the kernel's, and would count against the socket's cap. */ +class Queues +{ + /* `sockets' gives every socket's set. */ + constructor (sockets, { socketMax, totalMax }) + { + this.sockets = sockets; + this.socketMax = socketMax; + this.totalMax = totalMax; + this.shared = new Map(); /* held frame -> sockets holding it */ + this.sharedBytes = 0; /* their sizes, added up */ + this.ownBytes = 0; /* the rest, as of the last trim */ + this.corked = new Set(); + this.uncork = () => + { + for (const ws of this.corked) + ws._socket?.uncork(); + + this.corked.clear(); + }; + } + + put (ws, data, opts, cb) + { + if (!this.corked.has(ws)) + { + if (this.corked.size === 0) + setImmediate(this.uncork); + + this.corked.add(ws); + ws._socket.cork(); + ws.corkedBytes = 0; + } + + ws.send(data, opts, cb); + ws.corkedBytes += data.length; + + if (ws.corkedBytes >= CORK_MAX_BYTES) + { + this.corked.delete(ws); + ws._socket.uncork(); + } + } + + /* What `ws' has queued besides the frames it holds. */ + own (ws) + { + return ws.bufferedAmount - (ws.heldBytes ?? 0); + } + + /* `data' onto `ws''s queue unless it has half its cap queued + already, and no cut: for what one peer sends another, which a + flood of it must not cost the other its socket. Whether it was + sent. */ + offer (ws, data, opts = {}) + { + return this.own(ws) + data.length <= this.socketMax / 2 && + this.send(ws, data, opts); + } + + /* `data' onto `ws''s queue, or the socket cut if that puts it over + its cap. Whether it was sent. */ + send (ws, data, opts = {}) + { + if (ws.readyState !== ws.OPEN) + return false; + + if (this.own(ws) + data.length > this.socketMax) + { + this.cut(ws); + return false; + } + + this.put(ws, data, opts); + return true; + } + + /* A message `ws' asked for, in `parts', which may be over the cap and + may be shared with other sockets: held apart from the cap until it + has drained, unless the socket holds another, which it then counts + against. */ + sendHeld (ws, parts, opts = {}) + { + if (ws.readyState !== ws.OPEN) + return false; + + const n = parts.reduce((sum, p) => sum + p.length, 0); + const holding = ws.heldBytes > 0; + + if (holding && this.own(ws) + n > this.socketMax) + { + this.cut(ws); + return false; + } + + if (holding) + { + parts.forEach((p, i) => + this.put(ws, p, { ...opts, fin: i === parts.length - 1 })); + return true; + } + + if (ws.held === undefined) + { + ws.held = new Map(); + ws.heldBytes = 0; + ws.once('close', () => + { + for (const [p, k] of ws.held) + for (let i = 0; i < k; i++) + this.release(ws, p); + }); + } + + parts.forEach((p, i) => + { + ws.held.set(p, (ws.held.get(p) ?? 0) + 1); + ws.heldBytes += p.length; + + if (!this.shared.has(p)) + this.sharedBytes += p.length; + + this.shared.set(p, (this.shared.get(p) ?? 0) + 1); + this.put(ws, p, { ...opts, fin: i === parts.length - 1 }, + () => this.release(ws, p)); + }); + + /* Against the total as the last trim found it, not every + socket's again: a storm of syncs is many of these at once. */ + if (this.sharedBytes + this.ownBytes > this.totalMax) + this.trim(); + + return true; + } + + /* One of `ws''s held frames, gone: written, or the socket closed. + The last tells the socket's `drained', if it has one. */ + release (ws, p) + { + const k = ws.held.get(p); + + if (k === undefined) + return; + + if (k === 1) + ws.held.delete(p); + else + ws.held.set(p, k - 1); + + ws.heldBytes -= p.length; + + const holders = this.shared.get(p) - 1; + + if (holders === 0) + { + this.shared.delete(p); + this.sharedBytes -= p.length; + } + else + this.shared.set(p, holders); + + if (ws.heldBytes === 0) + ws.drained?.(); + } + + cut (ws) + { + if (ws.cut === undefined) + ws.terminate(); + else + ws.cut(); + } + + /* Relay-wide: a held frame counted once, and each socket charged its + share of it. Over the total, the socket holding the most goes now, + without waiting on a close: what it holds is what the relay is + short of. One gone already is not counted, though its queue is + until its close comes, or it would be cut for again. What a socket + has had queued for others is not its to free, and is bounded where + it is sent (FANOUT_BYTES). */ + trim () + { + const charged = []; + let total = 0; + + for (const set of this.sockets()) + for (const ws of set) + { + if (!ws._socket || ws._socket.destroyed) + continue; + + let bytes = this.own(ws); + + for (const [p, k] of ws.held ?? []) + bytes += p.length * k / this.shared.get(p); + + total += bytes; + charged.push([bytes, ws]); + } + + this.ownBytes = total - this.sharedBytes; + + if (total <= this.totalMax) + return; + + charged.sort((a, b) => b[0] - a[0]); + + for (const [bytes, ws] of charged) + { + if (total <= this.totalMax) + break; + + process.stderr.write(`relay: ${total} bytes queued; cutting a ` + + `socket holding ${Math.round(bytes)}\n`); + total -= bytes; + ws.terminate(); + } + } +} + +/* A token bucket on the relay's clock: `burst' at once, then + `perSecond'. */ +class Bucket extends TokenBucket +{ + constructor ([burst, perSecond]) + { + super({ burst, refillMs: 1000 / perSecond }, relayNow); + } +} + +/* What a value Yjs decoded holds, by STRUCT_BYTES' measures. A key + `__proto__' decoded sets an object's prototype, which is then one of + its values. */ +function valueBytes (value) +{ + const stack = [value]; + let bytes = 0; + + while (stack.length > 0) + { + const v = stack.pop(); + + bytes += VALUE_BYTES; + + if (typeof v === 'string') + bytes += 2 * v.length; + else if (v instanceof Uint8Array) + bytes += v.length; + else if (Array.isArray(v)) + for (const x of v) + stack.push(x); + else if (typeof v === 'object' && v !== null) + { + const proto = Object.getPrototypeOf(v); + + if (proto !== Object.prototype && proto !== null) + stack.push(proto); + + for (const k of Object.keys(v)) + { + bytes += 2 * k.length; + stack.push(v[k]); + } + } + } + + return bytes; +} + +/* What an item's content holds besides its struct. A deleted item's is + a length, in the struct. */ +function contentBytes (c) +{ + if (c instanceof Y.ContentString) + return 2 * c.str.length; + + if (c instanceof Y.ContentAny || c instanceof Y.ContentJSON) + return c.arr.reduce((sum, v) => sum + valueBytes(v), 0); + + if (c instanceof Y.ContentBinary) + return VALUE_BYTES + c.content.length; + + if (c instanceof Y.ContentEmbed) + return valueBytes(c.embed); + + if (c instanceof Y.ContentFormat) + return 2 * c.key.length + valueBytes(c.value); + + /* An XML element's name, or a hook's, is its type's. */ + if (c instanceof Y.ContentType) + return TYPE_BYTES + + 2 * (c.type.nodeName ?? c.type.hookName ?? '').length; + + /* A document of its own, which no page makes, and whose guid and + options could be anything: no room takes one. */ + if (c instanceof Y.ContentDoc) + return Infinity; + + return 0; +} + +/* What item `s' holds of its content from offset `from' to `to', which + a deletion of that much of it lets go of, as watch charges it: a text + or a list is split there, and whatever else is whole or not at all. */ +function partBytes (s, from, to) +{ + const { content } = s; + const [a, b] = [Math.max(0, from), Math.min(s.length, to)]; + + return content instanceof Y.ContentString ? 2 * (b - a) + : Array.isArray(content.arr) + ? content.arr.slice(a, b).reduce((sum, v) => sum + valueBytes(v), 0) + : a === 0 && b === s.length ? contentBytes(content) : 0; +} + +/* And with the key it is under in a map. */ +function itemBytes (s) +{ + return !(s instanceof Y.Item) ? 0 + : contentBytes(s.content) + + (s.parentSub === null ? 0 : VALUE_BYTES + 2 * s.parentSub.length); +} + +/* The most `doc' can be charged more for taking `update', read from the + two before it is applied, and 0 or less if it adds nothing: a root + type it names that the document has not got; and every struct of it + the document does not have, an item with what it holds and the + struct either of its origins splits, if it falls inside one of the + document's or the update's, or on one neither has, and the client of + one the document has none of; and every struct a deletion of it may + split, one at either end of its range that falls inside an item not + yet deleted, or past what the document has; less what a deletion + lets go of of the items it falls on that are not deleted yet. */ +function growthOf (doc, update, decode = Y.decodeUpdate) +{ + const { store, share } = doc; + const { structs, ds } = decode(update); + const roots = new Set(); + const clients = new Set(); + const own = new Map(); /* client -> its structs here */ + let bytes = 0; + + for (const s of structs) + { + if (!own.has(s.id.client)) + own.set(s.id.client, []); + + own.get(s.id.client).push(s); + } + + /* Whether an item's origin at `id' splits a struct: one at the last + clock of a struct, or a right origin (`end' false) at the first, + does not, and a document brought back whole has every one there. + One on a struct neither the document nor the update has may. */ + const splits = (id, end) => + { + if (id === null) + return 0; + + const at = id.clock < Y.getState(store, id.client) + ? store.clients.get(id.client) : own.get(id.client); + const last = at?.at(-1); + + if (!(at?.[0].id.clock <= id.clock && + id.clock < last.id.clock + last.length)) + return 1; + + const s = at[Y.findIndexSS(at, id.clock)]; + + return Number(s instanceof Y.Skip || + id.clock !== (end ? s.id.clock + s.length - 1 + : s.id.clock)); + }; + + for (const s of structs) + { + const { client, clock } = s.id; + + if (s instanceof Y.Item && typeof s.parent === 'string' && + !share.has(s.parent) && !roots.has(s.parent)) + { + roots.add(s.parent); + bytes += TYPE_BYTES + 2 * s.parent.length; + } + + if (s instanceof Y.Skip || + clock + s.length <= Y.getState(store, client)) + continue; + + if (!store.clients.has(client) && !clients.has(client)) + { + clients.add(client); + bytes += CLIENT_BYTES; + } + + bytes += s instanceof Y.Item + ? STRUCT_BYTES * (1 + splits(s.origin, true) + + splits(s.rightOrigin, false)) + itemBytes(s) + : STRUCT_BYTES; + } + + for (const [client, deletes] of ds.clients) + { + const has = store.clients.get(client); + const state = Y.getState(store, client); + const cuts = (c) => + { + if (c >= state) + return c > state; + + const s = has[Y.findIndexSS(has, c)]; + + return s.id.clock < c && !s.deleted; + }; + + for (const { clock, len } of deletes) + { + bytes += STRUCT_BYTES * (cuts(clock) + cuts(clock + len)); + + for (let i = clock < state ? Y.findIndexSS(has, clock) : Infinity; + i < (has?.length ?? 0) && has[i].id.clock < clock + len; i++) + if (!has[i].deleted) + bytes -= partBytes(has[i], clock - has[i].id.clock, + clock + len - has[i].id.clock); + } + } + + return bytes; +} + +/* The bytes of updates Yjs holds for `doc' until what they build on + arrives. */ +function pendingOf (doc) +{ + return (doc.store.pendingStructs?.update.length ?? 0) + + (doc.store.pendingDs?.length ?? 0); +} + /* A short random id: for a peer, and for nothing else. */ function newId () { return Math.random().toString(36).slice(2, 8); } -/* One room: a document and the peers in it. `accounts' says who a - session is of, and `tickets' is the relay's, which the document sockets - are let in by. */ +/* Whether a hello `m' is let in, and as whom: `{ account, asked }', an + account or the name a guest asked for, or `{ why, text }', why not. + `accounts' is null on a relay without them, which has no sessions to + know. */ +function admit (m, accounts) +{ + if (m.protocol !== PROTOCOL) + return { text: `protocol ${m.protocol}; this relay speaks ` + + `${PROTOCOL}` }; + + /* A page from before tickets would join the room and wait for ever on + a document socket that is never let in; told now, it says so, and + a reload is the fix. */ + if (m.tickets !== true) + return { why: 'old', + text: 'this page is older than the relay: press Update ' + + 'above, or close thinksynth\'s other tabs and reload' }; + + /* An account plays under its handle. A session the relay no longer + knows is refused rather than made a guest, or somebody would play + the room believing they were logged in. A guest goes by the name + asked for, if that is not an account's. */ + let account; + let asked; + const session = accounts === null ? undefined : m.session; + + try + { + account = session === undefined ? null + : accounts.sessionAccount({ session }); + asked = account?.handle ?? normalizeName(String(m.name ?? '')); + + if (accounts !== null && account === null && asked !== null && + !accounts.nameFree(asked)) + asked = undefined; + } + catch (e) + { + process.stderr.write(`relay: accounts: ${e.message}\n`); + return { why: 'accounts', + text: 'the relay cannot look up accounts right now; try ' + + 'again in a moment' }; + } + + if (session !== undefined && account === null) + return { why: 'session', + text: 'your session has ended; log in again' }; + + if (asked === undefined) + return { why: 'name', + text: `${normalizeName(String(m.name))} is an account's ` + + 'handle; log in, or pick another name' }; + + return { account, asked }; +} + +/* One room: a document and the peers in it. `ctx' is the relay's, the + same for every room: the `tickets' the document sockets are let in by + and their life (`ticketMs'), the `tree' pieces are read from, whether + a hello's session counts (`sessions'), the `traffic' counted and the + send `queues'; the `empties', room -> true, empty longest first, the + rooms' `costs' ({ bytes }), and `spare', whether they may be charged + so much more for this one; the `limits' (LIMITS), the most peers and + bytes a room may have (`peersMax', `bytesMax'), and `updatesIn', + whether a client's address may send so many bytes of updates. */ class Room { - constructor (name, seedWith, tree, - { accounts, tickets, ticketMs, sessions, traffic }) + constructor (name, seedWith, ctx) { this.name = name; - this.tree = tree; - this.accounts = accounts; - this.tickets = tickets; - this.ticketMs = ticketMs; - this.sessions = sessions; /* whether a hello's counts */ - this.traffic = traffic; + this.ctx = ctx; + this.bytes = 0; /* what this one is charged */ + this.lastBytes = 0; /* and was at the backstop's + last look */ + this.grew = 0; /* and the change since */ + this.gone = false; /* destroyed */ + this.structs = 0; /* the document's, as charged */ + this.clients = 0; /* and its clients */ + this.roots = 0; /* and its root types */ + this.pending = 0; /* what Yjs holds pending, as + charged */ + this.held = null; /* and what that was */ this.doc = new Y.Doc(); this.awareness = new awarenessProtocol.Awareness(this.doc); this.docConns = new Set(); /* document sockets */ this.peers = new Map(); /* id -> { ws, name, seat, - account, tickets, docs } */ + account, tickets, docs, + asks, docRates } */ this.seats = new Map(); /* seat -> peer id */ this.playing = null; /* the last transport start */ this.emptySince = relayNow(); + this.ctx.empties.set(this, true); /* The `switched' lines, in the order the switches were made; how many have been made; and the counter of the relay's own Plays, @@ -352,6 +1179,8 @@ class Room the document as that start named it, and every stamped command since, in arrival order. Null while stopped. */ this.run = null; + this.waiter = null; /* the run's snapshotAt */ + this.lookedAt = -Infinity; /* and when it last looked */ /* The awareness protocol's own clients: what to forget when a socket closes. */ @@ -360,8 +1189,11 @@ class Room this.docOwner = new Map(); /* ws -> its room socket's peer */ this.leaving = new Map(); /* room ws -> its leave() */ + this.charge(ROOM_BYTES); + this.watch(); + if (seedWith !== '') - seedFiles(this.doc, seedWith, tree); + seedFiles(this.doc, seedWith, ctx.tree); /* Every update to anyone, as y-websocket does: the document is the one thing here that has to reach every peer. */ @@ -416,32 +1248,276 @@ class Room return this.peers.size === 0 && this.docConns.size === 0; } + /* Every charge to the room, and so to the rooms' total, is made + here, and none once the room is gone: what it was charged went back + when it went, and a switch or a start of its still settling would + otherwise give back what was given already. */ + charge (bytes) + { + if (this.gone) + return; + + this.bytes += bytes; + this.ctx.costs.bytes += bytes; + } + + /* What the document holds, charged as it changes (STRUCT_BYTES): + after a transaction, what every item it added holds, less the + content of every item it deleted, which Yjs lets go of as the + transaction ends; and once it has, every struct, client and root + type there are, an update naming any number of the last. Yjs walks every client at every + transaction, for its state vectors, and so does this. */ + watch () + { + const { store, share } = this.doc; + + this.doc.on('afterTransaction', (tr) => + { + let bytes = 0; + + for (const [client, clock] of tr.afterState) + { + const before = tr.beforeState.get(client) ?? 0; + const structs = store.clients.get(client); + + for (let i = clock > before ? Y.findIndexSS(structs, before) + : structs.length; + i < structs.length; i++) + bytes += itemBytes(structs[i]); + } + + for (const [client, deletes] of tr.deleteSet.clients) + { + const structs = store.clients.get(client); + + for (const { clock, len } of deletes) + for (let i = Y.findIndexSS(structs, clock); + i < structs.length && + structs[i].id.clock < clock + len; + i++) + bytes -= contentBytes(structs[i].content); + } + + this.charge(bytes); + }); + + this.doc.on('afterTransactionCleanup', () => + { + let structs = 0; + let roots = 0; + + for (const s of store.clients.values()) + structs += s.length; + + for (const name of share.keys()) + if (roots++ >= this.roots) + this.charge(TYPE_BYTES + 2 * name.length); + + this.charge(STRUCT_BYTES * (structs - this.structs) + + CLIENT_BYTES * (store.clients.size - this.clients)); + this.structs = structs; + this.clients = store.clients.size; + this.roots = roots; + }); + } + + /* What Yjs holds pending after a frame from `ws', charged as what it + may add to the document when what it builds on comes, or let go of + (PENDING_MAX_BYTES). What it grows by is the socket's that sent the + frame, and over the most, the one that held most of it is cut. */ + settle (ws) + { + const { pendingStructs, pendingDs } = this.doc.store; + const bytes = pendingOf(this.doc); + let pending = bytes; + + if (pendingStructs?.update === this.held?.structs && + pendingDs === this.held?.ds) + return; + + ws.holding = (ws.holding ?? 0) + + Math.max(0, bytes - (this.held?.bytes ?? 0)); + + if (pending > PENDING_MAX_BYTES) + { + this.doc.store.pendingStructs = null; + this.doc.store.pendingDs = null; + pending = 0; + [...this.docConns].reduce((most, d) => + (d.holding ?? 0) > most.holding ? d : most, ws).terminate(); + } + else if (pending > 0) + pending = Math.max( + 2 * pending, + (pendingStructs === null ? 0 + : growthOf(this.doc, pendingStructs.update, + Y.decodeUpdateV2)) + + (pendingDs === null ? 0 + : growthOf(this.doc, pendingDs, Y.decodeUpdateV2))); + + if (pending === 0) + for (const d of this.docConns) + d.holding = 0; + + this.charge(pending - this.pending); + this.pending = pending; + this.held = { structs: this.doc.store.pendingStructs?.update, + ds: this.doc.store.pendingDs, + bytes: pendingOf(this.doc) }; + } + + /* Why the document may not take the update in sync message `bytes', + or null if it may: one is refused that could take the room past + its most, or the rooms past theirs with none left that is empty to + let go (growthOf). One that adds nothing -- a provider's step 2 in + answer to the relay's step 1, from a page with nothing new -- or + frees as much as it adds is taken, so a room at its most is still + joined, read and cut down. One that + fits only without the run's commands overflows the run + (LOG_BYTES_MAX), and one that does not fit even then leaves them + be. Asking whether the rooms may be charged it lets go of the empty + rooms it needs, so it is asked once for each. */ + refusal (bytes) + { + const dec = decoding.createDecoder(bytes); + + decoding.readVarUint(dec); + decoding.readVarUint(dec); + + const cost = growthOf(this.doc, decoding.readVarUint8Array(dec)); + const over = (bytes) => + this.bytes + bytes > this.ctx.bytesMax ? TOO_BIG + : !this.ctx.spare(this, bytes) ? RELAY_FULL : null; + + if (cost <= 0) + return null; + + const why = over(cost); + + if (why !== null && this.run?.bytes > 0 && + over(cost - this.run.bytes) === null) + { + this.overflow(); + return null; + } + + return why; + } + broadcastDoc (bytes) { let n = 0; for (const ws of this.docConns) - if (ws.readyState === ws.OPEN) - { - ws.send(bytes); + if (this.ctx.queues.send(ws, bytes)) n++; - } - this.traffic?.docOut(bytes, n); + this.ctx.traffic?.docOut(bytes, n); } - sendDoc (ws, bytes) + /* `held' for a sync of the whole document, which can be over a + socket's cap. */ + sendDoc (ws, bytes, held = false) { - ws.send(bytes); - this.traffic?.docOut(bytes); + if (held ? this.ctx.queues.sendHeld(ws, [bytes]) + : this.ctx.queues.send(ws, bytes)) + this.ctx.traffic?.docOut(bytes); } /* ---- the run ---- */ - begin (start, files = this.snapshotAt(start.piece ?? {})) + /* A run is charged its start, its commands (record), and the + document as its start named it, once that is known, and that as a + catch-up's JSON once one has been asked for: a copy of every text, + kept for as long as it plays. */ + begin (start, files = null) { this.playing = start; - this.run = { start, log: [], bytes: 0, overflowed: false, files }; + this.end(); + + const run = { start, + files: files ?? this.snapshotAt(start.piece ?? {}), + head: null, chunks: [], pending: [], length: 0, + bytes: 0, kept: valueBytes(start), overflowed: false }; + + this.run = run; + this.charge(run.kept); + run.files.then((snap) => + { + if (this.run !== run || snap === null) + return; + + const bytes = 2 * Object.values(snap.files) + .reduce((n, text) => n + text.length, 0); + + run.kept += bytes; + this.charge(bytes); + }); + } + + /* The run's commands, its document, what they were charged and the + wait for it, gone. */ + end () + { + this.waiter?.(null); + this.charge(-((this.run?.bytes ?? 0) + (this.run?.kept ?? 0))); + this.run = null; + } + + /* The run made one that cannot be caught up with: what it kept is of + no use to anyone and goes, and the head of its catch-up with it, + made again when one is asked for (catchupOf). */ + overflow () + { + const { run } = this; + const head = run.head?.length ?? 0; + + this.charge(-(run.bytes + head)); + run.overflowed = true; + run.bytes = 0; + run.kept -= head; + run.head = null; + run.chunks = []; + run.pending = []; + } + + /* What the run keeps for late joiners that it can do without, which + overflow lets go of. */ + get spareBytes () + { + return (this.run?.bytes ?? 0) + (this.run?.head?.length ?? 0); + } + + /* The catch-up for `run' as it stands, in parts. Commands no + catch-up has had yet are joined into one now, once, and every part + is shared by every catch-up that has it: twenty joiners of a long + run otherwise each cost a copy of all of it. An overflowed run + sends no commands, which the page does not use (jam.js, joinRun), + and the document, which it does (loadPassed). */ + catchupOf (run, files) + { + if (run.head === null) + { + run.head = Buffer.from( + JSON.stringify({ type: 'catchup', start: run.start, files }) + .slice(0, -1) + ',"log":['); + run.kept += run.head.length; + this.charge(run.head.length); + } + + if (run.overflowed) + return [run.head, OVERFLOWED]; + + if (run.pending.length > 0) + { + const s = run.pending.join(','); + + run.chunks.push(Buffer.from(run.chunks.length === 0 ? s + : `,${s}`)); + run.pending = []; + } + + return [run.head, ...run.chunks, CAUGHT_UP]; } /* A switch made while the room plays, played: a start of the relay's @@ -460,42 +1536,39 @@ class Room seek: 0 }; this.begin(start, Promise.resolve({ ...texts, matched: true })); - - const line = JSON.stringify({ type: 'transport', from: RELAY, - data: start }); - let n = 0; - - for (const p of this.peers.values()) - if (p.ws.readyState === p.ws.OPEN) - { - p.ws.send(line); - n++; - } - - this.traffic?.roomOut('transport', line, n); + sendLine(this.ctx.queues, this.ctx.traffic, + [...this.peers.values()].map((p) => p.ws), + { type: 'transport', from: RELAY, data: start }); } /* A stamped command, into the run it was made in. `runKey' is that run's start, as the sender knew it (runKeyOf); a copy that arrives after another Play has begun is the old run's straggler, stamped for - a time in a piece that is no longer playing, and is not kept. - `bytes' is the size of the frame it came in, which holds it: what - it counts against the caps, without stringifying it again on the - path every command takes. */ - record (cmd, runKey, bytes) + a time in a piece that is no longer playing, and is not kept. Past + the run's caps, or past what its room or the rooms may be charged, + it overflows. */ + record (cmd, runKey) { const { run } = this; - if (run === null || runKey !== runKeyOf(run.start)) + if (run === null || runKey !== runKeyOf(run.start) || + run.overflowed) return; - if (run.log.length >= LOG_MAX || bytes > LOG_ENTRY_MAX || - run.bytes + bytes > LOG_BYTES_MAX) - run.overflowed = true; + const json = JSON.stringify(cmd); + const bytes = Buffer.byteLength(json); + + if (run.length >= LOG_MAX || bytes > LOG_ENTRY_MAX || + run.bytes + bytes > LOG_BYTES_MAX || + this.bytes + bytes > this.ctx.bytesMax || + !this.ctx.spare(this, bytes)) + this.overflow(); else { - run.log.push(cmd); + run.pending.push(json); + run.length++; run.bytes += bytes; + this.charge(bytes); } } @@ -503,73 +1576,81 @@ class Room is there, or once an update brings it there. The starter sent its edits on the document socket before its Play on this one, but the two are separate sockets and nothing orders them. Past the wait, - what is here, marked as not what was asked for. */ + what is here, marked as not what was asked for. One start's is + waited for at a time, its run's: the next start, a stop or the + room going resolves it with null, which nobody reads, its run + being gone. A look is a snapshot of the whole document and its + hash, and is made no more often than SNAPSHOT_EVERY_MS in a room, + however many updates and starts come. */ snapshotAt ({ hash, seen } = {}) { const passed = seen === undefined ? null : readSeen(seen); return new Promise((resolve) => { - const timer = setTimeout(() => finish(snapshot(this.doc), false), - SNAPSHOT_WAIT); - let checking = false; + let timer = null; + let looking = false; let again = false; - let done = false; const finish = (snap, matched) => { - if (done) + if (this.waiter !== finish) return; - done = true; + this.waiter = null; clearTimeout(timer); - this.doc.off('update', check); - resolve({ ...snap, matched }); + clearTimeout(lapse); + this.doc.off('update', soon); + resolve(snap === null ? null : { ...snap, matched }); }; - const check = async () => - { - if (checking) - { - again = true; - return; - } + const lapse = setTimeout(() => finish(snapshot(this.doc), false), + SNAPSHOT_WAIT); - checking = true; + const look = async () => + { + timer = null; + looking = true; + this.lookedAt = relayNow(); - do - { - again = false; + /* The snapshot first and its hash after, so what is + handed over is what was hashed: an update landing + during the digest would otherwise be in the one and + not the other. */ + const snap = snapshot(this.doc); + const same = await hashOfFiles(snap.files) === hash; - if (done) - return; + looking = false; - /* The snapshot first and its hash after, so what is - handed over is what was hashed: an update landing - during the digest would otherwise be in the one and - not the other. */ - const snap = snapshot(this.doc); + if (this.waiter !== finish) + return; - if (await hashOfFiles(snap.files) === hash) - { - finish(snap, true); - return; - } + if (same) + finish(snap, true); - /* Gone past it: no update brings it back. */ - if (passed !== null && hasSeen(this.doc, passed)) - { - finish(snap, false); - return; - } + /* Gone past it: no update brings it back. */ + else if (passed !== null && hasSeen(this.doc, passed)) + finish(snap, false); + else if (again) + { + again = false; + soon(); } - while (again); + }; - checking = false; + const soon = () => + { + if (looking) + again = true; + else if (timer === null) + timer = setTimeout(look, Math.max( + 0, this.lookedAt + SNAPSHOT_EVERY_MS - relayNow())); }; - this.doc.on('update', check); - check(); + this.waiter?.(null); + this.waiter = finish; + this.doc.on('update', soon); + soon(); }); } @@ -582,14 +1663,14 @@ class Room const ticket = `t_${crypto.randomBytes(16).toString('hex')}`; for (const t of peer.tickets) - if (!(this.tickets.get(t)?.until > now)) + if (!(this.ctx.tickets.get(t)?.until > now)) { - this.tickets.delete(t); + this.ctx.tickets.delete(t); peer.tickets.delete(t); } - this.tickets.set(ticket, { room: this, peer, - until: now + this.ticketMs }); + this.ctx.tickets.set(ticket, { room: this, peer, + until: now + this.ctx.ticketMs }); peer.tickets.add(ticket); return ticket; } @@ -604,29 +1685,40 @@ class Room for (const p of this.peers.values()) if (p.account !== null && ends(p.account) && p.ws.readyState === p.ws.OPEN) - { - const line = JSON.stringify({ type: 'error', why: 'session', - text: why }); + this.drop(p.ws, 'session', why); + } - p.ws.send(line); - this.traffic?.roomOut('error', line); - p.ws.close(); - this.leaving.get(p.ws)(); - } + /* A room socket told why and closed, and out of the room now. Its + last word goes past its cap, and a client that does not read it + is cut in a few seconds. */ + drop (ws, why, text) + { + this.tell(ws, { why, text }); + this.leaving.get(ws)?.(); + } + + tell (ws, error) + { + const line = Buffer.from(JSON.stringify({ type: 'error', ...error })); + + ws.send(line, TEXT); + this.ctx.traffic?.roomOut('error', line.length); + ws.close(); + setTimeout(() => ws.terminate(), CUT_GRACE_MS).unref(); } /* Whether a peer is an account, as the room is told: null on a relay without accounts, where nobody is a guest for not being one. */ isAccount (peer) { - return this.sessions ? peer.account !== null : null; + return this.ctx.sessions ? peer.account !== null : null; } /* A peer's way into the document, gone. */ revoke (peer) { for (const t of peer.tickets) - this.tickets.delete(t); + this.ctx.tickets.delete(t); for (const d of peer.docs) d.terminate(); @@ -658,7 +1750,13 @@ class Room { const client = decoding.readVarUint(dec); const clock = decoding.readVarUint(dec); - let state = JSON.parse(decoding.readVarString(dec)); + const json = decoding.readVarString(dec); + + if (json.length > CLIENT_STATE_MAX) + throw new Error(`an awareness state of ${json.length} ` + + 'characters'); + + let state = JSON.parse(json); const holder = this.clientSocket.get(client); if (holder !== undefined && holder !== ws && @@ -691,26 +1789,27 @@ class Room throw new Error('more awareness clients than a page has'); } - /* A state, unless it is the one saying the client is gone: and - every one with the relay's name on it, whatever the page put - there or left out, so that none goes nameless and unmarked. */ + /* A state, unless it is the one saying the client is gone, as + the fields a page's has and no others: parsed, 4 KiB of + empty objects held 85 KiB. Every one has the relay's name + on it, whatever the page put there or left out, so that + none goes nameless and unmarked. */ if (state !== null) { - if (typeof state !== 'object' || Array.isArray(state)) - state = {}; - - state.user = { - ...(typeof state.user === 'object' ? state.user : {}), - name: shownName({ name: owner.name, - account: this.isAccount(owner) }), - account: this.isAccount(owner), + const user = state.user ?? {}; + + state = { + user: { name: shownName({ name: owner.name, + account: this.isAccount(owner) }), + account: this.isAccount(owner) }, + cursor: cursorOf(state.cursor), }; /* The colors end up in other pages' style attributes, so only the shape editor.js's colourOf makes goes through. */ for (const k of ['color', 'colorLight']) - if (!CURSOR_COLOR.test(String(state.user[k]))) - delete state.user[k]; + if (CURSOR_COLOR.test(String(user[k]))) + state.user[k] = user[k]; } kept.push([client, clock, state]); @@ -736,6 +1835,20 @@ class Room this.docOwner.set(ws, owner); ws.binaryType = 'arraybuffer'; + /* The peer's, across its document sockets, as its sync asks + are: a socket opened again would bring them all back. */ + const { frames, inBytes, drops } = owner.docRates; + + /* Said on the room socket, which a page shows: a document socket + has nothing to say it in. */ + ws.cut = () => + { + if (owner.ws.readyState === owner.ws.OPEN) + this.drop(owner.ws, 'slow', SLOW); + + ws.terminate(); + }; + ws.on('message', (data) => { let bytes; @@ -751,7 +1864,35 @@ class Room else bytes = new Uint8Array(data); - this.traffic?.docIn(bytes); + this.ctx.traffic?.docIn(bytes); + + if (!inBytes.take(bytes.length) || !frames.take()) + { + if (bytes[0] !== MSG_AWARENESS || + !drops.take(Math.ceil(bytes.length / DROP_BYTES))) + ws.terminate(); + + return; + } + + /* A frame is told apart here by its first bytes, its type + and a sync message's step, which y-websocket writes as one + byte each. Written in more, a step 1 here would be an + update to the decoder, taken without being read for what + it adds. */ + if (bytes[0] > 0x7f || (bytes[0] === MSG_SYNC && bytes[1] > 0x7f)) + { + ws.close(); + return; + } + + if (bytes[0] === MSG_SYNC && + bytes[1] === syncProtocol.messageYjsSyncStep1 && + !owner.asks.take()) + { + ws.close(); + return; + } /* The bytes are untrusted: an empty, truncated or garbage frame throws out of the decoder or out of Yjs. One bad @@ -764,15 +1905,42 @@ class Room switch (decoding.readVarUint(dec)) { case MSG_SYNC: + { + if (bytes[1] !== syncProtocol.messageYjsSyncStep1 && + !this.ctx.updatesIn(owner.client, bytes.length)) + { + ws.terminate(); + break; + } + + const no = + bytes[1] === syncProtocol.messageYjsSyncStep1 + ? null : this.refusal(bytes); + + /* Said on the room socket, and the document socket + closed, whose provider would only send it again; + the room socket stays, and its page with it. */ + if (no !== null) + { + sendLine(this.ctx.queues, this.ctx.traffic, + [owner.ws], + { type: 'refused', of: 'edit', ...no }); + ws.terminate(); + break; + } + encoding.writeVarUint(enc, MSG_SYNC); syncProtocol.readSyncMessage(dec, enc, this.doc, ws); + this.settle(ws); /* A reply only when there is one: step 2 in answer to step 1, or nothing in answer to an update. */ if (encoding.length(enc) > 1) - this.sendDoc(ws, encoding.toUint8Array(enc)); + this.sendDoc(ws, encoding.toUint8Array(enc), + true); break; + } case MSG_AWARENESS: awarenessProtocol.applyAwarenessUpdate( @@ -827,60 +1995,100 @@ class Room /* ---- the room socket ---- */ - attachRoom (ws) + /* A room socket whose hello `m' the relay has let in (admit), as + `account', or a guest by the name `asked', from `client'. */ + attachRoom (ws, m, { account, asked }, client) { let id = null; let ticketing = null; - let chatTokens = CHAT_BURST; - let chatAt = relayNow(); - - /* The length of the line sent, in bytes; 0 if it was not. */ - const send = (m) => - { - if (ws.readyState !== ws.OPEN) - return 0; - - const line = JSON.stringify(m); - - ws.send(line); - return this.traffic?.roomOut(m.type, line) ?? 0; - }; + const buckets = new Map(); /* RATES' type -> its Bucket */ + const told = new Map(); /* type -> when it was told */ + const drops = new Bucket(this.ctx.limits.drops); + const inBytes = new Bucket(this.ctx.limits.roomIn); + let answering = false; /* a catch-up awaits its run */ + let again = false; /* and one waits for another + to drain */ + + ws.cut = () => this.drop(ws, 'slow', SLOW); + const fanout = new Bucket(this.ctx.limits.fanout); + const runFanout = new Bucket(this.ctx.limits.runFanout); - /* To the peers `to' names -- one id, or a list of them -- or, - with no `to' at all, to everyone in the room but this one. + const send = (m) => + sendLine(this.ctx.queues, this.ctx.traffic, [ws], m); + + /* To the peers `to' names -- one id, or a list of them, each + once and no more than a room holds -- or, with no `to' at all, + to everyone in the room but this one. Whether it went: what + this socket says for others to hear comes out of the bucket + `said' (FANOUT_BYTES, RUN_FANOUT_BYTES), and past it does not; + what the relay says of it -- a join, a seat, a leave -- goes + regardless. * Serialised once, however many sockets it goes to: a command to a room of eight otherwise costs eight identical stringify calls on the one path that is meant to be cheap, because nothing here reads what it forwards. An id nobody is on is not an error: a - peer that left is a peer that left. */ - const toPeers = (m, to) => + peer that left is a peer that left. A gesture or a signal is + not queued for a peer already half way to its cap (offer). */ + const toPeers = (m, to, said = null) => { - const s = JSON.stringify(m); + const s = Buffer.from(JSON.stringify(m)); + const ids = to === undefined ? null + : new Set((Array.isArray(to) ? to.slice(0, this.ctx.peersMax) + : [to]).map(String)); + const lossy = m.type === 'relayed' || m.type === 'signal'; let n = 0; const put = (p) => { - if (p !== undefined && p.ws.readyState === p.ws.OPEN) - { - p.ws.send(s); + if (p !== undefined && + (lossy ? this.ctx.queues.offer(p.ws, s, TEXT) + : this.ctx.queues.send(p.ws, s, TEXT))) n++; - } }; - if (to === undefined) + if (said !== null && + !said.take(s.length * (ids?.size ?? this.peers.size - 1))) + return false; + + if (ids === null) { for (const [pid, p] of this.peers) if (pid !== id) put(p); } else - for (const pid of Array.isArray(to) ? to : [to]) - put(this.peers.get(String(pid))); + for (const pid of ids) + put(this.peers.get(pid)); - this.traffic?.roomOut(m.type, s, n); + this.ctx.traffic?.roomOut(m.type, s.length, n); + return true; }; - const others = (m) => toPeers(m); + /* `m', of RATES' `type', dropped, as `n' drops. */ + const refuse = (type, m, n = 1) => + { + const now = relayNow(); + + if (!drops.take(n)) + this.drop(ws, 'flood', 'this page sent more than the ' + + 'relay takes; rejoin'); + else if ((type === 'chat' && Number.isSafeInteger(m.n)) || + !(now - told.get(type) < 1000)) + { + /* With what the page matches it to: the piece it asked + for, its chat line, or what its transport command was. */ + told.set(type, now); + send({ type: 'refused', of: type, why: 'too fast; slow down', + ...(type === 'switch' && + { piece: typeof m.piece === 'string' + ? m.piece : '' }), + ...(type === 'chat' && Number.isSafeInteger(m.n) && + { n: m.n }), + ...(type === 'transport' && + typeof (m.data?.op ?? m.data?.type) === 'string' && + { op: m.data.op ?? m.data.type }) }); + } + }; const seatMap = () => { @@ -892,6 +2100,67 @@ class Room return out; }; + /* A late joiner, ready to play: the run as the relay has it, or + `start: null' when nothing is playing. A Play while the snapshot + is awaited makes the answer that run, which is what the joiner + should hear, and an answer still being made answers this ask as + well: a page takes one answer for every catch-up it is waiting + on (room.js). One made already, and still draining, may be for + a run since replaced: this ask is answered again once it has + drained, and asks meanwhile with it. */ + const answer = () => + { + const run = this.run; + + if (run === null) + { + answering = false; + const bytes = send({ type: 'catchup', start: null }); + + this.ctx.traffic?.catchup(bytes); + ws.drained(); + return; + } + + run.files.then((files) => + { + if (this.run !== run) + answer(); + else + { + const parts = this.catchupOf(run, files); + const bytes = parts.reduce((sum, p) => sum + p.length, 0); + + answering = false; + + if (!this.ctx.queues.sendHeld(ws, parts, TEXT)) + return; + + this.ctx.traffic?.roomOut('catchup', bytes); + this.ctx.traffic?.catchup(bytes); + } + }); + }; + const catchup = () => + { + if (answering) + return; + + again = ws.heldBytes > 0; + + if (!again) + { + answering = true; + answer(); + } + }; + + ws.drained = () => + { + if (again && ws.readyState === ws.OPEN) + catchup(); + }; + const heard = (data) => { /* Closing is final: after a refused hello, or a session ended @@ -900,6 +2169,13 @@ class Room if (ws.readyState !== ws.OPEN) return; + if (!inBytes.take(data.length)) + { + this.ctx.traffic?.roomIn('other', data.length); + refuse('bytes', undefined, Math.ceil(data.length / DROP_BYTES)); + return; + } + let m; try @@ -911,7 +2187,7 @@ class Room m = undefined; } - this.traffic?.roomIn(typeof m?.type === 'string' ? m.type + this.ctx.traffic?.roomIn(typeof m?.type === 'string' ? m.type : 'other', data.length); @@ -925,133 +2201,25 @@ class Room typeof m.type !== 'string') return; - /* The first message has to be a hello, and only the first. */ - if (id === null) - { - if (m.type !== 'hello') - { - send({ type: 'error', text: 'hello first' }); - return; - } - - if (m.protocol !== PROTOCOL) - { - send({ type: 'error', - text: `protocol ${m.protocol}; this relay ` + - `speaks ${PROTOCOL}` }); - ws.close(); - return; - } - - /* A page from before tickets would join the room and - wait for ever on a document socket that is never let - in; told now, it says so, and a reload is the fix. */ - if (m.tickets !== true) - { - send({ type: 'error', why: 'old', - text: 'this page is older than the relay: press ' + - 'Update above, or close thinksynth\'s ' + - 'other tabs and reload' }); - ws.close(); - return; - } - - /* An account plays under its handle. A session the relay - no longer knows is refused rather than made a guest, or - somebody would play the room believing they were logged - in. A guest goes by the name asked for, if that is not - an account's. */ - let account; - let asked; + const type = Object.hasOwn(this.ctx.limits.room, m.type) ? m.type + : 'other'; - /* A relay without accounts has no sessions to know. */ - const session = this.sessions ? m.session : undefined; + if (!buckets.has(type)) + buckets.set(type, new Bucket(this.ctx.limits.room[type])); - try - { - account = session === undefined ? null - : this.accounts.sessionAccount({ session }); - asked = account?.handle ?? - normalizeName(String(m.name ?? '')); - - if (this.sessions && account === null && - asked !== null && !this.accounts.nameFree(asked)) - asked = undefined; - } - catch (e) - { - process.stderr.write(`relay: accounts: ${e.message}\n`); - send({ type: 'error', why: 'accounts', - text: 'the relay cannot look up accounts right ' + - 'now; try again in a moment' }); - ws.close(); - return; - } + let forward = true; - if (session !== undefined && account === null) - { - send({ type: 'error', why: 'session', - text: 'your session has ended; log in again' }); - ws.close(); - return; - } + if (!buckets.get(type).take()) + { + refuse(type, m); - if (asked === undefined) - { - send({ type: 'error', why: 'name', - text: `${normalizeName(String(m.name))} is an ` + - 'account\'s handle; log in, or pick ' + - 'another name' }); - ws.close(); + /* Played all the same if the mesh carried it (transport), + unless the refusal cut the socket. */ + if (type !== 'transport' || m.data?.type !== 'transport' || + ws.readyState !== ws.OPEN) return; - } - - /* A page joining again names the last ticket it was - handed, which only its own room socket ever was: the - peer it was, on a socket a dropped network left open - until the heartbeat finds it, goes now, and its seat - and cursor with it. */ - const was = this.tickets.get(m.was); - - if (was?.room === this && was.until > relayNow() && - this.leaving.has(was.peer.ws)) - { - was.peer.ws.terminate(); - this.leaving.get(was.peer.ws)(); - } - id = newId(); - - while (this.peers.has(id) || id === RELAY) - id = newId(); - - const name = asked ?? id; - const peer = { ws, name, seat: null, account, - tickets: new Set(), docs: new Set() }; - - this.peers.set(id, peer); - ticketing = setInterval( - () => send({ type: 'ticket', ticket: this.issue(peer) }), - this.ticketMs * 2 / 5); - - send({ - type: 'welcome', - peer: id, - identity: { name, account: this.isAccount(peer) }, - ticket: this.issue(peer), - peers: [...this.peers].map(([pid, p]) => - ({ peer: pid, name: p.name, seat: p.seat, - account: this.isAccount(p) })), - seats: seatMap(), - piece: this.doc.getMap('meta').get('piece') ?? null, - playing: this.playing, - /* What a page cannot assume of an older relay. */ - features: ['switch'], - }); - - others({ type: 'joined', peer: id, name, - account: this.isAccount(peer) }); - return; + forward = false; } const me = this.peers.get(id); @@ -1085,7 +2253,7 @@ class Room const reply = { type: 'seats', seats: seatMap() }; send(reply); - others(reply); + toPeers(reply); break; } @@ -1097,8 +2265,9 @@ class Room the String, which turns a missing `to' into an id nobody has rather than into the whole room. */ case 'signal': - toPeers({ type: 'signal', from: id, data: m.data }, - String(m.to)); + if (!toPeers({ type: 'signal', from: id, data: m.data }, + String(m.to), fanout)) + refuse(type, m); break; /* A gesture the mesh could not carry: forwarded unread @@ -1107,8 +2276,9 @@ class Room at a time and sends one message for all of them. With no `to' at all it goes to the room. */ case 'relayed': - toPeers({ type: 'relayed', from: id, data: m.data }, - m.to); + if (!toPeers({ type: 'relayed', from: id, data: m.data }, + m.to, fanout)) + refuse(type, m); break; /* A transport start goes here as well as over the mesh: @@ -1134,6 +2304,26 @@ class Room break; } + if (m.data.op === 'start') + m.data = startOf(m.data); + + /* A start, a stop or a tempo goes over the mesh as + well (jam.js, send), and changes the run here + whether or not it goes on from here: the run is the + one the peers play. An edit goes from here alone, + and changes it only if it goes. */ + if (forward && !toPeers({ type: 'transport', from: id, + data: m.data }, undefined, + runFanout)) + { + refuse(type, m); + forward = false; + } + + if ((!forward && m.data.type !== 'transport') || + ws.readyState !== ws.OPEN) + break; + /* A stop names the run it stops, and one that has been replaced since -- by a switch's Play crossing it -- stops nothing here, as on the pages (jam.js). */ @@ -1145,13 +2335,12 @@ class Room m.data.run === runKeyOf(this.playing)) { this.playing = null; - this.run = null; + this.end(); } } else - this.record(m.data, m.run, data.length); + this.record(m.data, m.run); - others({ type: 'transport', from: id, data: m.data }); break; } @@ -1160,8 +2349,7 @@ class Room others have it. */ case 'log': if (typeof m.data === 'object' && m.data !== null) - this.record({ ...m.data, from: id }, m.run, - data.length); + this.record({ ...m.data, from: id }, m.run); break; /* A line of text, to everyone in the room and back to its @@ -1174,23 +2362,15 @@ class Room { const text = typeof m.text === 'string' ? m.text.trim() : ''; - const now = relayNow(); - - chatTokens = Math.min(CHAT_BURST, chatTokens + - (now - chatAt) * CHAT_PER_SECOND / - 1000); - chatAt = now; - const n = Number.isSafeInteger(m.n) ? m.n : undefined; /* Format characters alone are a line that shows as nothing, or reorders the lines around it. */ const why = m.channel !== 'stage' ? 'no such channel' - : text.replace(/\p{Cf}/gu, '').trim() === '' - ? 'nothing to send' : text.length > CHAT_MAX ? `longer than ${CHAT_MAX} characters` - : chatTokens < 1 ? 'too fast; wait a moment' + : text.replace(/\p{Cf}/gu, '').trim() === '' + ? 'nothing to send' : null; if (why !== null) @@ -1199,8 +2379,6 @@ class Room break; } - chatTokens--; - const line = { type: 'chat', channel: m.channel, from: id, name: me.name, account: this.isAccount(me), text }; @@ -1210,7 +2388,7 @@ class Room line.bar = m.bar; send({ ...line, n }); - others(line); + toPeers(line); break; } @@ -1220,14 +2398,27 @@ class Room merged into a document holding both pieces. Everyone is told who switched, and the revision the document is at after it. While the room plays, the relay plays the last - of the switches made together (playSwitch). */ + of the switches made together (playSwitch). A switch + leaves what it replaced in the document as deletions, + and is not an update read before it is taken: a room + past its most, or a relay past its budget, makes none, + and one goes past them by a piece at most. */ case 'switch': { const piece = typeof m.piece === 'string' ? m.piece : ''; + const full = this.bytes > this.ctx.bytesMax ? TOO_BIG + : !this.ctx.spare(this, 0) ? RELAY_FULL : null; + + if (full !== null) + { + send({ type: 'refused', of: 'switch', + why: full.text, piece }); + break; + } if (piece !== path.basename(piece) || !piece.endsWith('.gen') || - !seedFiles(this.doc, piece, this.tree)) + !seedFiles(this.doc, piece, this.ctx.tree)) { send({ type: 'refused', of: 'switch', why: 'no such piece', piece }); @@ -1253,7 +2444,7 @@ class Room piece, hash }; send(line); - others(line); + toPeers(line); if (playingThen === null) return; @@ -1261,52 +2452,16 @@ class Room await new Promise((r) => setTimeout(r, SWITCH_GATHER_MS)); - if (this.playing === playingThen && + if (!this.gone && this.playing === playingThen && made === this.switches) this.playSwitch(hash, seen, texts); }); break; } - /* A late joiner, ready to play: the run as the relay has - it, or `start: null' when nothing is playing. A Play - while the snapshot is awaited makes the answer that - run, which is what the joiner should hear. */ case 'catchup': - { - const answer = () => - { - const run = this.run; - - if (run === null) - { - const bytes = send({ type: 'catchup', - start: null }); - - this.traffic?.catchup(bytes); - return; - } - - run.files.then((files) => - { - if (this.run !== run) - answer(); - else - { - const bytes = send({ type: 'catchup', - start: run.start, files, - log: run.log, - overflowed: - run.overflowed }); - - this.traffic?.catchup(bytes); - } - }); - }; - - answer(); + catchup(); break; - } } }; @@ -1344,8 +2499,8 @@ class Room this.seats.delete(me.seat); this.peers.delete(id); - others({ type: 'left', peer: id }); - others({ type: 'seats', seats: seatMap() }); + toPeers({ type: 'left', peer: id }); + toPeers({ type: 'seats', seats: seatMap() }); this.touch(); }; @@ -1356,27 +2511,107 @@ class Room }); ws.on('error', () => ws.close()); this.leaving.set(ws, leave); + + /* The peer it was, on a socket a dropped network left open until + the heartbeat finds it, goes now, and its seat and cursor with + it. */ + const was = this.rejoining(m.was); + + if (was !== null) + { + was.terminate(); + this.leaving.get(was)(); + } + + id = newId(); + + while (this.peers.has(id) || id === RELAY) + id = newId(); + + const name = asked ?? id; + const peer = { ws, name, seat: null, account, client, + tickets: new Set(), docs: new Set(), + asks: new Bucket(this.ctx.limits.syncAsks), + docRates: { frames: new Bucket(this.ctx.limits.doc), + inBytes: new Bucket(this.ctx.limits.docIn), + drops: new Bucket( + this.ctx.limits.drops) } }; + + this.peers.set(id, peer); + this.ctx.empties.delete(this); + ticketing = setInterval( + () => send({ type: 'ticket', ticket: this.issue(peer) }), + this.ctx.ticketMs * 2 / 5); + + send({ + type: 'welcome', + peer: id, + identity: { name, account: this.isAccount(peer) }, + ticket: this.issue(peer), + peers: [...this.peers].map(([pid, p]) => + ({ peer: pid, name: p.name, seat: p.seat, + account: this.isAccount(p) })), + seats: seatMap(), + piece: this.doc.getMap('meta').get('piece') ?? null, + playing: this.playing, + /* What a page cannot assume of an older relay. */ + features: ['switch'], + }); + + toPeers({ type: 'joined', peer: id, name, + account: this.isAccount(peer) }); + } + + /* The room socket a hello joining again replaces, or null. A page + joining again names the last ticket it was handed, which only its + own room socket ever was. */ + rejoining (ticket) + { + const was = this.ctx.tickets.get(ticket); + + return was?.room === this && was.until > relayNow() && + this.leaving.has(was.peer.ws) ? was.peer.ws : null; } touch () { - if (this.empty) + if (this.empty && !this.gone) + { this.emptySince = relayNow(); + this.ctx.empties.delete(this); + this.ctx.empties.set(this, true); + } } /* Every socket cut and every timer stopped: the awareness keeps one of its own, to age out states, and a room that only dropped its - document would keep the process alive by it. */ - destroy () + document would keep the process alive by it. Its people are told + `error' (tell) if there is one. */ + destroy (error = null) { + this.charge(-this.bytes); + this.gone = true; + this.end(); + for (const ws of this.docConns) ws.terminate(); for (const p of this.peers.values()) - p.ws.terminate(); + if (error !== null && p.ws.readyState === p.ws.OPEN) + this.tell(p.ws, error); + else + p.ws.terminate(); this.awareness.destroy(); this.doc.destroy(); + this.ctx.empties.delete(this); + + /* A socket told why holds the room until its close completes, + a few seconds (tell): what the room holds goes now. */ + this.playing = this.held = null; + this.doc = new Y.Doc(); + this.awareness = new awarenessProtocol.Awareness(this.doc); + this.awareness.destroy(); } } @@ -1385,13 +2620,22 @@ class Room `trustProxy' are accountRoutes'; `passkeys' is passkeyConfig's, or null for none. `metricsPort' serves metrics on 127.0.0.1, or null for none; `server.metrics' is that server, and one that cannot listen rejects. - The two times are for a harness. */ + The two times, the caps and `limits', over LIMITS, are for a harness. */ export function relay ({ port = 8787, host = '0.0.0.0', tree = path.join(here, '..', '..'), db = ':memory:', corsOrigin = null, trustProxy = 0, passkeys = null, metricsPort = null, ticketMs = TICKET_MS, heartbeatMs = HEARTBEAT_MS, - sessionCheckMs = SESSION_CHECK_MS } = {}) + sessionCheckMs = SESSION_CHECK_MS, + queuedMaxBytes = QUEUED_MAX_BYTES, + queuedTotalMaxBytes = QUEUED_TOTAL_MAX_BYTES, + roomsMax = ROOMS_MAX, roomsMaxBytes = ROOMS_MAX_BYTES, + roomMaxBytes = ROOM_MAX_BYTES, + roomLimits = ROOM_LIMITS, joinLimits = JOIN_LIMITS, + peersMax = PEERS_PER_ROOM_MAX, limits = {}, + updateLimits = UPDATE_LIMITS, + heapMaxBytes = v8.getHeapStatistics().heap_size_limit, + memoryMaxBytes = null } = {}) { const rooms = new Map(); const tickets = new Map(); /* ticket -> { room, peer, until } */ @@ -1414,24 +2658,221 @@ export function relay ({ port = 8787, host = '0.0.0.0', /* Passkeys are an account's, and there are none without accounts. */ const keys = corsOrigin === null || passkeys === null ? null : new Passkeys({ accounts, ...passkeys }); + const keyOf = clientOf(trustProxy, accounts.log); const api = accountRoutes(accounts, { corsOrigin, trustProxy, - passkeys: keys }); + passkeys: keys, client: keyOf }); + + const empties = new Map(); + const unwelcomed = new Map(); /* client -> sockets not welcomed */ + const costs = { bytes: 0 }; + const rates = { ...LIMITS, ...limits, + room: { ...RATES, ...limits.room } }; + const roomLimit = new TieredLimit(roomLimits, relayNow, () => {}); + const joinLimit = new TieredLimit(joinLimits, relayNow, () => {}); + const updateLimit = new TieredLimit(updateLimits, relayNow, () => {}); + const updatesIn = (client, bytes) => + { + try + { + updateLimit.take(client, bytes); + return true; + } + catch + { + return false; + } + }; + + const forget = (r) => + { + r.destroy(); + rooms.delete(r.name); + }; + + /* The rooms empty longest, other than `keep', that would have to go + for the rooms to be charged `bytes' more and be `more' more, or + null if all that are empty would not do. */ + const toLetGo = (bytes, more, keep) => + { + const going = []; + let left = rooms.size + more; + let total = costs.bytes + bytes; + + for (const [old] of empties) + { + if (left <= roomsMax && total <= roomsMaxBytes) + break; + + if (old !== keep) + { + going.push(old); + left--; + total -= old.bytes; + } + } + + return left <= roomsMax && total <= roomsMaxBytes ? going : null; + }; - const room = (name, seedWith) => + /* Whether room `r' may be charged `bytes' more, the rooms empty + longest let go of to make room for it. */ + const spare = (r, bytes) => { - let r = rooms.get(name); + const going = toLetGo(bytes, 0, r); - if (r === undefined) + going?.forEach(forget); + return going !== null; + }; + + /* A new room for `client', or `{ why, text, retryMs }' if it may not + have one. Past the caps, the rooms empty longest go first to make + room, and none if all that are empty would not; what a room costs + is known once it is seeded, and before then the caps are looked + at as if it cost what an empty one does, so that a relay too full + for any room refuses one without reading a piece. One refused for + the caps before it is made is not one of the client's new rooms; + one made and seeded and then refused is, since that is most of what + a room costs. */ + const newRoom = (name, seedWith, client) => + { + const wait = roomLimit.waitMs(client); + const full = { why: 'rooms', + text: 'the relay has as many rooms as it can hold; ' + + 'join one already open, or try again later' }; + + if (wait > 0) + return { why: 'rooms', retryMs: wait, + text: 'too many new rooms from your address; try ' + + `again in ${Math.ceil(wait / 1000)} s` }; + + if (toLetGo(ROOM_BYTES, 1, null) === null) + return full; + + roomLimit.take(client); + + const r = new Room(name, seedWith, ctx); + const going = toLetGo(0, 1, r); + + if (going === null) { - r = new Room(name, seedWith, tree, - { accounts, tickets, ticketMs, traffic, - sessions: corsOrigin !== null }); - rooms.set(name, r); + r.destroy(); + return full; } + going.forEach(forget); + rooms.set(name, r); return r; }; + /* A room socket before its hello: nothing is made, or so much as + looked up, for one the relay would not welcome. A page says nothing + else first, and a socket that goes on saying something else is + cut. */ + const greet = (ws, name, seedWith, client) => + { + const deadline = setTimeout(() => ws.terminate(), HELLO_WITHIN_MS); + const welcomed = () => + { + const n = unwelcomed.get(client) - 1; + + clearTimeout(deadline); + ws.off('close', welcomed); + + if (n === 0) + unwelcomed.delete(client); + else + unwelcomed.set(client, n); + }; + + unwelcomed.set(client, (unwelcomed.get(client) ?? 0) + 1); + ws.on('close', welcomed); + + const before = new Bucket(rates.room.other); + const say = (m) => sendLine(queues, traffic, [ws], m); + const hello = (data) => + { + if (ws.readyState !== ws.OPEN) + return; + + let m; + + try + { + m = JSON.parse(data.toString()); + } + catch + { + m = null; + } + + traffic?.roomIn(typeof m?.type === 'string' ? m.type : 'other', + data.length); + + if (m?.type !== 'hello') + { + if (!before.take()) + ws.terminate(); + else if (m === null) + say({ type: 'error', text: 'not JSON' }); + else if (typeof m.type === 'string') + say({ type: 'error', text: 'hello first' }); + + return; + } + + /* Past its joins, nothing of the hello is looked up: a + session or a name is a read of the accounts' file. */ + const wait = joinLimit.waitMs(client); + const who = wait > 0 ? null + : admit(m, corsOrigin === null ? null : accounts); + const there = rooms.get(name); + + /* A page joining again counts once: its old peer goes. */ + const r = wait > 0 + ? { why: 'flood', retryMs: wait, + text: 'too many joins from your address; try ' + + `again in ${Math.ceil(wait / 1000)} s` } + : who.text !== undefined ? who + : there === undefined ? newRoom(name, seedWith, client) + : there.peers.size >= peersMax && + there.rejoining(m.was) === null + ? { why: 'full', + text: `the room is full: ${peersMax} people are in it` } + : there; + + if (r instanceof Room) + { + joinLimit.take(client); + welcomed(); + ws.off('message', heard); + r.attachRoom(ws, m, who, client); + } + else + { + say({ type: 'error', why: r.why, text: r.text, + retryMs: r.retryMs }); + ws.close(); + } + }; + + /* An exception here is the whole process. */ + const heard = (data) => + { + try + { + hello(data); + } + catch (e) + { + process.stderr.write(`relay: hello failed: ${e.stack}\n`); + ws.terminate(); + } + }; + + ws.on('message', heard); + ws.on('error', () => ws.close()); + }; + const server = http.createServer((req, res) => { const url = new URL(req.url, 'http://localhost'); @@ -1471,6 +2912,131 @@ export function relay ({ port = 8787, host = '0.0.0.0', maxPayload: ROOM_FRAME_MAX }); const docWss = new WebSocketServer({ noServer: true, maxPayload: DOC_FRAME_MAX }); + const queues = new Queues(() => [roomWss.clients, docWss.clients], + { socketMax: queuedMaxBytes, + totalMax: queuedTotalMaxBytes }); + v8.setFlagsFromString('--expose-gc'); + + const gc = vm.runInNewContext('gc'); + const shed = { rooms: 0, runs: 0, shedding: false, streak: 0 }; + let lastShare = 0; + let settling = false; + + /* What the relay holds, as the larger share of the two it may. */ + const held = () => + { + const { heapUsed, external } = process.memoryUsage(); + + return Math.max(heapUsed / heapMaxBytes, + (heapUsed + external) / (memoryMaxBytes ?? Infinity)); + }; + + const relieve = () => + { + queues.trim(); + + for (const r of rooms.values()) + [r.grew, r.lastBytes] = [r.bytes - r.lastBytes, r.bytes]; + + if (held() < (shed.shedding ? SHED_LOW : SHED_HIGH)) + { + shed.shedding = false; + shed.streak = 0; + settling = false; + return; + } + + gc(); + + const share = held(); + const fell = share < lastShare; + + lastShare = share; + shed.shedding = share >= SHED_LOW; + + if (!shed.shedding) + { + shed.streak = 0; + settling = false; + return; + } + + const say = (what) => process.stderr.write( + `relay: holding ${Math.round(100 * share)}% of its memory; ` + + `shedding ${what}\n`); + + /* Before the lists are made: a room forgotten has no run. */ + if (empties.size > 0) + { + say(`the empty rooms (${empties.size})`); + shed.rooms += empties.size; + [...empties.keys()].forEach(forget); + } + + if (settling) + { + settling = false; + return; + } + + const runs = [...rooms.values()].filter((r) => r.spareBytes > 0) + .sort((a, b) => b.spareBytes - a.spareBytes); + + /* With what its sockets have queued, which it is not charged: a + page that stops reading holds what it is sent until Queues.trim + cuts it. */ + const weight = (r) => [...r.peers.values()].map((p) => p.ws) + .concat([...r.docConns]) + .reduce((n, ws) => n + ws.bufferedAmount, r.bytes); + const most = [...rooms.values()].filter((r) => !r.empty) + .map((r) => [weight(r), r]) + .sort(([a, ra], [b, rb]) => + Math.max(rb.grew, 0) - Math.max(ra.grew, 0) || b - a) + .map(([, r]) => r); + + if (runs.length + most.length === 0) + return; + + for (let n = 2 ** shed.streak; + n > 0 && runs.length + most.length > 0; n--) + if (runs.length > 0) + { + const r = runs.shift(); + + say(`room ${r.name}'s ${r.spareBytes} B kept for late ` + + 'joiners'); + shed.runs++; + r.overflow(); + } + else if (most.length > 0) + { + const r = most.shift(); + + say(`room ${r.name}, charged ${r.bytes} B, grown ` + + `${r.grew} B, and its ${r.peers.size} people`); + shed.rooms++; + r.destroy(SHORT_OF_MEMORY); + rooms.delete(r.name); + + /* Its queues go now rather than at the closes, which a + page behind holds off for CUT_GRACE_MS. One that misses + the line for it takes the close as a drop, and joins + again all the same. */ + for (const p of r.peers.values()) + p.ws.terminate(); + } + + if (!fell) + shed.streak = Math.min(shed.streak + 1, SHED_DOUBLINGS_MAX); + + settling = true; + }; + const trim = setInterval(relieve, QUEUE_CHECK_MS); + const ctx = { tickets, ticketMs, tree, sessions: corsOrigin !== null, + traffic, queues, empties, costs, spare, limits: rates, + peersMax, bytesMax: roomMaxBytes, updatesIn }; + + trim.unref(); /* Nothing a client sends may throw out of here: an exception in an upgrade listener is the whole process. So is an error on a socket @@ -1504,9 +3070,9 @@ export function relay ({ port = 8787, host = '0.0.0.0', } /* The piece a new room is seeded with is named in the query, - and only counts for the first socket to reach the room. An - empty one seeds nothing: a page joining again a room the relay - has lost brings the document itself. */ + and only counts for the hello that makes the room. An empty one + seeds nothing: a page joining again a room the relay has lost + brings the document itself. */ const seedWith = url.searchParams.get('piece') ?? DEFAULT_PIECE; const given = url.searchParams.get('ticket'); const ticket = m[1] === 'doc' ? tickets.get(given) : null; @@ -1521,6 +3087,16 @@ export function relay ({ port = 8787, host = '0.0.0.0', return; } + const client = keyOf(req); + + if (m[1] === 'room' && + (unwelcomed.get(client) ?? 0) >= UNWELCOMED_MAX) + { + socket.end('HTTP/1.1 429 Too Many Requests\r\n' + + 'Connection: close\r\n\r\n'); + return; + } + const wss = m[1] === 'doc' ? docWss : roomWss; wss.handleUpgrade(req, socket, head, (ws) => @@ -1536,7 +3112,7 @@ export function relay ({ port = 8787, host = '0.0.0.0', else if (m[1] === 'doc') ticket.room.attachDoc(ws, ticket.peer); else - room(m[2], seedWith).attachRoom(ws); + greet(ws, m[2], seedWith, client); } catch (e) { @@ -1580,20 +3156,21 @@ export function relay ({ port = 8787, host = '0.0.0.0', recheck.unref(); - /* Empty rooms go after an hour, and lapsed tickets with them. */ + /* Empty rooms go after a while, and lapsed tickets with them. */ const sweep = setInterval(() => { for (const [t, { until }] of tickets) if (until <= relayNow()) tickets.delete(t); - for (const [name, r] of rooms) - if (r.empty && relayNow() - r.emptySince > EMPTY_FOR) - { - r.destroy(); - rooms.delete(name); - } - }, SWEEP_EVERY); + for (const [r] of empties) + { + if (relayNow() - r.emptySince <= EMPTY_FOR_MS) + break; + + forget(r); + } + }, SWEEP_EVERY_MS); sweep.unref(); @@ -1655,7 +3232,8 @@ export function relay ({ port = 8787, host = '0.0.0.0', timerLagMs: { max: Math.max(lagMax, overdue), now: overdue }, cpuPercent: (cpu.user + cpu.system) / 10 / (now - delayFrom), memoryBytes: process.memoryUsage(), - rooms: rooms.size, peers, roomSockets: roomWss.clients.size, + rooms: rooms.size, roomBytes: costs.bytes, peers, + roomSockets: roomWss.clients.size, docSockets: docWss.clients.size, awarenessClients, bufferedMaxBytes: [...roomWss.clients, ...docWss.clients] .reduce((max, ws) => Math.max(max, ws.bufferedAmount), 0), @@ -1663,6 +3241,7 @@ export function relay ({ port = 8787, host = '0.0.0.0', doc: traffic.doc, catchups: traffic.catchups, catchupMaxBytes: traffic.catchupMaxBytes, + shed, }; /* The lag up to now is this window's, and the timer's next run @@ -1702,6 +3281,7 @@ export function relay ({ port = 8787, host = '0.0.0.0', clearInterval(sweep); clearInterval(recheck); clearInterval(heartbeat); + clearInterval(trim); for (const r of rooms.values()) r.destroy(); @@ -1745,12 +3325,15 @@ if (process.argv[1] !== undefined && which there are no accounts; TRUST_PROXY how many proxies in front append to X-Forwarded-For (1 behind nginx alone); PASSKEY_RP_ID the site's domain, for passkeys (passkeyConfig); METRICS_PORT a port on - 127.0.0.1 to serve metrics on. */ + 127.0.0.1 to serve metrics on; MEMORY_MAX_BYTES the container's + memory, which the relay sheds load short of (SHED_HIGH). */ const opts = { db: process.env.DB || path.join(here, 'relay.db'), corsOrigin: process.env.CORS_ORIGIN || null, trustProxy: Number(process.env.TRUST_PROXY ?? 0), metricsPort: process.env.METRICS_PORT - ? Number(process.env.METRICS_PORT) : null }; + ? Number(process.env.METRICS_PORT) : null, + memoryMaxBytes: process.env.MEMORY_MAX_BYTES + ? Number(process.env.MEMORY_MAX_BYTES) : null }; if (args[0] === 'admin') { @@ -1793,6 +3376,13 @@ if (process.argv[1] !== undefined && process.exit(2); } + if (opts.memoryMaxBytes !== null && + !(Number.isSafeInteger(opts.memoryMaxBytes) && opts.memoryMaxBytes > 0)) + { + process.stderr.write('relay.mjs: MEMORY_MAX_BYTES is a count of bytes\n'); + process.exit(2); + } + try { opts.passkeys = passkeyConfig(process.env); diff --git a/wasm/web/relayload.mjs b/wasm/web/relayload.mjs index 33bce236..c9d4c087 100644 --- a/wasm/web/relayload.mjs +++ b/wasm/web/relayload.mjs @@ -84,7 +84,7 @@ const SCRAPE_EVERY_MIN_S = 0.1; /* A distribution's buckets per factor of e: each a percent wide. */ const HIST_STEPS = 100; -/* Under the relay's chat rate (relay.mjs, CHAT_PER_SECOND). */ +/* Under the relay's chat rate (relay.mjs, CHAT_BURST). */ const CHAT_RATE_MAX = 4; const USAGE = `usage: relayload.mjs [options] diff --git a/wasm/web/relaytest.mjs b/wasm/web/relaytest.mjs index 50b003ed..a2b32946 100644 --- a/wasm/web/relaytest.mjs +++ b/wasm/web/relaytest.mjs @@ -44,14 +44,18 @@ import net from 'node:net'; import os from 'node:os'; import path from 'node:path'; import { fileURLToPath } from 'node:url'; +import v8 from 'node:v8'; +import vm from 'node:vm'; import WebSocket, { WebSocketServer } from 'ws'; import * as Y from 'yjs'; import { WebsocketProvider } from 'y-websocket'; import * as encoding from 'lib0/encoding'; +import * as awarenessProtocol from 'y-protocols/awareness'; +import * as syncProtocol from 'y-protocols/sync'; -import { dspNames, fileNames, hashOf, pieceText, readFile, seenOf, snapshot } - from './doc.js'; +import { dspNames, fileNames, hashOf, pieceName, pieceText, readFile, seenOf, + snapshot } from './doc.js'; import { AccountStore, runAdmin } from './accounts.mjs'; import { PROTOCOL, relay } from './relay.mjs'; import { Room } from './room.js'; @@ -61,6 +65,18 @@ const tree = path.join(here, '..', '..'); let failures = 0; +v8.setFlagsFromString('--expose-gc'); + +const gc = vm.runInNewContext('gc'); + +/* What this process's heap holds once what nothing holds is gone: the + relay's, besides the test's own. */ +function heapUsed () +{ + gc(); + return process.memoryUsage().heapUsed; +} + function fail (what) { failures++; @@ -163,6 +179,29 @@ function refused (ws, ms = 2000) }); } +/* A provider's sync update: what `doc' has past state vector `sv', or + an update as it is. */ +function frame (doc, sv) +{ + const enc = encoding.createEncoder(); + + encoding.writeVarUint(enc, 0); + syncProtocol.writeUpdate(enc, doc instanceof Uint8Array + ? doc : Y.encodeStateAsUpdate(doc, sv)); + return encoding.toUint8Array(enc); +} + +/* A document socket into relay `q''s `room' on `ticket', open. */ +async function docSocket (q, room, ticket) +{ + const d = new WebSocket(`ws://127.0.0.1:${q.address().port}/doc/` + + `${room}?ticket=${ticket}`); + + d.on('error', () => {}); + await new Promise((r) => d.on('open', r)); + return d; +} + /* Who a room socket is, and what lets its document in: a relay of its own, on a file the admin commands can open beside it, and with times short enough to wait out. */ @@ -513,6 +552,26 @@ async function accountsInRooms () 'a cursor that names nobody is named by the relay'); bare.close(); + /* What is kept of a state is what a page's holds. */ + { + const s = await opened(); + const at = { type: { client: 1, clock: 2 }, item: null, + tname: null, assoc: 0 }; + + s.send(states([[4343, { + user: { name: 'X', pad: [{}] }, + cursor: { anchor: { ...at, pad: [{}] }, head: at, pad: 1 }, + pad: Array(250).fill({}) }]])); + await new Promise((r) => setTimeout(r, 300)); + check(JSON.stringify(watcher.awareness.getStates().get(4343)) === + JSON.stringify({ user: { name: 'Hob (guest)', + account: false }, + cursor: { anchor: at, head: at } }), + 'a state is kept as the fields a page\'s has, and ' + + 'nothing else it carries'); + s.close(); + } + /* A page echoes every state it hears: three cursors other sockets hold, in one update, are passed over, not cut. */ { @@ -543,6 +602,18 @@ async function accountsInRooms () `a socket claiming three clients ${what} is cut`); } + /* A cursor is a few hundred bytes: one far longer is cut, and + not kept for the room. */ + { + const s = await opened(); + const cut = refused(s); + + s.send(states([[8001, { pad: 'x'.repeat(64 * 1024) }]])); + check(await cut && !watcher.awareness.getStates().has(8001), + 'a socket sending a cursor far longer than a page\'s ' + + 'is cut, and the cursor not kept'); + } + /* Clients nobody has, said to be gone: nothing to forget, and nothing for the relay to keep a note of. */ { @@ -628,7 +699,10 @@ async function accountsInRooms () await new Promise((r) => setTimeout(r, 200)); const id = pj.awareness.clientID; - const seen = () => pw.awareness.getStates().get(id)?.user; + const moved = { type: { client: 9, clock: 9 }, item: null, + tname: null, assoc: 0 }; + const seen = () => pw.awareness.getStates().get(id); + const movedBy = () => seen()?.cursor?.anchor.type?.clock === 9; const again = docSocket('takeover', jt); const update = encoding.createEncoder(); const enc = encoding.createEncoder(); @@ -641,20 +715,21 @@ async function accountsInRooms () encoding.writeVarUint(update, id); encoding.writeVarUint(update, pj.awareness.meta.get(id).clock + 1); encoding.writeVarString(update, JSON.stringify( - { user: { name: 'Jo', at: 'again' } })); + { user: { name: 'Jo' }, + cursor: { anchor: moved, head: moved } })); encoding.writeVarUint(enc, 1); encoding.writeVarUint8Array(enc, encoding.toUint8Array(update)); again.send(encoding.toUint8Array(enc)); await new Promise((r) => setTimeout(r, 200)); - const taken = seen()?.at === 'again'; + const taken = movedBy(); /* And the old one dies as dead sockets do: no last word. */ pj.shouldConnect = false; pj.ws._socket.destroy(); await new Promise((r) => setTimeout(r, 200)); - check(taken && seen()?.at === 'again', + check(taken && movedBy(), 'a document socket of the same peer takes its cursor ' + 'over, ' + 'and keeps it when the old one closes'); @@ -938,201 +1013,2331 @@ async function metricsServed () } } -/* CORS_ORIGIN is an origin or nothing: the relay will not start on one - with a path, which no request's Origin would ever match, or on `*'. */ -for (const [value, status] of [['https://page.example.org', null], - ['*', 2], - ['https://page.example.org/', 2], - ['https://page.example.org/jam', 2], - ['page.example.org', 2]]) +/* What the relay queues for a socket is bounded, for each and for all of + them, on a relay with caps small enough to reach. */ +async function queuesBounded () { - const r = spawnSync(process.execPath, - [path.join(here, 'relay.mjs'), '--port', '0'], - { env: { ...process.env, CORS_ORIGIN: value, - DB: ':memory:' }, - timeout: 1500, encoding: 'utf8' }); + const q = await relay({ port: 0, host: '127.0.0.1', tree, + queuedMaxBytes: 1024 * 1024, + queuedTotalMaxBytes: 4 * 1024 * 1024, + limits: { fanout: [2 ** 30, 2 ** 30], + roomIn: [2 ** 30, 2 ** 30] } }); + let at = `ws://127.0.0.1:${q.address().port}`; + const join = async (room, name) => + { + const c = new Client(`${at}/room/${room}`, name); - check(r.status === status, - `the relay ${status === 2 ? 'refuses' : 'takes'} CORS_ORIGIN ` + - `${value} (${r.status ?? r.signal})`); -} + await c.open(); + c.send({ type: 'hello', name, protocol: PROTOCOL, tickets: true }); + c.welcome = await c.next('welcome'); + return c; + }; + const until = async (cond, ms = 5000) => + { + for (const end = performance.now() + ms; !cond();) + { + if (performance.now() > end) + return false; -const server = await relay({ port: 0, host: '127.0.0.1', tree }); -const port = server.address().port; -const base = `ws://127.0.0.1:${port}`; + await new Promise((r) => setTimeout(r, 20)); + } -try -{ - /* ---- the room socket ---- */ + return true; + }; + const relayed = (c) => c.got.filter((m) => m.type === 'relayed').length; + const pad = 'x'.repeat(64 * 1024); - const a = new Client(`${base}/room/test?piece=airports.gen`, 'A'); + try + { + /* One that stops reading: cut, and told why once it reads again; + the one that reads has everything. Gestures alone do not cut + it: a peer half way to its cap misses them instead. */ + { + const a = await join('slow', 'A'); + const b = await join('slow', 'B'); + const c = await join('slow', 'C'); - await a.open(); - a.send({ type: 'hello', name: 'Ann', protocol: PROTOCOL, tickets: true }); + const closed = refused(a.ws, 8000); + const transports = () => + c.got.filter((m) => m.type === 'transport').length; - const wa = await a.next('welcome'); + /* As fast as the one that reads takes them, which is under + its cap: a burst would be over it too. */ + const paced = async (m, got) => + { + for (let i = 0; i < 400; i += 20) + { + for (let k = i; k < i + 20; k++) + b.send(m(k)); - check(typeof wa.peer === 'string' && wa.peers.length === 1 && - wa.peers[0].name === 'Ann' && wa.piece === 'airports.gen' && - wa.playing === null, - 'a hello is welcomed with the room as it is'); + await until(() => got() >= i + 20); + } + }; - const b = new Client(`${base}/room/test`, 'B'); + a.ws._socket.pause(); + await paced((i) => ({ type: 'relayed', data: { i, pad } }), + () => relayed(c)); - await b.open(); - b.send({ type: 'hello', name: 'Bo', protocol: PROTOCOL, tickets: true }); + const spared = a.ws.readyState === WebSocket.OPEN && + q.rooms.get('slow').peers.has(a.welcome.peer); - const wb = await b.next('welcome'); - const ja = await a.next('joined'); + await paced((i) => ({ type: 'transport', + data: { type: 'transport', op: 'tempo', i, + pad } }), transports); - check(wb.peers.length === 2 && ja.peer === wb.peer && ja.name === 'Bo', - 'a second peer is told who is here, and the first is told'); + const left = await c.next('left', 5000); - const health = await (await fetch(`http://127.0.0.1:${port}/`)).json(); - const listed = health.rooms.find((r) => r.name === 'test'); + a.ws._socket.resume(); - check(listed?.peers === 2 && listed.piece === 'airports.gen' && - listed.playing === false, - 'the health line lists the room, its two people and its piece'); - check(health.accounts === false && health.passkeys === null, - 'and offers no accounts with no page origin to serve them to'); + const why = await a.next('error', 5000); + const all = relayed(c) === 400 && transports() === 400; - /* Nor any routes for them, from a page or not, and a hello's session - is nothing it knows. */ - { - const r = await fetch( - `http://127.0.0.1:${port}/api/account/register`, - { method: 'POST', headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify({ handle: 'Curl' }) }); - const s = new Client(`${base}/room/nosessions`, 'S'); + check(spared, 'a socket that stops reading misses gestures ' + + 'past half its cap, and is not cut for them'); + check(all && left.peer === a.welcome.peer && + why.why === 'slow' && await closed && + b.ws.readyState === WebSocket.OPEN, + 'a socket that stops reading is cut past its cap and told ' + + 'why, and the others are not'); + b.close(); + c.close(); + } - await s.open(); - s.send({ type: 'hello', name: 'Sid', protocol: PROTOCOL, - tickets: true, session: `s_${'2'.repeat(32)}` }); + /* A catch-up larger than the cap, to a socket that reads it. */ + { + const f = await join('big', 'F'); + const g = await join('big', 'G'); + const hash = await hashOf(q.rooms.get('big').doc); + const key = `${f.welcome.peer}#0`; - const w = await s.next('welcome'); + f.send({ type: 'transport', + data: { type: 'transport', op: 'start', origin: 1, + piece: { hash }, seed: 1, seq: 0, at: -1 } }); - check(r.status === 404 && w.identity.account === null && - w.identity.name === 'Sid' && - w.peers.every((p) => p.account === null), - 'without accounts the routes are not there, a session in a ' + - 'hello joins as anyone does, and nobody is marked a guest'); - s.close(); - } + for (let i = 1; i <= 32; i++) + f.send({ type: 'log', run: key, + data: { type: 'knob', at: i, seq: i, pad } }); - /* A page from before tickets is told to reload, not let in to wait on - a document it cannot open. */ - { - const old = new Client(`${base}/room/test`, 'Old'); + await new Promise((r) => setTimeout(r, 300)); + g.send({ type: 'catchup' }); - await old.open(); - old.send({ type: 'hello', name: 'Old', protocol: PROTOCOL }); + const run = await g.next('catchup'); - const e = await old.next('error'); + check(run.log?.length === 32 && + g.ws.readyState === WebSocket.OPEN, + 'a catch-up larger than a socket\'s cap reaches it'); - check(/older than the relay: press Update above/.test(e.text) && - await refused(old.ws), - 'a hello without tickets is told the page is old'); - } + /* Two asked for at once are one answer, which the page takes + for both: two would be over the cap. */ + g.ws._socket.cork(); + g.send({ type: 'catchup' }); + g.send({ type: 'catchup' }); + g.ws._socket.uncork(); + await g.next('catchup'); + check(await g.none('catchup', 500) && + g.ws.readyState === WebSocket.OPEN, + 'two catch-ups asked for at once are answered once'); + + /* Two at once are the same bytes, and one after another + command has it as well. */ + const raw = async (c) => + { + const got = new Promise((r) => c.ws.on('message', (d) => + { + if (JSON.parse(d).type === 'catchup') + r(d); + })); - /* Seats: first claim wins. */ - a.send({ type: 'seat', seat: 0 }); + c.send({ type: 'catchup' }); + return got; + }; + const h = await join('big', 'H'); + const [x, y] = await Promise.all([raw(g), raw(h)]); - const sa = await a.next('seats'); + f.send({ type: 'log', run: key, + data: { type: 'knob', at: 33, seq: 33 } }); + await new Promise((r) => setTimeout(r, 200)); - check(sa.seats[0] === wa.peer, 'a free seat is claimed'); + const z = JSON.parse(await raw(h)); - await b.next('seats'); - b.send({ type: 'seat', seat: 0 }); + check(x.equals(y) && z.log.map((c) => c.seq).join() === + Array.from({ length: 33 }, (_, i) => i + 1).join(), + 'two catch-ups at once are the same bytes, and a later ' + + 'one has the commands since'); - const sb = await b.next('seats'); + f.close(); + g.close(); + h.close(); + } + } + finally + { + q.shutdown(); + } - check(sb.seats[0] === wa.peer, 'a taken seat stays with its holder'); + /* A catch-up asked for while the last answer drains, after a Play + has replaced its run, is answered with the run since once that has + drained. */ + const k = await relay({ port: 0, host: '127.0.0.1', tree, + limits: { roomIn: [2 ** 30, 2 ** 30] } }); - b.send({ type: 'seat', seat: 3 }); + at = `ws://127.0.0.1:${k.address().port}`; - const sb2 = await b.next('seats'); + try + { + const f = await join('drains', 'F'); + const g = await join('drains', 'G'); + const hash = await hashOf(k.rooms.get('drains').doc); + const start = (seq) => f.send({ + type: 'transport', + data: { type: 'transport', op: 'start', origin: 1, + piece: { hash }, seed: 1, seq, at: -1 } }); + const answers = () => g.got.filter((m) => m.type === 'catchup'); + const big = 'x'.repeat(500 * 1024); - check(sb2.seats[3] === wb.peer && sb2.seats[0] === wa.peer, - 'another seat is free to take'); + start(0); - await a.next('seats'); - await a.next('seats'); + for (let i = 1; i <= 14; i++) + f.send({ type: 'log', run: `${f.welcome.peer}#0`, + data: { type: 'knob', at: i, seq: i, pad: big } }); - /* The clock. */ - const t0 = performance.now(); + await new Promise((r) => setTimeout(r, 500)); + g.ws._socket.pause(); + g.send({ type: 'catchup' }); + await new Promise((r) => setTimeout(r, 300)); - a.send({ type: 'ping', t0 }); + const held = [...k.rooms.get('drains').peers.values()] + .find((p) => p.name === 'G').ws.heldBytes > 0; - const pong = await a.next('pong'); + start(1); + await new Promise((r) => setTimeout(r, 200)); + g.send({ type: 'catchup' }); + await new Promise((r) => setTimeout(r, 200)); + g.ws._socket.resume(); + await until(() => answers().length === 2, 5000); + check(held && answers().map((m) => m.start.seq).join() === '0,1', + 'a catch-up asked for while the last drains is answered with ' + + 'the run as it is once that has drained'); + f.close(); + g.close(); + } + finally + { + k.shutdown(); + } - check(pong.t0 === t0 && typeof pong.t1 === 'number' && pong.t1 > 0, - 'a ping is answered with the relay\'s clock'); + /* Two that stop reading, each under its own cap: over the relay-wide + total, the one holding more is cut, and the other is left. */ + const w = await relay({ port: 0, host: '127.0.0.1', tree, + queuedTotalMaxBytes: 4 * 1024 * 1024 }); - /* Signalling: to one peer, opaque. */ - a.send({ type: 'signal', to: wb.peer, data: { sdp: 'offer' } }); + at = `ws://127.0.0.1:${w.address().port}`; - const sig = await b.next('signal'); + try + { + /* What one turn sends a socket goes in one write, and in the + order it was sent, small frames and large alike. */ + { + const x = await join('order', 'X'); + const y = await join('order', 'Y'); + const sizes = Array.from({ length: 300 }, + (_, i) => (i % 50 === 7 ? 100000 : i)); - check(sig.from === wa.peer && sig.data.sdp === 'offer', - 'a signal reaches the peer it names'); - check(await a.none('signal'), 'and not its sender'); + sizes.forEach((n, i) => + { + x.send({ type: 'relayed', data: { i, pad: 'x'.repeat(n) } }); + + if (i % 30 === 0) + y.send({ type: 'ping', t0: i }); + }); + + await until(() => relayed(y) === sizes.length); + + const order = y.got.filter((m) => m.type === 'relayed') + .map((m) => m.data.i); + const pongs = y.got.filter((m) => m.type === 'pong'); + + check(order.join() === sizes.map((n, i) => i).join() && + pongs.map((m) => m.t0).join() === + sizes.map((n, i) => i).filter((i) => i % 30 === 0) + .join(), + 'what a socket is sent arrives in order, written a turn ' + + 'at a time'); + x.close(); + y.close(); + } - /* Relayed gestures: to one, and to everyone. */ - b.send({ type: 'relayed', to: wa.peer, data: { type: 'knob' } }); + const a = await join('both', 'A'); + const b = await join('both', 'B'); + const s = await join('both', 'S'); + const queued = (c) => + w.rooms.get('both').peers.get(c.welcome.peer)?.ws + .bufferedAmount ?? 0; + const fill = (c, bytes) => until(() => + { + s.send({ type: 'relayed', to: c.welcome.peer, data: { pad } }); + return queued(c) > bytes || s.got.some((m) => m.type === 'left'); + }, 20000); + + a.ws._socket.pause(); + b.ws._socket.pause(); + await fill(a, 3 * 1024 * 1024); + await fill(b, 4 * 1024 * 1024); + + const left = await s.next('left'); + + check(left.peer === a.welcome.peer && await s.none('left', 300) && + w.rooms.get('both').peers.has(b.welcome.peer), + 'over the relay-wide total, the socket holding the most is ' + + 'cut'); + a.ws.terminate(); + b.ws.terminate(); + s.close(); + } + finally + { + w.shutdown(); + } - const rel = await a.next('relayed'); + /* A socket cut is not counted again before its close comes: a + second look at once cuts nobody else. */ + const x = await relay({ port: 0, host: '127.0.0.1', tree, + limits: { fanout: [2 ** 30, 2 ** 30], + roomIn: [2 ** 30, 2 ** 30] } }); - check(rel.from === wb.peer && rel.data.type === 'knob', - 'a relayed gesture reaches one peer'); + at = `ws://127.0.0.1:${x.address().port}`; - a.send({ type: 'relayed', data: { type: 'note' } }); + try + { + const [a, b, s] = await Promise.all(['A', 'B', 'S'].map((n) => + join('twice', n))); + const room = x.rooms.get('twice'); + const queued = (c) => + room.peers.get(c.welcome.peer)?.ws.bufferedAmount ?? 0; + const fill = (c, bytes) => until(() => + { + s.send({ type: 'relayed', to: c.welcome.peer, data: { pad } }); + return queued(c) > bytes; + }, 20000); + + a.ws._socket.pause(); + b.ws._socket.pause(); + await fill(a, 2 * 1024 * 1024); + await fill(b, 1024 * 1024); + room.ctx.queues.totalMax = queued(a) + queued(b) - 1; + room.ctx.queues.trim(); + room.ctx.queues.trim(); + await new Promise((r) => setTimeout(r, 300)); + check(!room.peers.has(a.welcome.peer) && + room.peers.has(b.welcome.peer), + 'a socket cut for the relay-wide total is not counted ' + + 'again before it closes'); + a.ws.terminate(); + b.ws.terminate(); + s.close(); + } + finally + { + x.shutdown(); + } - const rel2 = await b.next('relayed'); + /* What a socket has had queued for others lately is not what it + holds: over the total, one holding an edit it does not read goes, + and not one that sent others what they read. The total is above + one gesture to both readers, and each is read before the next is + sent, so only the edit takes the relay over it however slowly the + readers drain. */ + const t = await relay({ port: 0, host: '127.0.0.1', tree, + queuedTotalMaxBytes: 2.5 * 1024 * 1024 }); - check(rel2.from === wa.peer && rel2.data.type === 'note', - 'a relayed gesture reaches everyone else'); + at = `ws://127.0.0.1:${t.address().port}`; - /* A transport start is kept for a joiner. */ - a.send({ type: 'transport', data: { op: 'start', origin: 12345 } }); + try + { + const [a, c, e, f, g] = await Promise.all( + ['A', 'C', 'E', 'F', 'G'].map((n) => join('blame', n))); + const cd = await docSocket(t, 'blame', c.welcome.ticket); + const ad = await docSocket(t, 'blame', a.welcome.ticket); + const big = 'x'.repeat(900 * 1024); - const tr = await b.next('transport'); + await new Promise((r) => setTimeout(r, 200)); + cd._socket.pause(); - check(tr.from === wa.peer && tr.data.origin === 12345, - 'a transport start reaches the others'); + for (let i = 1; i <= 3; i++) + { + e.send({ type: 'relayed', to: [f.welcome.peer, g.welcome.peer], + data: { big } }); + await until(() => relayed(f) === i && relayed(g) === i); + } - const c = new Client(`${base}/room/test`, 'C'); + const y = new Y.Doc(); - await c.open(); - c.send({ type: 'hello', name: 'Cy', protocol: PROTOCOL, tickets: true }); + y.getText('t').insert(0, 'y'.repeat(3000 * 1000)); + ad.send(frame(y)); - const wc = await c.next('welcome'); + /* A paused socket reads no close: gone is the relay's to say. */ + const room = t.rooms.get('blame'); + const gone = await until(() => ![...room.docConns].some((x) => + room.docOwner.get(x) === room.peers.get(c.welcome.peer))); - check(wc.playing?.origin === 12345 && wc.peers.length === 3 && - wc.seats[0] === wa.peer && wc.seats[3] === wb.peer, - 'a joiner is told what is playing and who sits where'); + await until(() => relayed(f) === 3 && relayed(g) === 3); + check(gone && e.ws.readyState === WebSocket.OPEN && + relayed(f) === 3 && relayed(g) === 3, + 'over the relay-wide total, a socket holding an edit it ' + + 'does not read is cut, and not one that sent others what ' + + 'they read'); + ad.close(); + cd.terminate(); - await a.next('joined'); - await b.next('joined'); + for (const x of [a, c, e, f, g]) + x.close(); + } + finally + { + t.shutdown(); + } +} - /* And to several peers at once: what a page sends when more than one - of its channels is down, so the relay serialises the command once - rather than once per peer. */ - b.send({ type: 'relayed', to: [wa.peer, wc.peer], - data: { type: 'tempo' } }); +/* A room is made by a hello the relay welcomes and by nothing else, and + only so many, by count, by what they hold and by who asks. */ +async function roomsBounded () +{ + const open = async (q, room, query = '') => + { + const c = new Client(`ws://127.0.0.1:${q.address().port}/room/` + + `${room}${query}`, room); - const rel3 = await a.next('relayed'); - const rel4 = await c.next('relayed'); + await c.open(); + return c; + }; + const hello = async (q, room, query) => + { + const c = await open(q, room, query); - check(rel3.from === wb.peer && rel3.data.type === 'tempo' && - rel4.from === wb.peer && rel4.data.type === 'tempo', - 'a relayed gesture reaches every peer a list names'); + c.send({ type: 'hello', name: room, protocol: PROTOCOL, + tickets: true }); + c.said = await Promise.race([c.next('welcome'), c.next('error')]); + return c; + }; + const synced = (p) => new Promise((r) => + { + const timer = setTimeout(() => r(false), 5000); - /* Leaving releases the seat. */ - b.close(); + p.synced ? r(true) : p.once('synced', () => + { + clearTimeout(timer); + r(true); + }); + }); + const q = await relay({ port: 0, host: '127.0.0.1', tree, roomsMax: 2, + roomLimits: [{ burst: 3, refillMs: 60000 }, + null, null] }); - const left = await a.next('left'); - const seats = await a.next('seats'); + try + { + const bare = await open(q, 'bare'); + const wrong = await open(q, 'wrong'); + const doc = new WebSocket(`ws://127.0.0.1:${q.address().port}` + + '/doc/nodoc'); + + wrong.send({ type: 'hello', protocol: PROTOCOL + 1, tickets: true }); + await wrong.next('error'); + await refused(doc); + await new Promise((r) => setTimeout(r, 200)); + check(q.rooms.size === 0, + 'a room socket with no hello, one with a hello the relay ' + + 'refuses, and a document socket make no room'); + bare.close(); + + const a = await hello(q, 'one'); + const b = await hello(q, 'two'); + const c = await hello(q, 'three'); + const again = await hello(q, 'one'); + + check(a.said.type === 'welcome' && b.said.type === 'welcome' && + c.said.why === 'rooms' && /as many rooms/.test(c.said.text) && + again.said.type === 'welcome', + 'past the most rooms a new one is refused, and one already ' + + 'open is joined'); + + b.close(); + await a.next('left', 500).catch(() => null); + await new Promise((r) => setTimeout(r, 200)); + + const d = await hello(q, 'four'); + + check(d.said.type === 'welcome' && !q.rooms.has('two') && + q.rooms.has('one'), + 'and an empty room goes to make room for a new one'); + + const e = await hello(q, 'five'); + + check(e.said.why === 'rooms' && + /too many new rooms/.test(e.said.text) && e.said.retryMs > 0 && e.said.retryMs <= 60000, + 'a client that makes rooms too fast is refused another, and ' + + 'told when to try again; one refused for the most rooms was ' + + 'not one of its new rooms'); + + for (const x of [a, again, d, e, c]) + x.close(); + } + finally + { + q.shutdown(); + } + + /* So many joins from one address, into a room already open as into a + new one: each is told to the whole room. */ + const j = await relay({ port: 0, host: '127.0.0.1', tree, + joinLimits: [{ burst: 3, refillMs: 60000 }, + null, null] }); + + try + { + const joins = []; + + for (let i = 0; i < 4; i++) + joins.push(await hello(j, 'joins')); + + const [last] = joins.slice(-1); + + check(joins.slice(0, 3).every((c) => c.said.type === 'welcome') && + last.said.why === 'flood' && + /too many joins/.test(last.said.text) && + last.said.retryMs > 0 && last.said.retryMs <= 60000, + 'a client that joins too often is refused, and told when to ' + + 'try again'); + + /* Before its hello is read for anything else. */ + const wrong = await open(j, 'joins'); + + wrong.send({ type: 'hello', protocol: PROTOCOL + 1, tickets: true }); + check((await wrong.next('error')).why === 'flood', + 'and a hello past its joins is refused for them before ' + + 'anything in it is looked at'); + joins.push(wrong); + + for (const c of joins) + c.close(); + } + finally + { + j.shutdown(); + } + + /* So many people in a room and no more, a page joining again + aside. */ + const p = await relay({ port: 0, host: '127.0.0.1', tree, peersMax: 2 }); + + try + { + const a = await hello(p, 'full'); + const b = await hello(p, 'full'); + const c = await hello(p, 'full'); + const back = await open(p, 'full'); + + back.send({ type: 'hello', name: 'back', protocol: PROTOCOL, + tickets: true, was: b.said.ticket }); + + check(c.said.why === 'full' && /2 people/.test(c.said.text) && + (await back.next('welcome')).peers.length === 2, + 'a room of the most people refuses another, and lets one ' + + 'join again'); + + for (const x of [a, b, c, back]) + x.close(); + } + finally + { + p.shutdown(); + } + + /* A room is charged for what its document took, and no more than an + eighth of the budget: an edit that could take it past that is + refused, and the room is joined, read and edited as before. A + frame Yjs cannot read costs its socket and charges nothing. And + empty rooms go only as many as make room for a new one. */ + const e = await relay({ port: 0, host: '127.0.0.1', tree, roomsMax: 3, + roomsMaxBytes: 128 * 1024, + roomMaxBytes: 64 * 1024 }); + + try + { + const a = await hello(e, 'edits'); + const j = await docSocket(e, 'edits', a.said.ticket); + const junk = new Uint8Array(16 * 1024).fill(0xff); + const room = e.rooms.get('edits'); + const before = room.bytes; + const big = new Y.Doc(); + const text = big.getText('t'); + let sv = Y.encodeStateVector(big); + const typed = () => + { + const f = frame(big, sv); + + sv = Y.encodeStateVector(big); + return f; + }; + + junk.set([0, 2]); + j.send(junk); + + const junked = await refused(j) && room.bytes === before; + const d = await docSocket(e, 'edits', a.said.ticket); + + text.insert(0, 'x'.repeat(10 * 1024)); + d.send(typed()); + await new Promise((r) => setTimeout(r, 200)); + + const grown = room.bytes; + + /* Keystrokes, each splitting the text it lands in: an item, and + the piece of the text after it. */ + for (let i = 0; i < 10; i++) + { + text.insert(i * 500, 'k'); + d.send(typed()); + } + + await new Promise((r) => setTimeout(r, 200)); + + const keyed = room.bytes - grown; + const n = Math.floor((64 * 1024 - room.bytes) / 2) - 2048; + + text.insert(0, 'y'.repeat(n)); + d.send(typed()); + await new Promise((r) => setTimeout(r, 200)); + + const took = room.doc.getText('t').toString().startsWith('y'); + + text.insert(0, 'z'.repeat(20 * 1024)); + d.send(typed()); + + const told = await a.next('refused'); + + check(junked && grown > before + 20 * 1024 && + keyed > 10 * 2 * 256 && took && told.of === 'edit' && + told.why === 'big' && await refused(d) && + a.ws.readyState === WebSocket.OPEN && + room.bytes <= 64 * 1024 && + !room.doc.getText('t').toString().includes('z'), + 'a room is charged for what its document takes, and not ' + + 'for a frame it could not read, and an edit that could take ' + + 'it past its most is refused: the page is told, and its ' + + 'document socket closed and its room socket not'); + + const b = await hello(e, 'edits'); + const bd = new Y.Doc(); + const pb = new WebsocketProvider(`ws://127.0.0.1:${e.address().port}` + + '/doc', 'edits', bd, + { WebSocketPolyfill: WebSocket, + params: { ticket: b.said.ticket }, + disableBc: true }); + + const bSynced = await synced(pb); + + bd.getText('t').insert(0, 'ok'); + await new Promise((r) => setTimeout(r, 300)); + check(bSynced && b.ws.readyState === WebSocket.OPEN && + room.doc.getText('t').toString().startsWith('oky') && + bd.getText('t').toString() === room.doc.getText('t').toString(), + 'and the room is joined, read and edited after the refusal'); + pb.destroy(); + bd.destroy(); + b.close(); + await new Promise((r) => setTimeout(r, 200)); + + const [x, y, z] = [await hello(e, 'x'), await hello(e, 'y'), + await hello(e, 'z')]; + + x.close(); + await new Promise((r) => setTimeout(r, 200)); + y.close(); + await new Promise((r) => setTimeout(r, 200)); + + const huge = await hello(e, 'huge', '?piece=sunrise.gen'); + + check(huge.said.why === 'rooms' && e.rooms.has('x') && + e.rooms.has('y'), + 'a room the empty ones would not make room for is refused, ' + + 'and they stay'); + + const w = await hello(e, 'w'); + + check(w.said.type === 'welcome' && !e.rooms.has('x') && + e.rooms.has('y'), + 'and a new room lets go of the empty ones it needs, oldest ' + + 'first, and no more'); + + for (const c of [z, huge, w]) + c.close(); + } + finally + { + e.shutdown(); + } + + /* A room already past its most -- seeded with more than that -- is + joined and read by every page, and only what would add to it is + refused. */ + const o = await relay({ port: 0, host: '127.0.0.1', tree, + roomMaxBytes: 16 * 1024 }); + + try + { + const join = async () => + { + const c = await hello(o, 'over'); + const doc = new Y.Doc(); + const p = new WebsocketProvider( + `ws://127.0.0.1:${o.address().port}/doc`, 'over', doc, + { WebSocketPolyfill: WebSocket, + params: { ticket: c.said.ticket }, disableBc: true }); + + const ok = await synced(p); + + await new Promise((r) => setTimeout(r, 300)); + return { c, doc, p, ok }; + }; + const x = await join(); + const y = await join(); + const read = [x, y].every(({ c, doc, ok }) => + ok && c.ws.readyState === WebSocket.OPEN && + readFile(doc, 'airports.gen') === + readFile(o.rooms.get('over').doc, 'airports.gen')); + + x.doc.getMap('files').get('airports.gen')?.insert(0, '# more\n'); + + const told = await x.c.next('refused'); + + check(read && told.of === 'edit' && told.why === 'big' && + x.c.ws.readyState === WebSocket.OPEN && + !readFile(o.rooms.get('over').doc, 'airports.gen') + .startsWith('# more') && + y.c.ws.readyState === WebSocket.OPEN, + 'a room past its most is joined and read, and an edit that ' + + 'adds to it is refused'); + + /* A switch leaves what it replaced in the document, and is not + read before it is made: a room past its most makes none. */ + y.c.send({ type: 'switch', piece: 'ebb.gen' }); + + const unswitched = await y.c.next('refused'); + + check(unswitched.of === 'switch' && + /as large as/.test(unswitched.why) && + pieceName(o.rooms.get('over').doc) === 'airports.gen', + 'and so is a switch'); + + /* An edit that frees more than it splits is taken. */ + const was = o.rooms.get('over').bytes; + const gen = y.doc.getMap('files').get('airports.gen'); + + gen.delete(10, gen.length - 20); + await new Promise((r) => setTimeout(r, 300)); + check(readFile(o.rooms.get('over').doc, 'airports.gen').length === 20 && + o.rooms.get('over').bytes < was && await y.c.none('refused'), + 'and an edit that cuts it down is taken'); + + for (const { c, doc, p } of [x, y]) + { + p.destroy(); + doc.destroy(); + c.close(); + } + } + finally + { + o.shutdown(); + } + + /* What an update makes the document hold can be many times its size: + deleting every other character splits a text into a struct for + each, at 2 B of update apiece, and an empty object in an array, a + client or a root type is a few bytes of one. None takes a room past + its most, by what each of them holds of the heap as measured. */ + const m = await relay({ port: 0, host: '127.0.0.1', tree, + roomMaxBytes: 1024 * 1024 }); + + try + { + const N = 100000; + const text = new Y.Doc(); + + text.getText('t').insert(0, 'x'.repeat(N)); + + /* Every other character of `text', deleted, in updates of + `per'. */ + const split = (per) => + { + const out = []; + + for (let start = 1; start < N; start += 2 * per) + { + const e = encoding.createEncoder(); + const end = Math.min(N, start + 2 * per); + + encoding.writeVarUint(e, 0); + encoding.writeVarUint(e, 1); + encoding.writeVarUint(e, text.clientID); + encoding.writeVarUint(e, Math.ceil((end - start) / 2)); + + for (let i = start; i < end; i += 2) + { + encoding.writeVarUint(e, i); + encoding.writeVarUint(e, 1); + } + + out.push(encoding.toUint8Array(e)); + } + + return out; + }; + const made = (f) => + { + const d = new Y.Doc(); + + d.transact(() => f(d)); + return [Y.encodeStateAsUpdate(d)]; + }; + const structs = (doc) => [...doc.store.clients.values()] + .reduce((sum, s) => sum + s.length, 0); + + for (const [what, updates, holds] of [ + ['a text with every other character deleted', + [Y.encodeStateAsUpdate(text), ...split(N / 8)], + (doc) => 256 * structs(doc)], + ['an array of empty objects', + made((d) => d.getArray('a').push( + Array.from({ length: N }, () => ({})))), + (doc) => 66 * doc.getArray('a').length], + ['a client for every key of a map', + [Y.mergeUpdates(Array.from({ length: N / 5 }, () => made( + (d) => d.getMap('m').set('k', 1))[0]))], + (doc) => 490 * doc.store.clients.size], + ['a root type for every item', + made((d) => + { + for (let i = 0; i < N / 5; i++) + d.getMap(`r${i}`).set('k', 1); + }), + (doc) => 300 * doc.share.size], + ['an XML element with a long name', + made((d) => d.getXmlFragment('x').insert( + 0, [new Y.XmlElement('e'.repeat(6 * N))])), + (doc) => 2 * doc.getXmlFragment('x').toArray() + .reduce((sum, e) => sum + e.nodeName.length, 0)], + ['a document in the document', + made((d) => d.getArray('d').insert( + 0, [new Y.Doc({ guid: 'g'.repeat(6 * N) })])), + (doc) => [...doc.subdocs] + .reduce((sum, sub) => sum + 2 * sub.guid.length, 0)]]) + { + const name = what.replaceAll(' ', '-'); + const c = await hello(m, name); + const d = await docSocket(m, name, c.said.ticket); + const room = m.rooms.get(name); + + for (const u of updates) + if (d.readyState === WebSocket.OPEN) + { + d.send(frame(u)); + await new Promise((r) => setTimeout(r, 200)); + } + + check(room.bytes <= 1024 * 1024 && + holds(room.doc) <= 1024 * 1024 && + (await fetch(`http://127.0.0.1:${m.address().port}/`)).ok, + `${what} takes a room no further than its most`); + d.close(); + c.close(); + } + + /* A root type an update makes is charged once it is made. */ + { + const c = await hello(m, 'roots'); + const d = await docSocket(m, 'roots', c.said.ticket); + const room = m.rooms.get('roots'); + const [roots, before] = [room.doc.share.size, room.bytes]; + d.send(frame(made((doc) => + { + for (let i = 0; i < 10; i++) + doc.getMap(`r${i}`).set('k', 1); + })[0])); + await new Promise((r) => setTimeout(r, 200)); + check(room.doc.share.size === roots + 10 && + room.bytes - before >= 10 * 1024, + 'a root type an update makes is charged'); + d.close(); + c.close(); + } + + /* A document brought back whole to a room the relay lost, two + clients' keystrokes in turn, splits nothing, and is not + charged as if each of its items split two structs. */ + { + const c = await hello(m, 'restored', '?piece='); + const d = await docSocket(m, 'restored', c.said.ticket); + const doc = new Y.Doc(); + + for (let i = 0; i < 2000; i++) + { + doc.clientID = 1 + i % 2; + doc.getText('t').insert(i, 'x'); + } + + d.send(frame(doc)); + await new Promise((r) => setTimeout(r, 300)); + check(m.rooms.get('restored').doc.getText('t').length === 2000, + 'a document brought back whole is charged only the ' + + 'splits it can make ' + + `(${Math.round(m.rooms.get('restored').bytes / 1024)} KiB)`); + d.close(); + c.close(); + } + + /* Nor does an update whose type is written in two bytes, which + would make its first bytes a sync step 1's. */ + const c = await hello(m, 'long'); + const d = await docSocket(m, 'long', c.said.ticket); + const long = encoding.createEncoder(); + const cut = refused(d); + + encoding.writeUint8(long, 0x80); + encoding.writeUint8(long, 0); + encoding.writeVarUint(long, syncProtocol.messageYjsUpdate); + encoding.writeVarUint8Array(long, made((doc) => + doc.getText('t').insert(0, 'x'.repeat(1024 * 1024)))[0]); + d.send(encoding.toUint8Array(long)); + check(await cut && m.rooms.get('long').doc.getText('t').length === 0, + 'and nor does one whose type is written long'); + c.close(); + } + finally + { + m.shutdown(); + } + + /* The rooms' budget holds as they grow, as when they are made: an + edit lets go of the rooms empty longest that it needs, and with + too few to let go it is refused. */ + const g = await relay({ port: 0, host: '127.0.0.1', tree, + roomsMaxBytes: 272 * 1024, + roomMaxBytes: 224 * 1024 }); + + try + { + const total = () => [...g.rooms.values()] + .reduce((sum, r) => sum + r.bytes, 0); + const edit = async (name) => + { + const c = await hello(g, name); + const d = await docSocket(g, name, c.said.ticket); + const doc = new Y.Doc(); + let sv = Y.encodeStateVector(doc); + + c.errors = []; + c.ws.on('message', (m) => + { + if (JSON.parse(m).type === 'refused') + c.errors.push(JSON.parse(m)); + }); + c.type = async (kib) => + { + doc.getText('t').insert(0, 'x'.repeat(kib * 1024)); + d.send(frame(doc, sv)); + sv = Y.encodeStateVector(doc); + await new Promise((r) => setTimeout(r, 200)); + return g.rooms.get(name)?.doc.getText('t').length / 1024; + }; + return c; + }; + const e = await hello(g, 'idle'); + + e.close(); + await new Promise((r) => setTimeout(r, 200)); + + const a = await edit('a'); + const b = await edit('b'); + const took = [await a.type(40), await b.type(40)]; + + await a.type(40); + + const told = a.errors[0] ?? {}; + const under = total() <= 272 * 1024 && g.rooms.has('idle'); + + check(took.join() === '40,40' && told.why === 'rooms' && under && + g.rooms.get('a').doc.getText('t').length === 40 * 1024, + 'an edit that would take the rooms past their budget is ' + + 'refused, and no empty room goes for it in vain'); + check(await b.type(12) === 52 && !g.rooms.has('idle') && + total() <= 272 * 1024, + 'and one an empty room going makes room for lets it go'); + + for (const c of [a, b]) + c.close(); + } + finally + { + g.shutdown(); + } + + /* An update that builds on what the room does not have is held + until that arrives, and charged while it is; held past a little, + it goes, and so does the socket that sent it. */ + const h = await relay({ port: 0, host: '127.0.0.1', tree }); + + try + { + const c = await hello(h, 'held'); + const d = await docSocket(h, 'held', c.said.ticket); + const room = h.rooms.get('held'); + const doc = new Y.Doc(); + const text = doc.getText('t'); + const before = room.bytes; + const closed = refused(d); + let sv; + const after = async (insert) => + { + sv = Y.encodeStateVector(doc); + insert(); + d.send(frame(doc, sv)); + await new Promise((r) => setTimeout(r, 200)); + }; + + text.insert(0, 'withheld'); + await after(() => text.insert(0, 'k'.repeat(1024))); + + const small = room.bytes - before; + const open = d.readyState === WebSocket.OPEN; + + await after(() => text.insert(0, 'k'.repeat(100 * 1024))); + check(small >= 2 * 1024 && small < 4 * 1024 && open && + await closed && room.bytes === before && + room.doc.store.pendingStructs === null, + 'an update held for what it builds on is charged, and one ' + + 'that leaves much held goes, and its socket with it'); + + /* Held past the most by one socket's frame, most of it another's: + the other goes. */ + const holder = async (kib) => + { + const p = await hello(h, 'held'); + const s = await docSocket(h, 'held', p.said.ticket); + const own = new Y.Doc(); + + own.getText('t').insert(0, 'withheld'); + + const base = Y.encodeStateVector(own); + + own.getText('t').insert(0, 'k'.repeat(kib * 1024)); + s.send(frame(own, base)); + await new Promise((r) => setTimeout(r, 200)); + return s; + }; + const most = await holder(60); + const cut = refused(most); + const last = await holder(8); + + check(await cut && last.readyState === WebSocket.OPEN && + room.doc.store.pendingStructs === null, + 'and of the sockets holding it, the one that holds the most ' + + 'goes'); + last.close(); + c.close(); + } + finally + { + h.shutdown(); + } + + /* What a run keeps is charged to its room: past what the room or + the rooms may be charged, the run overflows, and an edit to the + document that needs what it holds overflows it. */ + const l = await relay({ port: 0, host: '127.0.0.1', tree, + roomsMaxBytes: 150 * 1024, + roomMaxBytes: 128 * 1024 }); + + try + { + const pad = 'x'.repeat(20 * 1024); + const play = async (name) => + { + const c = await hello(l, name); + + c.send({ type: 'transport', + data: { type: 'transport', op: 'start', seq: 0 } }); + c.log = async (n) => + { + for (let i = 0; i < n; i++) + c.send({ type: 'log', run: `${c.said.peer}#0`, + data: { type: 'knob', pad } }); + + await new Promise((r) => setTimeout(r, 200)); + return l.rooms.get(name).run; + }; + return c; + }; + const a = await play('la'); + const b = await play('lb'); + const doc = (await hello(l, 'la')).said; + const before = l.rooms.get('la').bytes; + const kept = (await a.log(4)).length; + const charged = l.rooms.get('la').bytes - before; + const seeded = l.rooms.get('lb').bytes; + const spilled = (await b.log(1)).overflowed; + + check(kept === 4 && charged > 80 * 1024 && spilled && + l.rooms.get('lb').bytes === seeded, + 'a run\'s commands are charged to its room, and one past ' + + 'what the rooms may be charged overflows the run'); + + /* An edit the room has no room for even without them leaves + them be. */ + const big = (await hello(l, 'la')).said; + const bd = await docSocket(l, 'la', big.ticket); + const huge = new Y.Doc(); + + huge.getText('h').insert(0, 'h'.repeat(100 * 1024)); + bd.send(frame(huge)); + await new Promise((r) => setTimeout(r, 200)); + check(!l.rooms.get('la').run.overflowed && + l.rooms.get('la').run.length === 4 && + l.rooms.get('la').doc.getText('h').length === 0, + 'an edit too large for the room even without the run\'s ' + + 'commands is refused, and leaves them be'); + + const d = await docSocket(l, 'la', doc.ticket); + const ydoc = new Y.Doc(); + + ydoc.getText('t').insert(0, 'y'.repeat(20 * 1024)); + d.send(frame(ydoc)); + await new Promise((r) => setTimeout(r, 200)); + check(l.rooms.get('la').run.overflowed && + l.rooms.get('la').doc.getText('t').length === 20 * 1024 && + l.rooms.get('la').bytes < before + 50 * 1024, + 'and an edit the room has room for only without the run\'s ' + + 'commands overflows the run, and is taken'); + d.close(); + a.close(); + b.close(); + } + finally + { + l.shutdown(); + } + + /* And it holds them as what they are charged: parsed, a command of + empty objects holds twenty times its JSON. */ + const ob = await relay({ port: 0, host: '127.0.0.1', tree }); + + try + { + const c = await hello(ob, 'objects'); + const room = ob.rooms.get('objects'); + const lines = Array.from({ length: 8 }, (_, i) => JSON.stringify( + { type: 'log', run: `${c.said.peer}#0`, + data: { type: 'knob', i, x: Array(100000).fill({}) } })); + + c.send({ type: 'transport', + data: { type: 'transport', op: 'start', seq: 0 } }); + await new Promise((r) => setTimeout(r, 200)); + + const heap = heapUsed(); + const before = room.bytes; + + for (const line of lines) + c.ws.send(line); + + await new Promise((r) => setTimeout(r, 500)); + + const held = heapUsed() - heap; + const charged = room.bytes - before; + + check(room.run.length === 8 && held < 2 * charged + 1024 * 1024, + 'a run\'s commands hold about what they are charged ' + + `(${Math.round(held / 1024)} KiB held, ` + + `${Math.round(charged / 1024)} KiB charged)`); + c.close(); + + /* And its start is the fields a page reads of one. */ + const s = await hello(ob, 'start'); + const start = JSON.stringify( + { type: 'transport', + data: { type: 'transport', op: 'start', seq: 0, + x: Array(300000).fill({}) } }); + const was = heapUsed(); + + s.ws.send(start); + await new Promise((r) => setTimeout(r, 300)); + + const kept = heapUsed() - was; + + check(ob.rooms.get('start').playing?.seq === 0 && + !('x' in ob.rooms.get('start').playing) && kept < 1024 * 1024, + 'a start is kept as the fields a page reads of one ' + + `(${Math.round(kept / 1024)} KiB held)`); + s.close(); + } + finally + { + ob.shutdown(); + } + + /* A room waits for one start's document at a time, however many + starts come: each would look at the whole document at every update + for as long as it waited. A stop ends the wait. */ + const k = await relay({ port: 0, host: '127.0.0.1', tree }); + + try + { + const c = await hello(k, 'starts'); + const room = k.rooms.get('starts'); + const waits = () => room.doc._observers.get('update').size; + const idle = waits(); + + for (let i = 0; i < 100; i++) + c.send({ type: 'transport', + data: { type: 'transport', op: 'start', seq: i, + piece: { hash: `h${i}` } } }); + + await new Promise((r) => setTimeout(r, 300)); + + const playing = waits(); + + c.send({ type: 'transport', + data: { type: 'transport', op: 'stop' } }); + await new Promise((r) => setTimeout(r, 200)); + check(room.playing === null && playing === idle + 1 && + waits() === idle, + 'a room waits for one start\'s document at a time, and a ' + + 'stop ends the wait'); + c.close(); + } + finally + { + k.shutdown(); + } + + /* A run is charged the document its start named, a copy of every + text kept for late joiners, and that as a catch-up's JSON once one + is asked for; a stop gives them back. */ + const v = await relay({ port: 0, host: '127.0.0.1', tree }); + + try + { + const c = await hello(v, 'kept'); + const room = v.rooms.get('kept'); + const texts = Object.values(snapshot(room.doc).files) + .reduce((n, t) => n + t.length, 0); + const idle = room.bytes; + + c.send({ type: 'transport', + data: { type: 'transport', op: 'start', seq: 0, + piece: { hash: await hashOf(room.doc) } } }); + await new Promise((r) => setTimeout(r, 300)); + + const named = room.bytes - idle; + + c.send({ type: 'catchup' }); + await c.next('catchup'); + + const asked = room.bytes - idle; + + c.send({ type: 'transport', + data: { type: 'transport', op: 'stop' } }); + await new Promise((r) => setTimeout(r, 200)); + check(named >= 2 * texts && asked >= named + texts && + room.bytes === idle, + 'a run is charged the document its start named, and its ' + + 'catch-up, until it stops'); + c.close(); + } + finally + { + v.shutdown(); + } + + /* A room let go of while a switch in it waits to be played is + charged nothing after: the switch's Play would give back its run's + commands a second time. */ + const s = await relay({ port: 0, host: '127.0.0.1', tree, + roomsMaxBytes: 160 * 1024, + roomMaxBytes: 128 * 1024 }); + + try + { + const b = await hello(s, 'grows'); + const a = await hello(s, 'goes'); + const d = await docSocket(s, 'grows', b.said.ticket); + const pad = 'x'.repeat(20 * 1024); + const ydoc = new Y.Doc(); + + a.send({ type: 'transport', + data: { type: 'transport', op: 'start', seq: 0 } }); + + for (let i = 0; i < 2; i++) + a.send({ type: 'log', run: `${a.said.peer}#0`, + data: { type: 'knob', pad } }); + + await new Promise((r) => setTimeout(r, 200)); + + const logged = s.rooms.get('goes').run.length; + + a.send({ type: 'switch', piece: 'airports.gen' }); + a.ws.terminate(); + await new Promise((r) => setTimeout(r, 15)); + ydoc.getText('t').insert(0, 'y'.repeat(40 * 1024)); + d.send(frame(ydoc)); + await new Promise((r) => setTimeout(r, 300)); + + const room = s.rooms.get('grows'); + + check(logged === 2 && !s.rooms.has('goes') && + room.doc.getText('t').length === 40 * 1024 && + room.ctx.costs.bytes === room.bytes, + 'a room let go of while a switch in it is played is charged ' + + 'nothing after'); + d.close(); + b.close(); + } + finally + { + s.shutdown(); + } + + /* A ticket past its life, not yet swept, lets in nobody past the + most. */ + const t = await relay({ port: 0, host: '127.0.0.1', tree, peersMax: 1, + ticketMs: 1000 }); + + try + { + const a = await hello(t, 'lapsed'); + + await new Promise((r) => setTimeout(r, 1050)); + + const late = await Promise.all([1, 2, 3].map(async () => + { + const c = await open(t, 'lapsed'); + + c.send({ type: 'hello', name: 'late', protocol: PROTOCOL, + tickets: true, was: a.said.ticket }); + c.said = await Promise.race([c.next('welcome'), c.next('error')]); + return c; + })); + + check(late.every((c) => c.said.why === 'full') && + t.rooms.get('lapsed').peers.size === 1 && + a.ws.readyState === WebSocket.OPEN, + 'a page joining again with a lapsed ticket counts against ' + + 'the most people'); + + for (const x of [a, ...late]) + x.close(); + } + finally + { + t.shutdown(); + } + + /* A seeded room is charged for what it was seeded with; one made and + seeded and then refused for it is one of its client's new rooms. */ + const w = await relay({ port: 0, host: '127.0.0.1', tree, + roomsMaxBytes: 200 * 1024, + roomLimits: [{ burst: 3, refillMs: 60000 }, + null, null] }); + + try + { + const big = await hello(w, 'big', '?piece=sunrise.gen'); + const bigger = await hello(w, 'bigger', '?piece=sunrise.gen'); + const small = await hello(w, 'small', '?piece='); + + check(big.said.type === 'welcome' && bigger.said.why === 'rooms' && + small.said.type === 'welcome', + 'rooms are charged for the piece they are seeded with'); + + const again = await hello(w, 'again', '?piece=sunrise.gen'); + + check(/too many new rooms/.test(again.said.text), + 'and a room made and refused for it counts as one of its ' + + 'client\'s new rooms'); + + for (const x of [big, bigger, small, again]) + x.close(); + } + finally + { + w.shutdown(); + } +} + +/* Every message a room socket sends is limited by type, as chat is: past + its rate it is dropped and the page told, and a socket that keeps on is + cut. A document socket's frames are limited too. */ +async function ratesLimited () +{ + const types = ['ping', 'signal', 'relayed', 'log', 'transport', 'seat', + 'switch', 'catchup', 'chat', 'other']; + const q = await relay({ port: 0, host: '127.0.0.1', tree, + limits: { room: Object.fromEntries( + types.map((t) => [t, [3, 0.001]])), + drops: [5, 0.001], doc: [3, 0.001] } }); + const at = `127.0.0.1:${q.address().port}`; + const join = async (room, name) => + { + const c = new Client(`ws://${at}/room/${room}`, name); + + await c.open(); + c.send({ type: 'hello', name, protocol: PROTOCOL, tickets: true }); + c.welcome = await c.next('welcome'); + return c; + }; + const count = (c, type) => c.got.filter((m) => m.type === type).length; + + try + { + /* Each type: what is sent, and what three of it come to. */ + for (const [type, m, seen] of [ + ['ping', { type: 'ping', t0: 1 }, (a) => count(a, 'pong')], + ['signal', (b) => ({ type: 'signal', to: b, data: {} }), + (a, b) => count(b, 'signal')], + ['relayed', { type: 'relayed', data: {} }, + (a, b) => count(b, 'relayed')], + ['log', { type: 'log', data: {}, run: 'x#0' }, null], + ['transport', { type: 'transport', data: { op: 'tempo' } }, + (a, b) => count(b, 'transport')], + ['seat', { type: 'seat', seat: 1 }, (a) => count(a, 'seats')], + ['switch', { type: 'switch', piece: 'ebb.gen' }, + (a) => count(a, 'switched')], + ['catchup', { type: 'catchup' }, (a) => count(a, 'catchup')], + ['chat', { type: 'chat', channel: 'stage', text: 'hi' }, + (a, b) => count(b, 'chat')], + ['other', { type: 'hello', protocol: PROTOCOL, tickets: true }, + null], + ['other', { type: 'constructor' }, null]]) + { + const a = await join(`rate-${type}`, 'A'); + const b = await join(`rate-${type}`, 'B'); + + for (let i = 0; i < 5; i++) + a.send(typeof m === 'function' ? m(b.welcome.peer) : m); + + await new Promise((r) => setTimeout(r, 300)); + + const told = a.got.filter((x) => x.type === 'refused'); + + check(told.length === 1 && told[0].of === type && + /too fast/.test(told[0].why) && + (seen === null || seen(a, b) === 3) && + a.ws.readyState === WebSocket.OPEN, + `${type === 'other' ? `${m.type} (as other)` : type} ` + + 'messages past their rate are dropped, and the ' + + 'page told once'); + a.close(); + b.close(); + } + + /* A Play past its rate is played all the same, the mesh having + carried it, but not the one whose refusal cuts the socket. */ + { + const a = await join('rate-cut', 'A'); + + for (let seq = 0; seq < 9; seq++) + a.send({ type: 'transport', + data: { type: 'transport', op: 'start', seq } }); + + const cut = await refused(a.ws); + + check(cut && q.rooms.get('rate-cut').playing?.seq === 7, + 'a Play that gets its socket cut for flooding does not ' + + `change the run (${q.rooms.get('rate-cut').playing?.seq})`); + } + + /* A catch-up past its rate is refused, and the page's wait for + it ends then. */ + { + const r = new Room(`ws://${at}`, 'rate-room', 'R'); + + await r.connect(); + + for (let i = 0; i < 3; i++) + await r.catchUp(2000); + + const t0 = performance.now(); + const why = await r.catchUp(5000).then(() => '', (e) => e.message); + + check(/refused a catchup: too fast/.test(why) && + performance.now() - t0 < 1000, + 'a catch-up the relay refuses ends the page\'s wait for ' + + `it (${why || 'answered'})`); + r.close(); + } + + /* A start past its rate is not passed on, and is the run all + the same: the mesh carried it. */ + { + const a = await join('rate-start', 'A'); + const b = await join('rate-start', 'B'); + + for (let seq = 0; seq < 5; seq++) + a.send({ type: 'transport', + data: { type: 'transport', op: 'start', seq } }); + + await new Promise((r) => setTimeout(r, 300)); + check(count(b, 'transport') === 3 && + q.rooms.get('rate-start').playing?.seq === 4, + 'a start past its rate is not passed on, and is the run ' + + 'all the same'); + a.close(); + b.close(); + } + + /* Each chat line the page counts is told of, past the first. */ + { + const a = await join('rate-chat-n', 'A'); + + for (let n = 1; n <= 5; n++) + a.send({ type: 'chat', channel: 'stage', text: 'hi', n }); + + await new Promise((r) => setTimeout(r, 300)); + check(a.got.filter((x) => x.type === 'refused') + .map((x) => x.n).join() === '4,5', + 'every counted chat line past the rate is refused with ' + + 'its count, more than one a second'); + a.close(); + } + + /* A switch refused names its piece, whatever was sent as one. */ + { + const a = await join('rate-evil', 'A'); + + for (let i = 0; i < 5; i++) + a.send({ type: 'switch', piece: { toString: 1, valueOf: 1 } }); + + await new Promise((r) => setTimeout(r, 300)); + check(a.got.some((x) => x.type === 'refused' && + /too fast/.test(x.why)) && + a.ws.readyState === WebSocket.OPEN && + (await fetch(`http://${at}/`)).ok, + 'a switch past its rate whose piece throws is refused, ' + + 'and the relay lives'); + a.close(); + } + + /* Before a hello, as after it. */ + { + const w = new WebSocket(`ws://${at}/room/early`); + + await new Promise((r) => w.on('open', r)); + + const cut = refused(w); + + for (let i = 0; i < 10; i++) + w.send(i % 2 ? '{' : JSON.stringify({ type: 'ping', t0: i })); + + check(await cut && !q.rooms.has('early'), + 'a socket that goes on sending something before its ' + + 'hello is cut'); + } + + /* And one that goes on is cut, and told why. */ + for (const m of [{ type: 'ping', t0: 1 }, + { type: 'chat', channel: 'stage', text: 'hi' }]) + { + const a = await join(`flood-${m.type}`, 'A'); + + for (let i = 0; i < 12; i++) + a.send(m); + + const e = await a.next('error'); + + check(e.why === 'flood' && await refused(a.ws), + `a socket that keeps on past its rate is cut (${m.type})`); + } + + /* A cursor past the document socket's rate is dropped; an update + past it cuts the socket. */ + { + const a = await join('docrate', 'A'); + const d = await docSocket(q, 'docrate', a.welcome.ticket); + const step1 = encoding.createEncoder(); + const cursor = encoding.createEncoder(); + const aw = new awarenessProtocol.Awareness(new Y.Doc()); + + encoding.writeVarUint(step1, 0); + syncProtocol.writeSyncStep1(step1, new Y.Doc()); + aw.setLocalState({}); + encoding.writeVarUint(cursor, 1); + encoding.writeVarUint8Array(cursor, + awarenessProtocol.encodeAwarenessUpdate(aw, [aw.clientID])); + + const closed = refused(d); + + for (let i = 0; i < 5; i++) + d.send(encoding.toUint8Array(i < 3 ? step1 : cursor)); + + await new Promise((r) => setTimeout(r, 300)); + + const open = d.readyState === WebSocket.OPEN; + + d.send(encoding.toUint8Array(step1)); + check(open && await closed, + 'a cursor past the document socket\'s rate is dropped, ' + + 'and an update past it cuts the socket'); + + /* The rate is the peer's: a socket opened again on its + ticket has none of it back. */ + const e = await docSocket(q, 'docrate', a.welcome.ticket); + const doc = new Y.Doc(); + const cut = refused(e); + + doc.getText('t').insert(0, 'again'); + e.send(frame(doc)); + check(await cut && + q.rooms.get('docrate').doc.getText('t').length === 0, + 'and one opened again in its place has none of the rate ' + + 'back'); + aw.destroy(); + a.close(); + } + } + finally + { + q.shutdown(); + } +} + +/* What one socket's messages have queued for others is bounded in bytes, + however its `to' is written. */ +async function fanoutBounded () +{ + const f = await relay({ port: 0, host: '127.0.0.1', tree, + limits: { fanout: [2 * 1024 * 1024, 1024], + runFanout: [2 * 1024 * 1024, 1024] } }); + const join = async (name) => + { + const c = new Client(`ws://127.0.0.1:${f.address().port}/room/fan`, + name); + + await c.open(); + c.send({ type: 'hello', name, protocol: PROTOCOL, tickets: true }); + c.welcome = await c.next('welcome'); + return c; + }; + + try + { + const [s, v, w] = [await join('S'), await join('V'), await join('W')]; + + s.send({ type: 'relayed', to: Array(1000).fill(v.welcome.peer), + data: { once: true } }); + await v.next('relayed'); + check(await v.none('relayed', 300), + 'a relayed gesture naming a peer many times reaches it once'); + + const pad = 'x'.repeat(900 * 1024); + + s.send({ type: 'relayed', data: { pad } }); + s.send({ type: 'relayed', data: { pad } }); + + const told = await s.next('refused'); + + await new Promise((r) => setTimeout(r, 300)); + check(told.of === 'relayed' && + [v, w].every((c) => c.got.filter((m) => + m.type === 'relayed').length === 1 && + c.ws.readyState === WebSocket.OPEN) && + s.ws.readyState === WebSocket.OPEN, + 'past its bytes for others a socket\'s gesture goes to ' + + 'nobody, and the page is told'); + + /* A start past its bytes goes to nobody from here, and is the + run here all the same: the mesh carried it. An edit, which only + goes from here, leaves the run as it was. */ + const start = (seq) => s.send({ type: 'transport', + data: { type: 'transport', + op: 'start', seq, + piece: { hash: pad } } }); + const edit = (text) => s.send({ type: 'transport', + run: `${s.welcome.peer}#2`, + data: { type: 'edit', text } }); + + start(1); + start(2); + + const no = await s.next('refused'); + + edit('x'); + edit(pad); + await new Promise((r) => setTimeout(r, 300)); + check(no.of === 'transport' && no.op === 'start' && + f.rooms.get('fan').playing?.seq === 2 && + f.rooms.get('fan').run.length === 1 && + [v, w].every((c) => c.got.filter((m) => + m.type === 'transport').length === 2), + 'past its bytes for others a transport command goes to ' + + 'nobody, and the page is told which; a start is the run ' + + 'all the same, and an edit is not kept'); + + s.close(); + check((await v.next('left')).peer === s.welcome.peer, + 'and its leaving is still told to the others'); + + for (const c of [v, w]) + c.close(); + } + finally + { + f.shutdown(); + } +} + +/* What a socket sends is bounded in bytes before it is read: a frame as + long as the relay takes goes through, and a socket sending them back to + back is cut. */ +async function inBytesBounded () +{ + const g = await relay({ port: 0, host: '127.0.0.1', tree, + limits: { roomIn: [2 * 1024 * 1024, 1024], + docIn: [64 * 1024, 1024], + drops: [40, 0.001] } }); + const at = `127.0.0.1:${g.address().port}`; + const join = async (name) => + { + const c = new Client(`ws://${at}/room/in`, name); + + await c.open(); + c.send({ type: 'hello', name, protocol: PROTOCOL, tickets: true }); + c.welcome = await c.next('welcome'); + return c; + }; + + try + { + const [s, v] = [await join('S'), await join('V')]; + const line = { type: 'relayed', data: 'x'.repeat(1023 * 1024) }; + + s.send(line); + await v.next('relayed'); + check(s.ws.readyState === WebSocket.OPEN, + 'a room line near the longest the relay takes goes through'); + + for (let i = 0; i < 6; i++) + s.send(line); + + const e = await s.next('error'); + + check(e.why === 'flood' && await refused(s.ws) && + s.got.filter((m) => m.type === 'refused' && + m.of === 'bytes').length === 1 && + v.got.filter((m) => m.type === 'relayed').length === 1, + 'a socket sending the longest lines back to back is refused ' + + 'past its bytes, and then cut'); + + const d = await docSocket(g, 'in', v.welcome.ticket); + const doc = new Y.Doc(); + const text = doc.getText('t'); + const closed = refused(d); + + text.insert(0, 'x'.repeat(40 * 1024)); + d.send(frame(doc)); + await new Promise((r) => setTimeout(r, 200)); + + const took = g.rooms.get('in').doc.getText('t').length; + + text.insert(0, 'x'.repeat(40 * 1024)); + d.send(frame(doc)); + check(took === 40 * 1024 && await closed && + g.rooms.get('in').doc.getText('t').length === took, + 'a document update past the socket\'s bytes cuts it'); + doc.destroy(); + v.close(); + } + finally + { + g.shutdown(); + } + + /* And an address's peers' updates between them: one past what they + have sent together is cut, however little its own peer has. */ + const u = await relay({ port: 0, host: '127.0.0.1', tree, + updateLimits: [{ burst: 64 * 1024, + refillMs: 1000 }, null, null] }); + + try + { + const edit = async (name) => + { + const c = new Client(`ws://127.0.0.1:${u.address().port}/room/up`, + name); + + await c.open(); + c.send({ type: 'hello', name, protocol: PROTOCOL, tickets: true }); + + const { ticket } = await c.next('welcome'); + const d = await docSocket(u, 'up', ticket); + const doc = new Y.Doc(); + const cut = refused(d, 500); + + doc.getText(name).insert(0, 'x'.repeat(40 * 1024)); + d.send(frame(doc)); + return [c, await cut]; + }; + const [a, aCut] = await edit('a'); + const [b, bCut] = await edit('b'); + const { doc } = u.rooms.get('up'); + + check(!aCut && bCut && doc.getText('a').length === 40 * 1024 && + doc.getText('b').length === 0, + 'an update past what its address\'s peers have sent ' + + 'between them cuts its socket'); + a.close(); + b.close(); + } + finally + { + u.shutdown(); + } +} + +/* A sync step 1 is answered with the whole document, so a document + socket may ask for one only so often. */ +async function syncAsksBounded () +{ + const y = await relay({ port: 0, host: '127.0.0.1', tree }); + + try + { + const at = `127.0.0.1:${y.address().port}`; + const a = new Client(`ws://${at}/room/asks`, 'A'); + + await a.open(); + a.send({ type: 'hello', name: 'A', protocol: PROTOCOL, + tickets: true }); + + const { ticket } = await a.next('welcome'); + const d = await docSocket(y, 'asks', ticket); + const step1 = encoding.createEncoder(); + let answers = 0; + + encoding.writeVarUint(step1, 0); + syncProtocol.writeSyncStep1(step1, new Y.Doc()); + d.on('message', (x) => + { + const b = new Uint8Array(x); + + if (b[0] === 0 && b[1] === syncProtocol.messageYjsSyncStep2) + answers++; + }); + + const closed = refused(d); + + for (let i = 0; i < 10; i++) + d.send(encoding.toUint8Array(step1)); + + check(await closed && answers === 4, + `a document socket asking for the document again and again ` + + `is answered ${answers} times, and closed`); + + const e = await docSocket(y, 'asks', ticket); + let more = 0; + + e.on('message', (x) => + { + const b = new Uint8Array(x); + + if (b[0] === 0 && b[1] === syncProtocol.messageYjsSyncStep2) + more++; + }); + + const shut = refused(e); + + e.send(encoding.toUint8Array(step1)); + check(await shut && more === 0, + 'and one opened again on the same ticket is not answered ' + + 'again'); + a.close(); + } + finally + { + y.shutdown(); + } +} + +/* Past its share of memory, whatever the rooms are charged, the relay + sheds load a thing at a time: the empty rooms, then what a run keeps + for late joiners, and only then the room charged the most, whose + people are told why. */ +async function memoryShed () +{ + const q = await relay({ port: 0, host: '127.0.0.1', tree, metricsPort: 0, + memoryMaxBytes: 2 ** 40 }); + const at = `127.0.0.1:${q.address().port}`; + const join = async (room, name) => + { + const c = new Client(`ws://${at}/room/${room}`, name); + + await c.open(); + c.send({ type: 'hello', name, protocol: PROTOCOL, tickets: true }); + c.welcome = await c.next('welcome'); + return c; + }; + const until = async (cond) => + { + for (let i = 0; i < 100 && !cond(); i++) + await new Promise((r) => setTimeout(r, 50)); + }; + const usage = process.memoryUsage; + const write = process.stderr.write; + const shed = []; + + try + { + (await join('idle', 'E')).close(); + + const [b, p] = [await join('big', 'B'), await join('played', 'P')]; + const d = await docSocket(q, 'big', b.welcome.ticket); + const doc = new Y.Doc(); + + doc.getText('t').insert(0, 'x'.repeat(100 * 1024)); + d.send(frame(doc)); + p.send({ type: 'transport', + data: { type: 'transport', op: 'start', seq: 0 } }); + + for (let i = 0; i < 4; i++) + p.send({ type: 'log', run: `${p.welcome.peer}#0`, + data: { type: 'knob', pad: 'x'.repeat(1024) } }); + + await until(() => q.rooms.get('played').run?.length === 4); + await until(() => q.rooms.get('idle').empty); + + process.stderr.write = (line) => + (/shedding/.test(line) ? shed.push(line) + : write.call(process.stderr, line)); + process.memoryUsage = () => ({ ...usage(), external: 2 ** 40 }); + await until(() => shed.length >= 3); + process.memoryUsage = usage; + process.stderr.write = write; + + const scraped = await (await fetch( + `http://127.0.0.1:${q.metrics.address().port}/`)).json(); + + const told = await b.next('error'); + + check(/the empty rooms \(1\)/.test(shed[0]) && + /room played's \d+ B kept for late joiners/.test(shed[1]) && + /room big/.test(shed[2]) && !q.rooms.has('idle') && + !q.rooms.has('big') && + scraped.shed.rooms >= 2 && scraped.shed.runs === 1, + 'past its share of memory the relay sheds the empty rooms, ' + + 'then a run\'s log, then the largest room ' + + `(${shed.map((l) => l.replace(/^.*shedding /, '').trim()) + .join('; ')})`); + check(told.why === 'memory' && told.retryMs > 0 && + /short of memory/.test(told.text), + 'and the people in a room cut for memory are told to join ' + + `again shortly (${JSON.stringify(told)})`); + p.close(); + } + finally + { + process.memoryUsage = usage; + process.stderr.write = write; + q.shutdown(); + } +} + +/* The backstop where little is left to shed. */ +async function memoryShedEdges () +{ + const q = await relay({ port: 0, host: '127.0.0.1', tree, metricsPort: 0, + memoryMaxBytes: 2 ** 40 }); + const usage = process.memoryUsage; + const write = process.stderr.write; + const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); + + try + { + const p = new Client(`ws://127.0.0.1:${q.address().port}/room/played`, + 'P'); + + await p.open(); + p.send({ type: 'hello', name: 'P', protocol: PROTOCOL, tickets: true }); + + const w = await p.next('welcome'); + + p.send({ type: 'transport', + data: { type: 'transport', op: 'start', seq: 0 } }); + p.send({ type: 'log', run: `${w.peer}#0`, + data: { type: 'knob', v: 1 } }); + + for (let i = 0; i < 40 && q.rooms.get('played').run?.length !== 1; i++) + await sleep(25); + + p.close(); + + for (let i = 0; i < 40 && !q.rooms.get('played').empty; i++) + await sleep(25); + + process.stderr.write = (line) => + (/shedding/.test(line) ? true : write.call(process.stderr, line)); + process.memoryUsage = () => ({ ...usage(), external: 2 ** 40 }); + await sleep(600); + check(!q.rooms.has('played'), + 'past its share of memory the relay lets go of an empty room ' + + 'that is still playing, and goes on'); + + let [last, worst] = [Date.now(), 0]; + const tick = setInterval(() => + { + worst = Math.max(worst, Date.now() - last); + last = Date.now(); + }, 10); + + await sleep(2000); + clearInterval(tick); + + const { shed } = await (await fetch( + `http://127.0.0.1:${q.metrics.address().port}/`)).json(); + + check(worst < 500 && shed.streak === 0, + 'and with nothing left to shed it looks again as often, ' + + `counting no more to shed (${worst} ms, ${shed.streak})`); + } + finally + { + process.memoryUsage = usage; + process.stderr.write = write; + q.shutdown(); + } +} + +/* One page that stops reading, its queue most of what the relay holds + and under the queues' caps: its room is the one shed, and the memory + goes with it, so that no other room is. */ +async function memoryShedQueued () +{ + const memoryMaxBytes = 100 * 1024 * 1024; + const q = await relay({ port: 0, host: '127.0.0.1', tree, memoryMaxBytes, + queuedMaxBytes: 2 ** 27, + queuedTotalMaxBytes: 2 ** 30, + limits: { fanout: [2 ** 30, 2 ** 30], + roomIn: [2 ** 30, 2 ** 30], + room: { relayed: [1e6, 1e6] } } }); + const join = async (room, name) => + { + const c = new Client(`ws://127.0.0.1:${q.address().port}/room/${room}`, + name); + + await c.open(); + c.send({ type: 'hello', name, protocol: PROTOCOL, tickets: true }); + c.welcome = await c.next('welcome'); + return c; + }; + const usage = process.memoryUsage; + const write = process.stderr.write; + const others = []; + + try + { + /* Nothing held until the queue is: this process's own heap would + otherwise count against the relay's memory while it fills. */ + process.memoryUsage = () => ({ ...usage(), heapUsed: 0, external: 0 }); + + for (let i = 0; i < 12; i++) + others.push(await join(`other${i}`, `O${i}`)); + + const [a, s] = [await join('slow', 'A'), await join('slow', 'S')]; + const pad = 'x'.repeat(256 * 1024); + const queued = () => + q.rooms.get('slow').peers.get(a.welcome.peer).ws.bufferedAmount; + + a.ws._socket.pause(); + + while (queued() < 40 * 1024 * 1024) + { + s.send({ type: 'relayed', to: a.welcome.peer, data: { pad } }); + await new Promise((r) => setImmediate(r)); + } + + const { heapUsed, external } = (gc(), usage()); + const offset = 0.8 * memoryMaxBytes - heapUsed - external; + + process.stderr.write = (line) => + (/shedding/.test(line) ? true : write.call(process.stderr, line)); + process.memoryUsage = () => + { + const u = usage(); + + return { ...u, external: u.external + offset }; + }; + await new Promise((r) => setTimeout(r, 3000)); + process.memoryUsage = usage; + process.stderr.write = write; + + const kept = others.filter((c, i) => q.rooms.has(`other${i}`)); + + check(!q.rooms.has('slow') && kept.length === others.length, + 'a page that stops reading, its queue most of the relay\'s ' + + 'memory, costs its own room and no other ' + + `(${others.length - kept.length} others shed)`); + s.close(); + } + finally + { + process.memoryUsage = usage; + process.stderr.write = write; + others.forEach((c) => c.close()); + q.shutdown(); + } +} + +/* Room sockets that do not say hello: one address holds no more than + UNWELCOMED_MAX of them open (relay.mjs), and each is cut once its time + to say hello is up. */ +async function unwelcomedBounded () +{ + const q = await relay({ port: 0, host: '127.0.0.1', tree }); + const at = `ws://127.0.0.1:${q.address().port}/room/quiet`; + const open = (ws) => new Promise((r) => + { + ws.on('open', () => r(true)); + ws.on('error', () => r(false)); + }); + + try + { + const held = []; + + for (let i = 0; i < 128; i++) + held.push(new WebSocket(at)); + + const opened = await Promise.all(held.map(open)); + const over = new WebSocket(at); + const refusedOver = !(await open(over)); + const t0 = Date.now(); + const cut = await Promise.all(held.map((ws) => refused(ws, 8000))); + const cutMs = Date.now() - t0; + const after = new WebSocket(at); + const openAfter = await open(after); + + check(opened.every(Boolean) && refusedOver, + 'one address holds no more than 128 room sockets open that ' + + 'have not said hello'); + check(cut.every(Boolean) && cutMs > 3000 && openAfter, + 'and each is cut once its time to say hello is up, which ' + + `frees its place (${cutMs} ms)`); + after.terminate(); + } + finally + { + q.shutdown(); + } +} + +/* CORS_ORIGIN is an origin or nothing: the relay will not start on one + with a path, which no request's Origin would ever match, or on `*'. */ +for (const [value, status] of [['https://page.example.org', null], + ['*', 2], + ['https://page.example.org/', 2], + ['https://page.example.org/jam', 2], + ['page.example.org', 2]]) +{ + const r = spawnSync(process.execPath, + [path.join(here, 'relay.mjs'), '--port', '0'], + { env: { ...process.env, CORS_ORIGIN: value, + DB: ':memory:' }, + timeout: 1500, encoding: 'utf8' }); + + check(r.status === status, + `the relay ${status === 2 ? 'refuses' : 'takes'} CORS_ORIGIN ` + + `${value} (${r.status ?? r.signal})`); +} + +const server = await relay({ port: 0, host: '127.0.0.1', tree }); +const port = server.address().port; +const base = `ws://127.0.0.1:${port}`; + +/* Its bytes in unbounded: the log's tests send a long run's commands at + once. */ +const logs = await relay({ port: 0, host: '127.0.0.1', tree, + limits: { roomIn: [2 ** 30, 2 ** 30] } }); +const logsBase = `ws://127.0.0.1:${logs.address().port}`; + +try +{ + /* ---- the room socket ---- */ + + const a = new Client(`${base}/room/test?piece=airports.gen`, 'A'); + + await a.open(); + a.send({ type: 'hello', name: 'Ann', protocol: PROTOCOL, tickets: true }); + + const wa = await a.next('welcome'); + + check(typeof wa.peer === 'string' && wa.peers.length === 1 && + wa.peers[0].name === 'Ann' && wa.piece === 'airports.gen' && + wa.playing === null, + 'a hello is welcomed with the room as it is'); + + const b = new Client(`${base}/room/test`, 'B'); + + await b.open(); + b.send({ type: 'hello', name: 'Bo', protocol: PROTOCOL, tickets: true }); + + const wb = await b.next('welcome'); + const ja = await a.next('joined'); + + check(wb.peers.length === 2 && ja.peer === wb.peer && ja.name === 'Bo', + 'a second peer is told who is here, and the first is told'); + + const health = await (await fetch(`http://127.0.0.1:${port}/`)).json(); + const listed = health.rooms.find((r) => r.name === 'test'); + + check(listed?.peers === 2 && listed.piece === 'airports.gen' && + listed.playing === false, + 'the health line lists the room, its two people and its piece'); + check(health.accounts === false && health.passkeys === null, + 'and offers no accounts with no page origin to serve them to'); + + /* Nor any routes for them, from a page or not, and a hello's session + is nothing it knows. */ + { + const r = await fetch( + `http://127.0.0.1:${port}/api/account/register`, + { method: 'POST', headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ handle: 'Curl' }) }); + const s = new Client(`${base}/room/nosessions`, 'S'); + + await s.open(); + s.send({ type: 'hello', name: 'Sid', protocol: PROTOCOL, + tickets: true, session: `s_${'2'.repeat(32)}` }); + + const w = await s.next('welcome'); + + check(r.status === 404 && w.identity.account === null && + w.identity.name === 'Sid' && + w.peers.every((p) => p.account === null), + 'without accounts the routes are not there, a session in a ' + + 'hello joins as anyone does, and nobody is marked a guest'); + s.close(); + } + + /* A page from before tickets is told to reload, not let in to wait on + a document it cannot open. */ + { + const old = new Client(`${base}/room/test`, 'Old'); + + await old.open(); + old.send({ type: 'hello', name: 'Old', protocol: PROTOCOL }); + + const e = await old.next('error'); + + check(/older than the relay: press Update above/.test(e.text) && + await refused(old.ws), + 'a hello without tickets is told the page is old'); + } + + /* Seats: first claim wins. */ + a.send({ type: 'seat', seat: 0 }); + + const sa = await a.next('seats'); + + check(sa.seats[0] === wa.peer, 'a free seat is claimed'); + + await b.next('seats'); + b.send({ type: 'seat', seat: 0 }); + + const sb = await b.next('seats'); + + check(sb.seats[0] === wa.peer, 'a taken seat stays with its holder'); + + b.send({ type: 'seat', seat: 3 }); + + const sb2 = await b.next('seats'); + + check(sb2.seats[3] === wb.peer && sb2.seats[0] === wa.peer, + 'another seat is free to take'); + + await a.next('seats'); + await a.next('seats'); + + /* The clock. */ + const t0 = performance.now(); + + a.send({ type: 'ping', t0 }); + + const pong = await a.next('pong'); + + check(pong.t0 === t0 && typeof pong.t1 === 'number' && pong.t1 > 0, + 'a ping is answered with the relay\'s clock'); + + /* Signalling: to one peer, opaque. */ + a.send({ type: 'signal', to: wb.peer, data: { sdp: 'offer' } }); + + const sig = await b.next('signal'); + + check(sig.from === wa.peer && sig.data.sdp === 'offer', + 'a signal reaches the peer it names'); + check(await a.none('signal'), 'and not its sender'); + + /* Relayed gestures: to one, and to everyone. */ + b.send({ type: 'relayed', to: wa.peer, data: { type: 'knob' } }); + + const rel = await a.next('relayed'); + + check(rel.from === wb.peer && rel.data.type === 'knob', + 'a relayed gesture reaches one peer'); + + a.send({ type: 'relayed', data: { type: 'note' } }); + + const rel2 = await b.next('relayed'); + + check(rel2.from === wa.peer && rel2.data.type === 'note', + 'a relayed gesture reaches everyone else'); + + /* A transport start is kept for a joiner. */ + a.send({ type: 'transport', data: { op: 'start', origin: 12345 } }); + + const tr = await b.next('transport'); + + check(tr.from === wa.peer && tr.data.origin === 12345, + 'a transport start reaches the others'); + + const c = new Client(`${base}/room/test`, 'C'); + + await c.open(); + c.send({ type: 'hello', name: 'Cy', protocol: PROTOCOL, tickets: true }); + + const wc = await c.next('welcome'); + + check(wc.playing?.origin === 12345 && wc.peers.length === 3 && + wc.seats[0] === wa.peer && wc.seats[3] === wb.peer, + 'a joiner is told what is playing and who sits where'); + + await a.next('joined'); + await b.next('joined'); + + /* And to several peers at once: what a page sends when more than one + of its channels is down, so the relay serialises the command once + rather than once per peer. */ + b.send({ type: 'relayed', to: [wa.peer, wc.peer], + data: { type: 'tempo' } }); + + const rel3 = await a.next('relayed'); + const rel4 = await c.next('relayed'); + + check(rel3.from === wb.peer && rel3.data.type === 'tempo' && + rel4.from === wb.peer && rel4.data.type === 'tempo', + 'a relayed gesture reaches every peer a list names'); + + /* Leaving releases the seat. */ + b.close(); + + const left = await a.next('left'); + const seats = await a.next('seats'); check(left.peer === wb.peer && seats.seats[3] === undefined, 'a peer that leaves gives up its seat'); @@ -1417,8 +3622,8 @@ try command and in all, past which it is a run that cannot be caught up with; and who made a start or a command is the relay's to say. */ { - const f = new Client(`${base}/room/logcap`, 'F'); - const g = new Client(`${base}/room/logcap`, 'G'); + const f = new Client(`${logsBase}/room/logcap`, 'F'); + const g = new Client(`${logsBase}/room/logcap`, 'G'); await Promise.all([f.open(), g.open()]); f.send({ type: 'hello', name: 'Fay', protocol: PROTOCOL, @@ -1430,7 +3635,7 @@ try tickets: true }); await g.next('welcome'); - const hash = await hashOf(server.rooms.get('logcap').doc); + const hash = await hashOf(logs.rooms.get('logcap').doc); const caughtUp = async (seq, log) => { f.send({ type: 'transport', @@ -1470,11 +3675,13 @@ try 'a command is measured in the bytes it is sent as, not ' + 'its characters'); - const many = await caughtUp(4, Array.from({ length: 66 }, + const many = await caughtUp(4, Array.from({ length: 17 }, (_, i) => edit(5 + i, 510))); - check(many.overflowed === true && many.log.length < 66, - 'and so do more commands than the run keeps bytes for'); + check(many.overflowed === true && many.log.length === 0 && + many.files?.piece === 'airports.gen', + 'and so do more commands than the run keeps bytes for, and ' + + 'none of them is sent, but the document is'); f.close(); g.close(); @@ -1545,6 +3752,10 @@ try await i.next('chat'); await h.next('chat'); + /* A refused line counts against the rate as well: a full bucket + first. */ + await new Promise((r) => setTimeout(r, 1000)); + for (let k = 0; k < 10; k++) h.send({ type: 'chat', channel: 'stage', text: `burst ${k}` }); @@ -1804,9 +4015,25 @@ try ['a garbage', new Uint8Array([255, 255, 255, 255, 255])]]) { - const bad = new WebSocket(`${base}/doc/test?ticket=${ticket}`); + /* The last one's close can reach this side before the relay has + counted it gone: past DOCS_PER_PEER the ticket is refused + until it has. */ + let bad = null; + + for (let tries = 0; bad === null && tries < 50; tries++) + { + const w = new WebSocket(`${base}/doc/test?ticket=${ticket}`); + + if (await new Promise((r) => + { + w.on('open', () => r(true)); + w.on('error', () => r(false)); + })) + bad = w; + else + await new Promise((r) => setTimeout(r, 50)); + } - await new Promise((r) => bad.on('open', r)); bad.send(bytes); const closed = await new Promise((r) => @@ -1884,6 +4111,16 @@ try await accountsInRooms(); await metricsServed(); + await queuesBounded(); + await roomsBounded(); + await ratesLimited(); + await fanoutBounded(); + await inBytesBounded(); + await syncAsksBounded(); + await memoryShed(); + await memoryShedEdges(); + await memoryShedQueued(); + await unwelcomedBounded(); } catch (e) { @@ -1891,6 +4128,7 @@ catch (e) } server.shutdown(); +logs.shutdown(); process.stdout.write(`\n${failures === 0 ? 'the relay does what it says' : `${failures} failed`}\n`); diff --git a/wasm/web/room.js b/wasm/web/room.js index b62f952d..a1b3dacf 100644 --- a/wasm/web/room.js +++ b/wasm/web/room.js @@ -97,7 +97,9 @@ export class Room ? `?piece=${this.piece}` : '')); let welcomed = false; - /* The relay's last word, if it said one: { text, why }. */ + /* The relay's last word, if it said one: { text, why }, and + `retryMs' when it says how long until it would take + another. */ let refused = null; this.ws = ws; @@ -126,7 +128,7 @@ export class Room reject(Object.assign(new Error( refused?.text ?? `the relay at ${this.url} closed ` + 'the connection before welcoming us'), - { why: refused?.why })); + { why: refused?.why, retryMs: refused?.retryMs })); for (const c of this.catchups.splice(0)) c.reject(new Error('the relay closed the connection')); @@ -235,7 +237,13 @@ export class Room this.emit('ticket', m.ticket); break; + /* A catch-up past its rate is not coming: the last + one asked for, which the others' answer covers. */ case 'refused': + if (m.of === 'catchup') + this.catchups.pop()?.reject(new Error( + `the relay refused a catchup: ${m.why}`)); + this.emit('refused', m); break; @@ -249,7 +257,8 @@ export class Room break; case 'error': - refused = { text: m.text, why: m.why }; + refused = { text: m.text, why: m.why, + retryMs: m.retryMs }; this.emit('error', m.text); break; }