From f86706dc1c6005be73c2c3ce7ec59ffb633c2ab0 Mon Sep 17 00:00:00 2001 From: Tam Nguyen Duc <1218621+tamnd@users.noreply.github.com> Date: Tue, 18 Aug 2026 21:03:33 +0700 Subject: [PATCH] INT64 as a number, for the program that asked for it `bigIntMode` says how INT64 is spelled on the way out, on a statement or on a connection, and a statement on a connection that named one may name the other. The default does not move: it is `bigint`, everywhere, because zu's integers go to 2^63 and a JavaScript number stops telling one integer from the next at 2^53. An integer past that is refused rather than rounded. The refusal is a ZuUsageError naming the column and the value, so a program that guessed wrong about its own data gets a failure it can act on instead of an answer that is quietly off by one. That is the whole hazard, stated in the types, in the README and in the doc comment: which integers a database holds is a property of the data and not of the program, so the failure arrives at read time on somebody else's machine. The mode reaches the INT64 columns of a result and nothing else. A node's offset, an edge's src, dst and ord, and the nanosecond counts on the temporal classes stay `bigint`, since they are properties of classes the addon registers once rather than values a statement can respell. Ten tests, covering both places the mode is named, the override in either direction, the edge of the exact range, the refusal inside a list and inside a record, a stream that spells its rows the same way, and a mode nobody can spell, which is refused at connect time before a database is created. On 50k rows one INT64 column costs about 190ns a row as numbers against about 220ns as bigints. --- README.md | 26 +++++- bench/query.mjs | 10 +++ binding.d.cts | 44 +++++++++- src/conn.rs | 104 +++++++++++++++++++----- src/stream.rs | 11 +-- src/value.rs | 117 +++++++++++++++++++++++++-- test/bigint.test.mjs | 189 +++++++++++++++++++++++++++++++++++++++++++ test/types/cjs.cts | 13 +++ test/types/esm.mts | 15 ++++ types/header.d.ts | 38 ++++++++- 10 files changed, 528 insertions(+), 39 deletions(-) create mode 100644 test/bigint.test.mjs diff --git a/README.md b/README.md index 70814f6..082367b 100644 --- a/README.md +++ b/README.md @@ -27,7 +27,7 @@ The rows are an array, so iterating them is `for (const row of rows)` and nothin ## Decisions worth knowing before you start -- **INT64 is `bigint`.** Always, by default. A JavaScript number stops being exact at 2^53 and zu's integers go to 2^63, so a count that came back as a number would be a count you cannot trust. `{ bigIntMode: "number" }` is planned, will be documented with its precision hazard, and will never be the default. +- **INT64 is `bigint`.** By default, everywhere. A JavaScript number stops being exact at 2^53 and zu's integers go to 2^63, so a count that came back as a number would be a count you cannot trust. `{ bigIntMode: "number" }` asks for the other spelling, and the section below is what it costs. - **Nothing blocks the event loop.** Every native call runs on libuv's threadpool and hands back a promise before the statement has started. There is no synchronous variant, and the ones that arrive later will say in their own documentation that they belong in scripts, not servers. - **`await using` is the intended scoping.** A connection is `Symbol.asyncDispose`, and `close()` stays public for callers who cannot use the syntax. - **A failure is an ordinary `Error`.** Every `catch`, logger and rejection handler already knows what to do with one. What makes it a zu error is the fields, and none of them has to be parsed back out of the message: `code` is the GQLSTATUS and picks the branch, `condition` is the standard's own words for it, `line` and `column` and `excerpt` underline the token, and `retryable` decides whether a retry loop goes round again. A mistake this client caught before the engine saw it carries no `code` and is named `ZuUsageError`, so a caller mapping codes to branches can tell a missing code from one it does not recognize. `isZuError(caught)` is the exported guard for the `catch` clause, where the value is `unknown` and could be anything at all, and in TypeScript it narrows to the full shape. @@ -35,7 +35,7 @@ The rows are an array, so iterating them is `for (const row of rows)` and nothin ## What works today -`connect`, `query`, `exec`, `stream`, `close`, `dispose` and `await using`. Named parameters both ways, including lists, records and nesting. Every scalar the engine has, plus nodes, edges and paths with their tables named rather than numbered, and `ZuDate`, `ZuTime`, `ZuTimestamp` and `ZuDuration`. Read-only connections, memory and thread limits. An `AbortSignal` on any statement. The full error surface above, and `isZuError` to recognize it. Streaming, as an async iterable, as batches and as a Web Stream. Both module formats, typed separately. +`connect`, `query`, `exec`, `stream`, `close`, `dispose` and `await using`. Named parameters both ways, including lists, records and nesting. Every scalar the engine has, plus nodes, edges and paths with their tables named rather than numbered, and `ZuDate`, `ZuTime`, `ZuTimestamp` and `ZuDuration`. Read-only connections, memory and thread limits. `bigIntMode`, per statement or per connection. An `AbortSignal` on any statement. The full error surface above, and `isZuError` to recognize it. Streaming, as an async iterable, as batches and as a Web Stream. Both module formats, typed separately. Build it with `npm run build`, and run the suite with `npm test`. Nothing is published yet, so `npm i zudb` is not a thing you can type at anybody's terminal, but everything it will do is built and installed on every run of the release workflow. @@ -74,6 +74,26 @@ Between the statement and the loop sit two batches, which is the whole of the bu A statement that has to see every row before it can give one, which is `ORDER BY`, `DISTINCT` and the aggregates, runs whole and is handed over in batches afterwards. The loop is the same either way and `summary.streamed` is what tells them apart. +## Asking for numbers instead of bigints + +`bigIntMode` says how INT64 is spelled on the way out. It goes on one statement, or on a connection for all of them, and a statement on a connection that named one may still name the other: + +```ts +const conn = await connect("social.zu1", { bigIntMode: "number" }); +const rows = await conn.query<{ id: number }>(`MATCH (p:Person) RETURN p.id AS id`); +JSON.stringify(rows); // works, which it does not with bigints in it + +const exact = await conn.query(`MATCH (p:Person) RETURN count(*) AS n`, null, { + bigIntMode: "bigint", +}); +``` + +Two things are usually behind the ask. `JSON.stringify` throws on a `bigint`, so a row holding one cannot be handed straight to a response, and arithmetic on a `bigint` will not mix with a `number`, so every `+` in the reporting code needs a conversion. Numbers are also slightly cheaper to make: on 50k rows here, one INT64 column costs about 190ns a row as numbers against about 220ns as bigints. + +What is traded for that is worth stating plainly, because it is the reason this is never the default. Which integers a database holds is a property of the data and not of the program, so a query that returned numbers for every row of a test database is a query that can meet a larger one in production. This client refuses that row rather than rounding it: an integer past 2^53 raises a `ZuUsageError` naming the column and the value, so the failure is loud and local instead of an answer that is quietly off by one. It is still a failure that arrives at read time, on a machine that is not yours. + +The mode reaches the INT64 columns of a result and nothing else. A node's `offset`, an edge's `src`, `dst` and `ord`, and the nanosecond counts on the temporal classes stay `bigint` in both modes, because they are properties of classes the addon registers once rather than values a statement can respell. + ## Importing it, either way ```ts @@ -106,7 +126,7 @@ Anything outside that table has no binary and no source build to fall back on, s ## Still to come -`bigIntMode`. `toTemporal()` and `{ temporal: true }`, for the runtimes where Temporal is unflagged: it reached Stage 4 in March 2026 and is unflagged in Node 26, but Node 24 is still the active LTS and Safari is still behind a flag, which is why the stable types are the four classes above. Bun and Deno in CI, and the WASM build for the browser. +`toTemporal()` and `{ temporal: true }`, for the runtimes where Temporal is unflagged: it reached Stage 4 in March 2026 and is unflagged in Node 26, but Node 24 is still the active LTS and Safari is still behind a flag, which is why the stable types are the four classes above. Bun and Deno in CI, and the WASM build for the browser. ## Runtimes diff --git a/bench/query.mjs b/bench/query.mjs index 3f1c5dc..42845ea 100644 --- a/bench/query.mjs +++ b/bench/query.mjs @@ -73,6 +73,16 @@ const cases = [ per: 'row', run: () => conn.query('MATCH (p:person) RETURN p.id AS id'), }, + { + // The same column spelled as a number instead of a `bigint`, which + // is the cost the other mode is usually asked for: a `bigint` is an + // allocation and a double is not, so the difference between these + // two is what a program buys with the hazard it takes on. + name: 'scan, one INT64 as a number', + per: 'row', + run: () => + conn.query('MATCH (p:person) RETURN p.id AS id', null, { bigIntMode: 'number' }), + }, { name: 'scan, whole nodes', per: 'row', diff --git a/binding.d.cts b/binding.d.cts index ac674f7..b34343c 100644 --- a/binding.d.cts +++ b/binding.d.cts @@ -7,7 +7,9 @@ * INT64 is `bigint` and FLOAT is `number`, which is the one rule worth * learning before anything else here: a JavaScript number stops being * exact at 2^53 and zu's integers go to 2^63, so a count that came back - * as a number would be a count you cannot trust. + * as a number would be a count you cannot trust. `bigIntMode` changes + * that for a statement or for a connection, with the hazard it + * documents. */ export type ZuValue = | null @@ -148,6 +150,34 @@ export interface ZuStreamOptions extends ZuStatementOptions { readonly batchRows?: number } +/** + * How INT64 is spelled on the way out. + * + * `bigint` is the default and is the only one of the two that is always + * right, because zu's integers go to 2^63 and a JavaScript number stops + * telling one integer from the next at 2^53. + * + * `number` is for the program that has already decided its integers are + * small: ids that count in millions, a `count(*)` over a table that + * will never be one, a row about to be handed to `JSON.stringify`, + * which cannot serialize a `bigint` at all. It is worth knowing exactly + * what is being traded for that. Which integers a database holds is a + * property of the data and not of the program, so a query that returned + * numbers for every row of a test database is a query that can meet a + * larger one in production. This client refuses that row rather than + * rounding it, with a `ZuUsageError` naming the column and the value, + * so the failure is loud and local rather than an answer that is quietly + * off by one. It is still a failure that arrives at read time, on a + * machine that is not yours, which is why the default is the other one. + * + * The mode reaches the INT64 columns of a result and nothing else. A + * node's `offset`, an edge's `src`, `dst` and `ord`, and the nanosecond + * counts of the temporal classes stay `bigint` in both modes, because + * they are properties of classes the addon registers once and not + * values a statement can respell. + */ +export type ZuBigIntMode = 'bigint' | 'number' + /** * What a statement takes beside its parameters. * @@ -156,6 +186,12 @@ export interface ZuStreamOptions extends ZuStatementOptions { * is one nobody can read at a call site. */ export interface ZuStatementOptions { + /** + * How INT64 comes back from this statement. `bigint` unless the + * connection was opened with the other mode, and either way a + * statement may name the one it wants. + */ + readonly bigIntMode?: ZuBigIntMode /** * Stops the statement when it fires, through the same interrupt a * shell answers `Ctrl-C` with: the executor notices at the boundary it @@ -485,6 +521,12 @@ export interface ConnectOptions { memoryLimit?: bigint /** How many threads the executor may use. */ threads?: number + /** + * How INT64 comes back, for every statement on this connection. + * `bigint` unless it is said otherwise here, and a statement may + * say otherwise again for itself. + */ + bigIntMode?: ZuBigIntMode } /** The version of the client. */ diff --git a/src/conn.rs b/src/conn.rs index 3f92a54..5d928a5 100644 --- a/src/conn.rs +++ b/src/conn.rs @@ -28,7 +28,7 @@ use zudb::{Config, Database, DiagnosticRecord, Interrupt, ZuError}; use crate::cancel::Watch; use crate::error::{aborted, raise, usage}; use crate::stream::{self, Started, ZuCursor}; -use crate::value::{Names, from_js, to_js}; +use crate::value::{Ints, Shape, from_js, to_js}; /// What a connection can be opened with. /// @@ -45,6 +45,11 @@ pub struct ConnectOptions { pub memory_limit: Option, /// How many threads the executor may use. pub threads: Option, + /// How INT64 comes back, for every statement on this connection. + /// `bigint` unless it is said otherwise here, and a statement may + /// say otherwise again for itself. + #[napi(ts_type = "ZuBigIntMode")] + pub big_int_mode: Option, } /// One connection to one database. @@ -76,6 +81,9 @@ pub struct Connection { /// to the session and is the same one for the connection's whole /// life, so once is enough. interrupt: Interrupt, + /// How this connection's statements spell INT64, unless one of them + /// asks for the other spelling. + ints: Ints, path: String, read_only: bool, } @@ -96,7 +104,7 @@ pub struct ConnectTask { } impl<'task> ScopedTask<'task> for ConnectTask { - type Output = std::result::Result; + type Output = std::result::Result; type JsValue = ClassInstance<'task, Connection>; fn compute(&mut self) -> Result { @@ -105,6 +113,20 @@ impl<'task> ScopedTask<'task> for ConnectTask { .as_ref() .and_then(|options| options.read_only) .unwrap_or(false); + // Before the open, because a mode nobody can spell is a mistake + // in the calling program and a database created on the way to + // finding it out is a file the caller did not ask for. + let ints = match self + .options + .as_ref() + .and_then(|options| options.big_int_mode.as_deref()) + { + Some(mode) => match Ints::named(mode) { + Ok(ints) => ints, + Err(message) => return Ok(Err(Failure::Usage(message))), + }, + None => Ints::default(), + }; let mut config = Config::new().read_only(read_only); if let Some(options) = &self.options { if let Some(limit) = &options.memory_limit { @@ -115,15 +137,18 @@ impl<'task> ScopedTask<'task> for ConnectTask { config = config.threads(threads as usize); } } - Ok(open(PathBuf::from(&self.path), read_only, config)) + Ok(open(PathBuf::from(&self.path), read_only, config) + .map(|opened| Opened { ints, ..opened }) + .map_err(Failure::Engine)) } fn resolve(&mut self, env: &'task Env, output: Self::Output) -> Result { - let opened = output.map_err(|err| raise(env, err))?; + let opened = output.map_err(|failure| failed(env, failure, None))?; let mut instance = Connection { interrupt: opened.conn.interrupt(), inner: Arc::new(Mutex::new(Some(opened.conn))), alive: Arc::new(AtomicBool::new(true)), + ints: opened.ints, path: opened.path, read_only: opened.read_only, } @@ -158,6 +183,7 @@ fn wire_disposal(env: &Env, instance: &mut ClassInstance<'_, Connection>) -> Res pub struct Opened { conn: zudb::Connection, + ints: Ints, path: String, read_only: bool, } @@ -178,6 +204,7 @@ fn open(path: PathBuf, read_only: bool, config: Config) -> std::result::Result (params, batch_rows, watch, None), - Err(message) => (Vec::new(), None, None, Some(message)), + let (params, ints, batch_rows, watch, refused) = match bound { + Ok((params, ints, batch_rows, watch)) => (params, ints, batch_rows, watch, None), + Err(message) => (Vec::new(), self.ints, None, None, Some(message)), }; stream::open( Started { @@ -285,6 +314,7 @@ impl Connection { alive: Arc::clone(&self.alive), statement, params, + ints, batch_rows, guard: watch.as_ref().map(Watch::guard), }, @@ -313,19 +343,25 @@ impl Connection { // the thread that owns the runtime may do. So is adding the // listener the signal is watched through. let bound = if self.alive.load(Ordering::Acquire) { - bind(env, params) - .and_then(|params| Ok((params, watch(env, options, self.interrupt.clone())?))) + int_mode(options.as_ref(), self.ints).and_then(|ints| { + Ok(( + bind(env, params)?, + ints, + watch(env, options, self.interrupt.clone())?, + )) + }) } else { Err(CLOSED.to_string()) }; - let (params, watch, refused) = match bound { - Ok((params, watch)) => (params, watch, None), - Err(message) => (Vec::new(), None, Some(message)), + let (params, ints, watch, refused) = match bound { + Ok((params, ints, watch)) => (params, ints, watch, None), + Err(message) => (Vec::new(), self.ints, None, Some(message)), }; QueryTask { inner: Arc::clone(&self.inner), statement, params, + ints, watch, refused, } @@ -488,6 +524,30 @@ fn batch_rows(options: Option<&Object<'_>>) -> std::result::Result, } } +/// Reads `options.bigIntMode`, which is how this statement spells the +/// INT64s it gives back. +/// +/// Absent means the connection's own, which is `bigint` unless the +/// program said otherwise when it connected. A statement may say +/// otherwise again, because the reason to ask for numbers is usually +/// one query whose rows are about to be serialized rather than a whole +/// program's worth of them. +fn int_mode(options: Option<&Object<'_>>, connection: Ints) -> std::result::Result { + let Some(options) = options else { + return Ok(connection); + }; + let mode: Unknown<'_> = options + .get_named_property("bigIntMode") + .map_err(|err| err.reason)?; + match mode.get_type().map_err(|err| err.reason)? { + ValueType::Undefined | ValueType::Null => Ok(connection), + ValueType::String => Ints::named(&String::from_unknown(mode).map_err(|err| err.reason)?), + other => Err(format!( + "bigIntMode is a {other}, and a mode is named by a string" + )), + } +} + /// Reads the parameter object into the values the engine binds. /// /// Every failure comes back as the message to refuse the call with, @@ -516,6 +576,8 @@ pub struct QueryTask { inner: Arc>>, statement: String, params: Vec<(String, Value)>, + /// How this statement spells the INT64s it gives back. + ints: Ints, /// The signal watching this statement, when the caller gave one. watch: Option, /// Why this statement is not going to run, when it is not. @@ -528,7 +590,7 @@ impl QueryTask { /// The names are read while the lock is held, because a catalog /// borrowed from the connection cannot outlive it and a result that /// names its tables has to carry them. - fn run(&mut self) -> std::result::Result<(QueryResult, Names), Failure> { + fn run(&mut self) -> std::result::Result<(QueryResult, Shape), Failure> { if let Some(message) = self.refused.take() { return Err(Failure::Usage(message)); } @@ -564,7 +626,7 @@ impl QueryTask { .iter() .map(|(name, value)| (name.as_str(), value.clone())) .collect(); - let names = Names::of(conn.session_mut().catalog()); + let shape = Shape::of(conn.session_mut().catalog(), self.ints); let result = conn.query_with(&self.statement, ¶ms); if let Some(watch) = &self.watch { watch.leave(); @@ -577,7 +639,7 @@ impl QueryTask { return Err(Failure::Aborted); } } - Ok((result?, names)) + Ok((result?, shape)) } /// The exception this rejects the caller's promise with. @@ -595,7 +657,7 @@ impl QueryTask { } impl<'task> ScopedTask<'task> for QueryTask { - type Output = std::result::Result<(QueryResult, Names), Failure>; + type Output = std::result::Result<(QueryResult, Shape), Failure>; type JsValue = Array<'task>; fn compute(&mut self) -> Result { @@ -603,8 +665,8 @@ impl<'task> ScopedTask<'task> for QueryTask { } fn resolve(&mut self, env: &'task Env, output: Self::Output) -> Result { - let (result, names) = output.map_err(|failure| self.failed(env, failure))?; - rows(env, &result, &names) + let (result, shape) = output.map_err(|failure| self.failed(env, failure))?; + rows(env, &result, &shape) } fn finally(mut self, env: Env) -> Result<()> { @@ -623,12 +685,12 @@ impl<'task> ScopedTask<'task> for QueryTask { /// that does not. Out of the way means not enumerable, so that the /// array spreads, stringifies, deep-equals a plain array and answers /// `Object.keys` as though they were not there at all. -fn rows<'env>(env: &'env Env, result: &QueryResult, names: &Names) -> Result> { +fn rows<'env>(env: &'env Env, result: &QueryResult, shape: &Shape) -> Result> { let mut array = env.create_array(result.rows.len() as u32)?; for (ix, row) in result.rows.iter().enumerate() { let mut object = Object::new(env)?; for (column, value) in result.columns.iter().zip(row) { - object.set(column.as_str(), to_js(env, value, names)?)?; + object.set(column.as_str(), to_js(env, column, value, shape)?)?; } array.set(ix as u32, object)?; } diff --git a/src/stream.rs b/src/stream.rs index 5335677..ab9ac21 100644 --- a/src/stream.rs +++ b/src/stream.rs @@ -35,7 +35,7 @@ use zudb::{Batch, Flow, Streamed, ZuError}; use crate::cancel::{Guard, Watch}; use crate::conn::{CLOSED, Failure, beside, failed, notices}; -use crate::value::{Names, to_js}; +use crate::value::{Ints, Shape, to_js}; /// How many batches may sit between the statement and the reader. /// @@ -65,7 +65,7 @@ const POISONED: &str = "the stream was left in an unknown state by a thread that /// them instead of a copy. pub struct Head { columns: Vec, - names: Arc, + shape: Arc, } /// What the thread running the statement sends back. @@ -197,6 +197,7 @@ pub struct Started { pub alive: Arc, pub statement: String, pub params: Vec<(String, Value)>, + pub ints: Ints, pub batch_rows: Option, pub guard: Option, } @@ -489,7 +490,7 @@ fn batch<'env>(env: &'env Env, head: &Head, rows: &[Vec]) -> Result = self .params .iter() @@ -620,7 +621,7 @@ impl Started { let head = Arc::clone(head.get_or_insert_with(|| { Arc::new(Head { columns: batch.columns().to_vec(), - names: Arc::clone(&names), + shape: Arc::clone(&shape), }) })); Ok(self.hand(live, head, batch.rows().to_vec())) diff --git a/src/value.rs b/src/value.rs index 8ac4886..824250b 100644 --- a/src/value.rs +++ b/src/value.rs @@ -6,13 +6,17 @@ //! a class with named fields rather than a tuple whose third element //! is the ordinal if you remember the order. //! -//! INT64 is `bigint`, always, which is ADR 0003 in the engine +//! INT64 is `bigint` by default, which is ADR 0003 in the engine //! repository. A JavaScript number is an IEEE double and stops being //! exact at 2^53, zu's integers go to 2^63, and `count(*)` over a //! large graph gets there on its own. A client that returns a number //! here is a client whose users file "the id came back wrong" a year //! later. //! +//! A caller may ask for numbers anyway, with `bigIntMode: "number"`, +//! and then an integer outside what a double holds exactly is refused +//! rather than rounded. See [`Ints`]. +//! //! Going the other way a `number` that is a whole number binds as //! INT64 and one that is not binds as FLOAT, because `{ id: 1 }` is //! what a caller writes and refusing it would be pedantry. A `bigint` @@ -27,6 +31,62 @@ use zu_common::{DurationKind, Temporal}; use zudb::query::Value; use zudb::zu1::catalog::Catalog; +use crate::error::usage; + +/// How an INT64 is spelled on the way out. +/// +/// `bigint` is the default and the only one that is always right. The +/// other exists because a program that already knows its integers are +/// small spends a `Number(...)` on every one of them otherwise, and +/// because a `bigint` has no JSON spelling, so a result holding one +/// cannot be handed to `JSON.stringify` at all. +/// +/// The hazard of the second is the whole reason it is not the default: +/// which integers a database holds is a property of the data and not of +/// the program, so a query that worked on every row of a test database +/// is a query that can fail on the one row where an id passed 2^53. +/// This client refuses that row rather than rounding it, which turns a +/// wrong answer into a failure that names the column, and that is the +/// most a client can do about a decision the caller has already made. +#[derive(Clone, Copy, Default, PartialEq, Eq)] +pub enum Ints { + #[default] + BigInt, + Number, +} + +impl Ints { + /// The mode `bigIntMode` names, or what is wrong with what it named. + pub fn named(mode: &str) -> std::result::Result { + match mode { + "bigint" => Ok(Ints::BigInt), + "number" => Ok(Ints::Number), + other => Err(format!( + "bigIntMode is \"{other}\", and the modes are \"bigint\" and \"number\"" + )), + } + } +} + +/// What a result needs on the way out. +/// +/// The table names are the statement's, and so is the spelling of its +/// integers, so both are settled once where the connection is held and +/// then read by every value of every row. +pub struct Shape { + names: Names, + ints: Ints, +} + +impl Shape { + pub fn of(catalog: &Catalog, ints: Ints) -> Shape { + Shape { + names: Names::of(catalog), + ints, + } + } +} + /// What the tables in a result are called. /// /// A node value carries the id of the table it came from and nothing @@ -357,14 +417,25 @@ fn day_time(nanos: i64) -> ZuDuration { } /// Turns an engine value into the JavaScript value it is. -pub fn to_js<'env>(env: &'env Env, value: &Value, names: &Names) -> Result> { +/// +/// `column` is the name the value arrived under, carried the whole way +/// down for the same reason [`from_js`] carries one up: the only thing +/// that can fail here is an integer that will not fit the spelling the +/// caller asked for, and a caller told which column that was can act on +/// it, while one told the number alone has to go looking. +pub fn to_js<'env>( + env: &'env Env, + column: &str, + value: &Value, + shape: &Shape, +) -> Result> { match value { Value::Null => Null.into_unknown(env), Value::Bool(b) => (*b).into_unknown(env), - Value::Int(n) => BigInt::from(*n).into_unknown(env), + Value::Int(n) => int(env, column, *n, shape.ints), Value::Float(f) => (*f).into_unknown(env), Value::Str(s) => s.as_str().into_unknown(env), - Value::Node { table, offset } => node(*table, *offset, names) + Value::Node { table, offset } => node(*table, *offset, &shape.names) .into_instance(env)? .into_unknown(env), Value::Rel { @@ -372,25 +443,25 @@ pub fn to_js<'env>(env: &'env Env, value: &Value, names: &Names) -> Result rel(*table, *src, *dst, *ord, names) + } => rel(*table, *src, *dst, *ord, &shape.names) .into_instance(env)? .into_unknown(env), Value::List(items) => { let mut array = env.create_array(items.len() as u32)?; for (ix, item) in items.iter().enumerate() { - array.set(ix as u32, to_js(env, item, names)?)?; + array.set(ix as u32, to_js(env, column, item, shape)?)?; } array.into_unknown(env) } Value::Record(fields) => { let mut object = Object::new(env)?; for (name, field) in fields { - object.set(name.as_str(), to_js(env, field, names)?)?; + object.set(name.as_str(), to_js(env, column, field, shape)?)?; } object.into_unknown(env) } Value::Temporal(t) => temporal(env, *t), - Value::Path(walk) => path(env, walk, names), + Value::Path(walk) => path(env, walk, &shape.names), // The three the executor keeps to itself. A chain is settled // into an edge list before any value leaves the pipeline, and a // graph or a binding table is a handle to something that has no @@ -401,6 +472,36 @@ pub fn to_js<'env>(env: &'env Env, value: &Value, names: &Names) -> Result(env: &'env Env, column: &str, n: i64, ints: Ints) -> Result> { + match ints { + Ints::BigInt => BigInt::from(n).into_unknown(env), + Ints::Number if !(-EXACT..=EXACT).contains(&n) => Err(usage( + env, + format!( + "column {column} holds {n}, which a JavaScript number cannot tell from \ + its neighbours, and this statement asked for bigIntMode: \"number\"" + ), + )), + Ints::Number => (n as f64).into_unknown(env), + } +} + fn node(table: u32, offset: u64, names: &Names) -> ZuNode { ZuNode { table: names.node(table), diff --git a/test/bigint.test.mjs b/test/bigint.test.mjs new file mode 100644 index 0000000..a394c3e --- /dev/null +++ b/test/bigint.test.mjs @@ -0,0 +1,189 @@ +// How INT64 is spelled on the way out. +// +// The default is a `bigint` and that is not in question here. What is +// in question is the other mode: that a program has to ask for it, that +// asking is possible at the two places a program would look for it, and +// that the integer it cannot hold is refused rather than rounded, which +// is the whole of the difference between an opt-in and a trap. + +import assert from 'node:assert/strict' +import { existsSync } from 'node:fs' +import { join } from 'node:path' +import test from 'node:test' + +import { connect } from 'zudb' + +import { fresh, isZuError, twoPeople } from './helper.mjs' + +// 2^53 - 1, the largest integer a JavaScript number tells from the next +// one, and the first two past it. +const EXACT = 9007199254740991n +const OVER = 9007199254740992n +const FURTHER = 9007199254740993n + +async function one(conn, value, options) { + const rows = await conn.query('RETURN $v AS v', { v: value }, options) + return rows[0].v +} + +test('an INT64 is a bigint when nobody says otherwise', async (t) => { + const { conn } = await twoPeople(t) + + const rows = await conn.query('MATCH (p:person) RETURN p.id AS id ORDER BY id') + assert.equal(typeof rows[0].id, 'bigint') + // Named explicitly, which is the same thing and worth being able to + // write down: a program can say what it is relying on. + assert.equal(typeof (await one(conn, 1n, { bigIntMode: 'bigint' })), 'bigint') +}) + +test('a statement that asks for numbers gets numbers', async (t) => { + const { conn } = await twoPeople(t) + + const rows = await conn.query('MATCH (p:person) RETURN p.id AS id ORDER BY id', null, { + bigIntMode: 'number', + }) + assert.deepEqual( + rows.map((row) => row.id), + [1, 2], + ) + assert.equal(typeof rows[0].id, 'number') + // The statement beside it is unaffected, because the mode belongs to + // the call and not to the connection it was made on. + assert.equal(typeof (await one(conn, 1n)), 'bigint') +}) + +test('a connection opened for numbers gives them to every statement', async (t) => { + const { conn } = await fresh(t, { bigIntMode: 'number' }) + await conn.exec("INSERT (p:person {id: 7, name: 'ada'})") + + const rows = await conn.query('MATCH (p:person) RETURN p.id AS id') + assert.equal(rows[0].id, 7) + assert.equal(typeof rows[0].id, 'number') + + // And a statement on it can still ask for the other one, which is the + // direction that matters: the one query that counts something large + // should not have to be run on a connection of its own. + assert.equal(await one(conn, EXACT + 10n, { bigIntMode: 'bigint' }), 9007199254741001n) +}) + +test('an integer a number cannot hold is refused, not rounded', async (t) => { + const { conn } = await fresh(t) + + assert.equal(await one(conn, EXACT, { bigIntMode: 'number' }), 9007199254740991) + assert.equal(await one(conn, -EXACT, { bigIntMode: 'number' }), -9007199254740991) + + for (const value of [OVER, FURTHER, -OVER, 9223372036854775807n]) { + await assert.rejects( + () => one(conn, value, { bigIntMode: 'number' }), + (err) => { + assert.ok(isZuError(err, 'ZuUsageError'), `${err.name} for ${value}`) + // The column and the value, because a caller with twelve + // columns has to know which one to widen and a caller deciding + // whether to widen at all has to know by how much. + assert.match(err.message, /column v holds/) + assert.match(err.message, new RegExp(String(value))) + assert.match(err.message, /bigIntMode/) + return true + }, + ) + } + + // The mistake is the mode and not the connection, so the next + // statement runs. + assert.equal(await one(conn, FURTHER), 9007199254740993n) +}) + +test('the mode reaches the integers inside a list and a record', async (t) => { + const { conn } = await fresh(t) + + const rows = await conn.query('RETURN [1, 2] AS xs, {a: 3} AS r', null, { + bigIntMode: 'number', + }) + assert.deepEqual(rows[0].xs, [1, 2]) + assert.deepEqual(rows[0].r, { a: 3 }) + + await assert.rejects( + () => conn.query('RETURN [$v] AS xs', { v: FURTHER }, { bigIntMode: 'number' }), + (err) => isZuError(err, 'ZuUsageError') && /column xs holds/.test(err.message), + ) +}) + +test('a float is a number in both modes, and a string is a string', async (t) => { + const { conn } = await fresh(t) + + for (const bigIntMode of ['bigint', 'number']) { + const rows = await conn.query("RETURN 1.5 AS f, 'ada' AS s, true AS b", null, { bigIntMode }) + assert.deepEqual({ ...rows[0] }, { f: 1.5, s: 'ada', b: true }) + } +}) + +test('a node keeps its bigint offset in either mode', async (t) => { + const { conn } = await twoPeople(t) + + const rows = await conn.query('MATCH (p:person) RETURN p AS p ORDER BY p.id', null, { + bigIntMode: 'number', + }) + // The classes are registered once by the addon and their getters + // cannot change shape per statement, so this is the documented edge + // of the mode rather than an oversight. + assert.equal(typeof rows[0].p.offset, 'bigint') +}) + +test('a stream spells its integers the way it was asked to', async (t) => { + const { conn } = await twoPeople(t) + + const stream = conn.stream('MATCH (p:person) RETURN p.id AS id ORDER BY id', null, { + bigIntMode: 'number', + }) + const seen = [] + for await (const row of stream) seen.push(row.id) + assert.deepEqual(seen, [1, 2]) + + // And the refusal arrives where every other failure of a stream + // arrives, which is the read that found the row. + const over = conn.stream('RETURN $v AS v', { v: FURTHER }, { bigIntMode: 'number' }) + await assert.rejects( + async () => { + for await (const row of over) void row + }, + (err) => isZuError(err, 'ZuUsageError') && /column v holds/.test(err.message), + ) +}) + +test('rows of numbers are what JSON.stringify can take', async (t) => { + const { conn } = await twoPeople(t) + + // The reason a program asks for this mode as often as any other: a + // `bigint` has no JSON spelling at all, so a row holding one throws + // on the way out of a request handler. + const rows = await conn.query('MATCH (p:person) RETURN p.id AS id, p.name AS name ORDER BY id', null, { + bigIntMode: 'number', + }) + assert.equal(JSON.stringify(rows), '[{"id":1,"name":"ada"},{"id":2,"name":"zoe"}]') + const bigints = await conn.query('RETURN 1 AS n') + assert.throws(() => JSON.stringify(bigints), TypeError) +}) + +test('a mode nobody can spell is refused wherever it was named', async (t) => { + const { conn, dir } = await twoPeople(t) + + for (const bigIntMode of ['string', 'BigInt', '', 5, {}]) { + await assert.rejects( + () => conn.query('RETURN 1 AS n', null, { bigIntMode }), + (err) => { + assert.ok(isZuError(err, 'ZuUsageError'), `${JSON.stringify(bigIntMode)} was accepted`) + assert.match(err.message, /bigIntMode/) + return true + }, + ) + } + + const path = join(dir, 'never.zu1') + await assert.rejects( + () => connect(path, { bigIntMode: 'strings' }), + (err) => isZuError(err, 'ZuUsageError') && /bigIntMode/.test(err.message), + ) + // The mode is read before the database is opened, so a typo in the + // options does not leave a database behind that nobody asked for. + assert.equal(existsSync(path), false) +}) diff --git a/test/types/cjs.cts b/test/types/cjs.cts index 91b52a4..e094d86 100644 --- a/test/types/cjs.cts +++ b/test/types/cjs.cts @@ -4,6 +4,19 @@ import { connect, isZuError, type ZuParam, type ZuStream } from 'zudb' +export async function total(path: string): Promise { + // The mode is on the statement here, so the rows it gives back are + // numbers and adding them up needs no conversion. A `bigint` row type + // over the same call would not compile, which is the point of writing + // it down. + const conn = await connect(path, { bigIntMode: 'bigint' }) + const rows = await conn.query<{ n: number }>('MATCH (p:person) RETURN count(*) AS n', null, { + bigIntMode: 'number', + }) + conn.close() + return rows.reduce((sum, row) => sum + row.n, 0) +} + export async function insert(path: string, values: Record): Promise { const conn = await connect(path) try { diff --git a/test/types/esm.mts b/test/types/esm.mts index 8a26bd2..2a73372 100644 --- a/test/types/esm.mts +++ b/test/types/esm.mts @@ -7,6 +7,7 @@ import { isZuError, ZuDate, type ZuBatch, + type ZuBigIntMode, type ZuError, type ZuRows, type ZuStream, @@ -75,6 +76,20 @@ export async function counted(path: string): Promise { return rows } +export async function serialized(path: string, mode: ZuBigIntMode): Promise { + // A connection with a mode of its own, and a statement that names one + // for itself. The row type is the caller's either way, which is the + // part TypeScript cannot check for them: a mode is a string at + // runtime and `id` is whatever they said it was. + await using conn = await connect(path, { bigIntMode: mode }) + const rows: ZuRows<{ id: number; name: string }> = await conn.query( + 'MATCH (p:person) RETURN p.id AS id, p.name AS name', + null, + { bigIntMode: 'number' }, + ) + return JSON.stringify(rows.map((row) => ({ ...row, id: row.id + 1 }))) +} + export function retryable(caught: unknown): boolean { // `catch` gives `unknown`, and the guard is what narrows it. Reading // `caught.retryable` without it does not compile. diff --git a/types/header.d.ts b/types/header.d.ts index b823edd..a39f21c 100644 --- a/types/header.d.ts +++ b/types/header.d.ts @@ -7,7 +7,9 @@ * INT64 is `bigint` and FLOAT is `number`, which is the one rule worth * learning before anything else here: a JavaScript number stops being * exact at 2^53 and zu's integers go to 2^63, so a count that came back - * as a number would be a count you cannot trust. + * as a number would be a count you cannot trust. `bigIntMode` changes + * that for a statement or for a connection, with the hazard it + * documents. */ export type ZuValue = | null @@ -148,6 +150,34 @@ export interface ZuStreamOptions extends ZuStatementOptions { readonly batchRows?: number } +/** + * How INT64 is spelled on the way out. + * + * `bigint` is the default and is the only one of the two that is always + * right, because zu's integers go to 2^63 and a JavaScript number stops + * telling one integer from the next at 2^53. + * + * `number` is for the program that has already decided its integers are + * small: ids that count in millions, a `count(*)` over a table that + * will never be one, a row about to be handed to `JSON.stringify`, + * which cannot serialize a `bigint` at all. It is worth knowing exactly + * what is being traded for that. Which integers a database holds is a + * property of the data and not of the program, so a query that returned + * numbers for every row of a test database is a query that can meet a + * larger one in production. This client refuses that row rather than + * rounding it, with a `ZuUsageError` naming the column and the value, + * so the failure is loud and local rather than an answer that is quietly + * off by one. It is still a failure that arrives at read time, on a + * machine that is not yours, which is why the default is the other one. + * + * The mode reaches the INT64 columns of a result and nothing else. A + * node's `offset`, an edge's `src`, `dst` and `ord`, and the nanosecond + * counts of the temporal classes stay `bigint` in both modes, because + * they are properties of classes the addon registers once and not + * values a statement can respell. + */ +export type ZuBigIntMode = 'bigint' | 'number' + /** * What a statement takes beside its parameters. * @@ -156,6 +186,12 @@ export interface ZuStreamOptions extends ZuStatementOptions { * is one nobody can read at a call site. */ export interface ZuStatementOptions { + /** + * How INT64 comes back from this statement. `bigint` unless the + * connection was opened with the other mode, and either way a + * statement may name the one it wants. + */ + readonly bigIntMode?: ZuBigIntMode /** * Stops the statement when it fires, through the same interrupt a * shell answers `Ctrl-C` with: the executor notices at the boundary it