# DianaDB

> DianaDB is the first fully featured DBMS written in Node.js — a column-oriented NoSQL database in TypeScript. It stores documents in a columnar format where every field acts as an index. It supports schema-enforced collections, spatial and time-based queries, cross-database lookups, materialized views that update on the fly, ACID transactions with document-level locking, change subscriptions, migrations, TLS/mTLS, and S3-compatible backup storage.

DianaDB is authored and maintained by Data Bikers Limited (Hong Kong). It is conceptually close to MongoDB — the query language is similar to MQL — but architecturally distinct: columnar storage, mandatory document schemas, and a TypeScript-native ODM.

Current release: Server v3.0.0 / ODM v3.0.0. Server 3.0.0 requires ODM >= 3.0.0 for subscriptions; ODM 3.0.0 also works with older servers, so upgrade the ODM first.

---

## What is DianaDB

DianaDB is a NoSQL, column-oriented database. It operates on documents and collections, but stores data in a columnar format: each document is decomposed into separate columns, each column tailored to a specific data type with its own storage and processing logic. Data integrity is enforced through document schemas that define the expected structure of each collection. Because of this layout every field in a document behaves like an index.

DianaDB supports relationships between documents across collections, including remove triggers similar to relational cascades. It supports cross-database lookups on the same server. It provides a `TIME` type with utilities for calendar and scheduling queries, and a `POSITION` type for spatial queries. It provides ACID transactions, a built-in migration store, and subscriptions to database changes per collection or globally.

---

## Server Installation

### Linux — Debian

```bash
sudo echo "deb [arch=amd64 trusted=yes] https://dist.databikers.com stable main" | sudo tee /etc/apt/sources.list.d/diana-db.list
sudo apt-get update
sudo apt-get install -y diana-db
```

```bash
sudo service diana-db [status|start|stop]
```

### Linux — Alpine

```bash
wget -q -O /etc/apk/keys/sgerrand.rsa.pub https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub
wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.35-r1/glibc-2.35-r1.apk
apk add glibc-2.35-r1.apk
wget https://dist.databikers.com/x86_64/diana-db-3.0.0-r0.apk
apk add --allow-untrusted ./diana-db-3.0.0-r0.apk
```

```bash
sudo rc-service diana-db [status|start|stop]
```

No default database user is created. You must add users yourself.

### CLI Commands

The `diana-db` CLI manages the daemon, users, access, and backups.

| Command | Flags | Description |
|---|---|---|
| `--settings` | `-s` | Displays current settings |
| `--version` | `-v` | Displays current version |
| `start` | `-c <conf>` | Stops any running instance, then launches `diana-db-server` in the background |
| `stop` | `-c <conf>` | Kills the running `diana-db-server` process |
| `add-user` | `-u <user> -p <pass>` | Creates a user. Only the key derived from the password is stored (since 3.0.0) |
| `remove-user` | `-u <user>` | Removes the user |
| `grant-access` | `-u <user> -db <db:access>` | Grants database access. Access levels: `1` read, `2` write, `3` manage (`0` no access) |
| `remove-access` | `-u <user> -db <db>` | Removes database access |
| `list-backups` | – | Displays the backup list |
| `create-backup` | `-n <name>` | Creates `[name].didb`. Server must be stopped |
| `apply-backup` | `-n <name>` | Applies `[name].didb` and deletes the write-ahead log files. Server must be stopped |
| `remove-backup` | `-n <name>` | Removes `[name].didb` |

```bash
diana-db start -c /path/to/diana-db.conf
diana-db add-user -u alice -p s3cr3t
diana-db grant-access -u alice -db main:3
diana-db remove-access -u alice -db main
diana-db create-backup -n test_backup
```

### Config File (`/etc/diana-db/diana-db.conf`)

Simple `key=value` syntax; `#` starts a comment. Each key maps to an environment variable.

| Key | Environment Variable | Type | Description |
|---|---|---|---|
| `port` | `DIANA_DB_PORT` | Integer | Listening TCP port (34567) |
| `dump_create_interval` | `DIANA_DB_DUMP_CREATE_INTERVAL` | `hour` or `day` | How often to back up the current dump (default `day`); the first backup is taken one interval after start |
| `logs_directory` | `DIANA_DB_LOG_DIRECTORY` | Path | Server log directory |
| `dump_directory` | `DIANA_DB_DUMP_DIRECTORY` | Path | Dump directory |
| `logs_ttl_value` | `DIANA_DB_LOG_TTL` | String, e.g. `7d` | Log retention (default `7d`) |
| `backup_name` | `DIANA_DB_BACKUP_NAME` | String | Default backup filename when `-n` is omitted |
| `current_dump_name` | `DIANA_DB_DUMP_CURRENT_NAME` | String | Filename for the latest dump |
| `store_strategy` | `DIANA_DB_STORE_STRATEGY` | String | `local` (default) or `s3` |
| `store_s3_endpoint` | `DIANA_DB_STORE_S3_ENDPOINT` | URL | Required when `store_strategy=s3` |
| `store_s3_bucket` | `DIANA_DB_STORE_S3_BUCKET` | String | Required when `store_strategy=s3` |
| `store_s3_region` | `DIANA_DB_STORE_S3_REGION` | String | Required when `store_strategy=s3` |
| `store_s3_access_key_id` | `DIANA_DB_STORE_S3_ACCESS_KEY_ID` | String | Required when `store_strategy=s3` |
| `store_s3_secret_access_key` | `DIANA_DB_STORE_S3_SECRET_ACCESS_KEY` | String | Required when `store_strategy=s3` |
| `ssl_cert_file` | `DIANA_DB_SSL_CERT_FILE` | Path | TLS certificate. Enables TLS together with `ssl_key_file` |
| `ssl_key_file` | `DIANA_DB_SSL_KEY_FILE` | Path | TLS private key |
| `ssl_ca_file` | `DIANA_DB_SSL_CA_FILE` | Path | CA used to verify client certificates (mTLS) |
| `ssl_request_cert` | `DIANA_DB_SSL_REQUEST_CERT` | Boolean (0/1) | Request a client certificate (mTLS) |
| `ssl_reject_unauthorized` | `DIANA_DB_SSL_REJECT_UNAUTHORISED` | Boolean (0/1) | Reject certificates that fail verification |
| `transactions_min_auto_rollback_value` | `DIANA_DB_TRANSACTION_AUTOROLLBACK_MIN` | Integer (ms) | Smallest `autoRollBackAfterMs` a client may request (default 1000) |
| `transactions_max_auto_rollback_value` | `DIANA_DB_TRANSACTION_AUTOROLLBACK_MAX` | Integer (ms) | Largest `autoRollBackAfterMs` a client may request, and the timeout of transactions started without one (default 60000) |
| `wal` | `DIANA_DB_WAL` | Boolean (0/1) | Write-ahead log, `local` storage only. On by default |
| `wal_fsync` | `DIANA_DB_WAL_FSYNC` | Boolean (0/1) | fsync the log before answering (default 1). `0` trades crash safety for latency |

Warning: before switching to `store_strategy=s3`, manually upload the existing `users` file and the current data (`/var/lib/diana-db/*`) to the target bucket. DianaDB does not migrate local data automatically; switching without uploading starts the server with an empty dataset.

### Durability: Snapshots and the Write-Ahead Log

The server keeps data in memory and saves a snapshot of every database to `dump_directory` about every 37 seconds. Since 3.0.0 it also keeps a write-ahead log there (`wal-*.log`):

- Every change (inserted documents, final values of updates, removed ids, schema, view and migration changes) is appended to the log. Changes from one event-loop turn are written and fsynced together, and the server answers those requests only after that, so an acknowledged write is on disk.
- When a snapshot is taken the log switches to a new file; files the snapshot covers are deleted once it is on disk.
- At start-up the snapshot is restored and the remaining log is replayed (idempotent), then a fresh snapshot is written.
- Snapshots are written to a temporary file, fsynced and renamed, so a crash or full disk during a save never leaves a truncated dump. `SIGINT`/`SIGTERM` save every database before exit.
- If the log cannot be written, the change stays applied in memory and the client receives a `DURABILITY` error.
- The log is used with `store_strategy=local` only; with `s3`, writes since the last snapshot can be lost in a crash.

---

## Docker Installation

### Debian

```bash
# entrypoint.sh
#!/bin/sh
set -e
touch /var/lib/diana-db/current.didb
touch /var/lib/diana-db/users
/usr/bin/diana-db add-user -u admin -p admin
/usr/bin/diana-db grant-access -u admin -db test:3
/usr/bin/diana-db grant-access -u admin -db another_db:3
/usr/bin/diana-db-server
```

```dockerfile
FROM databikers/diana-db:debian
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
```

### Alpine

```bash
# entrypoint.sh
#!/bin/sh
set -e
/usr/bin/diana-db add-user -u admin -p admin
/usr/bin/diana-db grant-access -u admin -db test:3
/usr/bin/diana-db grant-access -u admin -db another_db:3
exec /usr/bin/diana-db-server
```

```dockerfile
FROM databikers/diana-db:alpine
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
```

### Run

```bash
docker build -t local-diana-db .
docker run -p 34567:34567 local-diana-db
```

### Docker Compose

```yaml
version: '3.8'
services:
  diana-db:
    image: local-diana-db
    ports:
      - "34567:34567"
    volumes:
      - ./data/etc:/etc/diana-db
      - ./data/lib:/var/lib/diana-db
      - ./data/log:/var/log/diana-db
    restart: unless-stopped
    container_name: diana-db
    entrypoint: ["/entrypoint.sh"]
```

---

## Security

- **TLS / mTLS.** Connections can be encrypted in transit. Optional mutual TLS lets the server verify client identity via certificates.
- **The password is never sent over the wire.** After a client identifies itself with a username, every request and response is encrypted with AES-256-GCM using a key derived from the user's password (SHA-256). This can be layered under TLS.
- **Stored keys, not passwords.** Since 3.0.0 the `users` file stores only the derived key; users created by older versions keep working.
- **Authenticated subscribers.** Subscriber connections send `user:<name>:subscriber:<token>`, where the token is the encrypted text `user:<name>:subscriber:<UTC date YYYY-MM-DD>`. The server accepts yesterday to tomorrow and closes subscriber connections without a valid token. Updates are only sent for databases the user can read.
- **Segregated access control.** Access is granted per user and per database via `diana-db grant-access`, and checked before a database is looked up, so requests can't create databases their user has no rights to. `$lookup` into another database needs read access to it.
- **No document data in logs.** Server logs record request metadata only (request id, user, action, database, collection, timing, error).
- **Private storage.** Dumps and backups can be written to a private S3-compatible bucket rather than local disk alone.

How request encryption works: (1) the client opens a connection and sends its username; (2) the client encrypts each request with AES-256-GCM using the password-derived key, one base64 line per message; (3) the server decrypts with the same key — a request that fails to decrypt closes the connection; (4) the response is encrypted the same way.

| Access level | Allows |
|---|---|
| `1` read | find, count, distinct, max, min, findByView, countByView, getCollectionNames, getCollectionSchema, addView, getDatabases, getHealth, getVersion, subscriptions |
| `2` write | read, plus insert, update, remove, removeView, createCollection, transactions, migrations |
| `3` manage | write, plus createDatabase, removeDatabase, removeCollection, schema update, lock/unlock, getCollectionSchemaLock, getCollectionViews |

Caveat: passwords are never transmitted, but the user name in the first line of each connection is sent in plaintext. Enable TLS if that must stay confidential.

---

## Desktop Application

A desktop GUI for working directly with a DianaDB server: browsing and managing data, editing schemas and their relationships, creating and inspecting materialized views, and testing queries against real data.

Builds (v1.1.2):

- Linux: `https://dist.databikers.com/gui/diana-db-gui/DianaDB-GUI-1.1.2.AppImage`
- Windows: `https://dist.databikers.com/gui/diana-db-gui/DianaDB-GUI%20Setup%201.1.2.exe`

---

## ODM Overview

```bash
npm i -s @diana-db/odm
```

Server 3.0.0 requires ODM >= 3.0.0 for subscriptions. ODM 3.0.0 also works with older servers, so upgrade the ODM first.

### Create a DianaDB Instance

```typescript
import { readFileSync } from 'node:fs';
import { DianaDb } from '@diana-db/odm';

export const dianaDb = new DianaDb({
  host: 'db.example.com',
  port: 34567,
  user: 'db_user',
  password: 'some-password',
  connectionPoolSize: 5,
  connectTimeoutValue: 5000,
  reconnectTimeoutValue: 1000,
  requestTimeoutValue: 30000,
  database: 'test',          // default database for migrations (optional)
  secureServer: true,        // if the server uses TLS
  tls: {
    ca: readFileSync('/path/to/ca'),     // CA that signed the server certificate
    rejectUnauthorized: true,            // verify the certificate against host
    cert: readFileSync('/path/to/cert'), // only for mTLS
    key: readFileSync('/path/to/key'),   // only for mTLS
  },
});

await dianaDb.connect(5000);
```

A connection string may be used instead:

```typescript
import { DianaDb } from '@diana-db/odm';

export const dianaDb = new DianaDb('diana-db://user:pass@localhost:34567?connectionPoolSize=5');
export const dianaDbS = new DianaDb('diana-dbs://user:pass@db.example.com:34567/test?rejectUnauthorized=true'); // TLS, default database "test"
```

The connection string accepts `connectionPoolSize`, `connectTimeoutValue`, `reconnectTimeoutValue` and `requestTimeoutValue` as query parameters, and `rejectUnauthorized=true` with `diana-dbs://`. The optional path sets the `database` option. User name and password are URL-decoded (`p%40ss` for `p@ss`). The `diana-dbs://` scheme sets `secureServer: true`. Any other protocol throws a configuration error.

You can create as many instances as you like; each has its own migrations and subscribers and connects independently. A model sends its requests through the instance passed as `dianaDb` in its options, or else through the instance most recently constructed before the model.

### Connection Behaviour

- `connect()` resolves as soon as one pooled connection is up; the rest connect in the background.
- If every connection is down (server restart, network blip), new requests are queued and sent when a connection is back, or fail with a `TIMEOUT` error after `requestTimeoutValue`.
- Requests already sent on a connection that drops fail at once with a `CONNECTION` error.
- A wrong password fails the request at once: the server closes connections whose requests it cannot decrypt.
- After `disconnect()` requests fail immediately until `connect()` is called again. `disconnect()` also closes the subscriber connection, so the process can exit.

### Constructor Options

| Property | Description | Type | Default | Constraint |
|---|---|---|---|---|
| `host` | Server host | String | – | required |
| `port` | Server port | Integer | `34567` | 1025–65535 |
| `user` | Username | String | – | required |
| `password` | Password | String | – | required |
| `connectionPoolSize` | Number of pooled connections | Integer | `5` | >= 1 |
| `connectTimeoutValue` | Connection timeout (ms) | Integer | `30000` | – |
| `reconnectTimeoutValue` | Delay between reconnection attempts (ms) | Integer | `1000` | – |
| `requestTimeoutValue` | Default timeout of every request (ms) | Integer | `30000` | >= 1 |
| `database` | Default database for `migrateUp`, `migrateDown`, `getMigrations` | String | – | – |
| `secureServer` | Whether the server uses TLS | Boolean | `false` | – |
| `tls` | `{ ca?, rejectUnauthorized?, servername?, cert?, key? }` — Buffers or PEM strings; `ca` may be an array; other `tls.connect` options pass through | Object | – | – |
| `logger` | Logger with `log` and `error` | Object | `console` | – |

`tls.rejectUnauthorized` defaults to `false`, so self-signed servers work out of the box. Set it to `true` with `ca` in production; the certificate is then checked against `host` (or `servername`), so connect by the name in the certificate, not by IP.

### Client Methods

| Method | Arguments | Description | Returns |
|---|---|---|---|
| `connect` | `connectTimeoutValue?: number` | Connects to the server | `Promise<boolean>` |
| `disconnect` | – | Closes every connection, the subscriber connection included | `Promise<void>` |
| `subscribe` | `key: string`, `subscriber: (update: DatabaseUpdate) => void` | Adds a subscriber for database changes; several per key | `Promise<void>` |
| `unsubscribe` | `key: string`, `subscriber?` | Removes one subscriber, or every subscriber of the key | `Promise<void>` |
| `addMigration` | `migration: Migration` | Registers a migration script. Throws if the index or name is already used | `void` |
| `migrateUp` | `database?: string` | Applies pending migrations in ascending index order | `Promise<MigrationInfo[]>` |
| `migrateDown` | `database?: string`, `steps?: number` | Reverts applied migrations in descending index order — all, or the last `steps` | `Promise<MigrationInfo[]>` |
| `getMigrations` | `database?: string` | Names of the migrations recorded on the server | `Promise<string[]>` |
| `startTransaction` | `{ database, autoRollBackAfterMs? }` | Starts a transaction with optional auto-rollback | `Promise<string>` (transactionId) |
| `commitTransaction` | `{ database, transactionId }` | Commits an open transaction | `Promise<TransactionInfo>` |
| `rollbackTransaction` | `{ database, transactionId }` | Rolls back an open transaction | `Promise<TransactionInfo>` |
| `getDatabases` | – | Database names the current user can read | `Promise<string[]>` |
| `createDatabase` | `database: string` | Creates an empty database. Requires manage access | `Promise<{ database, action }>` |
| `removeDatabase` | `database: string` | Removes a database | `Promise<{ database, action }>` |
| `getCollections` | `database: string` | Collection names in a database | `Promise<string[]>` |
| `createCollection` | `database, collection, schema` | Creates a collection with the given schema | `Promise<{ collection, action }>` |
| `removeCollection` | `database, collection` | Removes a collection | `Promise<{ collection, action }>` |
| `getCollectionSchema` | `database, collection` | Reads the stored schema and its lock state | `Promise<{ schema, locked }>` |
| `getCollectionSchemaLock` | `database, collection` | Reads the lock state | `Promise<{ collection, locked }>` |
| `lockCollectionSchema` | `database, collection` | Locks the schema, rejecting structural drift | `Promise<{ collection, locked }>` |
| `unlockCollectionSchema` | `database, collection` | Unlocks the schema | `Promise<{ collection, locked }>` |
| `getCollectionViews` | `database, collection` | Materialized views defined on a collection | `Promise<Record<string, { transformQueries }>>` |
| `getHealth` | – | Current server health | `Promise<HealthInfo>` |
| `getVersion` | – | Server version | `Promise<{ version: string }>` |

```typescript
type HealthInfo = {
  memory: number; // memory usage (MB)
  cpu: number;    // CPU usage (%)
  gcp: number;    // garbage collector pressure (%)
  elu: number;    // event loop utilization (%)
  eld: number;    // event loop delay (ms)
};
```

### Errors

Every failed request rejects with an `Error` carrying a `code` (type `DianaDbError`):

| `code` | Meaning | Applied? |
|---|---|---|
| `SERVER` | The server rejected the request (validation, access, unknown collection, locked schema…) | No |
| `DURABILITY` | Applied in memory, but the write-ahead log could not be written; may be lost in a crash before the next snapshot | Yes — don't retry |
| `TIMEOUT` | No answer within `requestTimeoutValue` | Unknown |
| `CONNECTION` | Not connected, or the connection closed before the answer | Unknown (No when not connected) |

Validation errors raised by the ODM itself (schema, queries, options, migrations without a database) are thrown before anything is sent and carry no `code`.

---

## Document Schema

A schema is a configuration object describing the structure of stored documents. Keys map to document properties; values are field configuration objects. `_id` is added automatically and does not need to be declared.

### Schema Field Options

| Option | Type | Description | Required | Default |
|---|---|---|---|---|
| `type` | `Types` | Field type | Yes | – |
| `required` | boolean | Property must be present | No | `false` |
| `unique` | boolean | Property must be unique | No | `false` |
| `mutable` | boolean | Value may change after creation | No | `true` |
| `lowercase` | boolean | Lowercases the value. **STRING only** | No | `false` |
| `uppercase` | boolean | Uppercases the value. **STRING only** | No | `false` |
| `default` | `T`, `() => T`, or `() => Promise<T>` | Default value or factory, evaluated in the context of the document | No | – |
| `get` | function | Getter applied to each found document. Not applied when transform queries are used | No | – |
| `set` | function | Setter applied on insert and update before validation | No | – |
| `items` | `Types` | Item type. **ARRAY only** | Yes (if `type` is ARRAY) | – |
| `reference` | string | Referenced collection name | Yes (if `type` is REFERENCE, or `items` is REFERENCE) | – |
| `triggerRemove` | boolean | Remove the document when the referenced item is deleted. **REFERENCE only** | No | `false` |
| `ttl` | number | Time to live in ms, counted from the field's time value, minimum 5. **TIME only** | No | – |
| `precision` | integer 0–8 | Decimal places retained. Without it numbers are stored as given (since 3.0.0; earlier versions required it). **NUMBER only** | No | – |

Mutual exclusions enforced by the validator:

- `lowercase` and `uppercase` cannot both be set.
- At most one of `default`, `set`, `get` may be set on a field.
- `lowercase`, `uppercase`, `items`, `precision`, `ttl` and `triggerRemove` are **rejected** on types they do not apply to — including `precision` on an ARRAY of NUMBER.

### Field Types

| Type | Description | Notes |
|---|---|---|
| `Types.STRING` | String value | Supports `lowercase` / `uppercase` |
| `Types.NUMBER` | Numeric value | Optional `precision` |
| `Types.BOOLEAN` | Boolean value | – |
| `Types.POSITION` | `{ x: number, y: number }` | Enables spatial queries |
| `Types.OBJECT_ID` | Hex string identifier | Not BSON. Pattern: `/^[0-9a-f]{12}-[0-9a-f]{1,}-[0-9a-f]{11}-[0-9a-f]{9}$/` |
| `Types.REFERENCE` | Like OBJECT_ID, linked to another collection | Requires `reference`. Enables `$subQuery` |
| `Types.TIME` | ISO 8601 string **or** epoch milliseconds | Supports time-part filters and `ttl` |
| `Types.ARRAY` | Homogeneous array, item type from `items` | Any type may be an item. Mixed types not allowed. Set-like: unique values only |

### Schema Examples

```typescript
import { Types, Schema } from '@diana-db/odm';

type User = {
  name: string;
  position: { x: number; y: number };
  isActive: boolean;
  createdAt: string;
};

export const userSchema: Schema<User> = {
  name: { type: Types.STRING, unique: true, required: true, lowercase: true },
  isActive: { type: Types.BOOLEAN, default: true },
  position: { type: Types.POSITION },
  createdAt: { type: Types.TIME, default: () => new Date().toISOString() },
};

export const postSchema = {
  user: { type: Types.REFERENCE, reference: 'user', required: true, triggerRemove: true },
  title: { type: Types.STRING, required: true },
  content: { type: Types.STRING, required: true },
  isPublished: { type: Types.BOOLEAN, default: true },
  createdAt: { type: Types.TIME, required: true },
  publishedAt: { type: Types.TIME, required: true },
};
```

Warning: schema changes are detected automatically and applied server-side when your code runs. Removing fields can lose data. Keep schemas in one shared package so they cannot diverge, or lock the collection schema with `lockCollectionSchema`.

---

## Model

```typescript
import { Model } from '@diana-db/odm';

const userModel = new Model<User>({
  database: 'test',
  collection: 'user',
  name: 'User',
  schema: userSchema,
});

await userModel.init();
```

`init()` is safe to call concurrently (callers share one attempt) and can be retried after a failure. Creating a collection needs write access; updating a changed schema needs manage access and fails if the remote schema is locked. `insert` and `update` apply defaults and setters to a copy, so the object you pass is not modified. `find`, `count`, `update` and `remove` accept one FindQuery or an array; `find()` and `count()` without filters match every document.

### Model Options

| Property | Description |
|---|---|
| `database` | Database name |
| `collection` | Collection name |
| `name` | Model name, used in validation errors |
| `schema` | Schema definition object |
| `dianaDb` | Optional DianaDb instance the model uses. Defaults to the instance most recently constructed before the model |

### Model Methods

| Method | Description | Returns |
|---|---|---|
| `init()` | Initializes the model: creates the collection or reconciles the schema. Called lazily by every other method if not called explicitly | `Promise<void>` |
| `Model.init()` (static) | Initializes every constructed model | `Promise<void>` |
| `insert(data, transactionId?)` | Applies defaults and setters, validates, inserts | `Promise<T & { _id: string }>` |
| `find(filters?, transforms?, sort?, skip?, limit?, transactionId?)` | Finds documents | `Promise<T[]>` |
| `count(filters?, transforms?, transactionId?)` | Counts matching documents | `Promise<number>` |
| `update(filters, updateData, transactionId?)` | Updates matching documents | `Promise<{ found: number, modified: number }>` |
| `remove(filters, transactionId?)` | Removes matching documents | `Promise<{ found: number, removed: number }>` |
| `distinct(key)` | Unique values for a key | `Promise<any[]>` |
| `max(key)` | Documents holding the maximum value for a key | `Promise<T[]>` |
| `min(key)` | Documents holding the minimum value for a key | `Promise<T[]>` |
| `createView(name, transformQueries)` | Creates a materialized view | `Promise<{ view: string, action: 'created' }>` |
| `removeView(name)` | Removes a materialized view | `Promise<{ view: string, action: 'removed' }>` |
| `getViews()` | Views defined on this collection | `Promise<Record<string, { transformQueries }>>` |
| `findByView(name, filter?, sort?, skip?, limit?)` | Queries a materialized view | `Promise<any[]>` |
| `countByView(name, filter?)` | Counts results in a view. **Two arguments only** | `Promise<number>` |
| `lock()` | Locks this collection's schema | `Promise<{ collection, locked }>` |
| `unlock()` | Unlocks this collection's schema | `Promise<{ collection, locked }>` |

---

## Insert Documents

```typescript
const user = await userModel.insert({
  name: 'John',
  position: { x: 1, y: 1 },
  createdAt: new Date().toISOString(),
});
// {
//   _id: 'dc4628514b81-939f-198a25f76b7-bb1a9a777',
//   name: 'john',
//   position: { x: 1, y: 1 },
//   isActive: true,
//   createdAt: '2025-06-18T06:30:24.977Z'
// }
```

`_id` is added automatically. Fields with `default` are filled in when absent; fields with `set` are passed through the setter before validation. The returned document is a plain object with no special methods.

---

## Find and Count Documents

```typescript
const [user] = await userModel.find(
  [{ name: { $eq: 'John' } }],    // FindQuery — array behaves as OR
  [{ $project: { name: true } }], // Transform Queries
  { _id: -1 },                    // Sorting
  0,                              // Skip
  1,                              // Limit
);
```

All arguments are optional; without filters every document matches. `count` takes the first two arguments plus `transactionId` and returns a number instead of documents.

### Find Arguments

| Argument | Description |
|---|---|
| `FindQuery` (or array) | Keys match schema fields, values are query operands. Operands within one FindQuery are ANDed; an array of FindQueries is ORed |
| `TransformQueries` | Array of pipeline stages |
| `SortingQuery` | `{ field: 1 }` ascending, `{ field: -1 }` descending. Multi-key supported |
| `Skip` | Integer >= 0 |
| `Limit` | Positive integer |
| `transactionId` | Executes the query inside a transaction scope |

### Query Operands

| Operand | Types | Description |
|---|---|---|
| `$eq` | all | Equals. Shorthand: `{ name: 'John' }` |
| `$ne` | all | Not equal |
| `$in` | all | Value is in the given array |
| `$nin` | all | Value is not in the given array |
| `$gt`, `$gte`, `$lt`, `$lte` | NUMBER | Range comparison |
| `$regex` | STRING | Regular expression match |
| `$startsWith`, `$endsWith`, `$notStartsWith`, `$notEndsWith` | STRING | Prefix / suffix match |
| `$cn`, `$nc` | STRING | Contains / does not contain |
| `$subQuery` | REFERENCE, OBJECT_ID | Nested query against the referenced collection |

On OBJECT_ID, REFERENCE and BOOLEAN fields exactly **one** operand may be used per field, so `$subQuery` must stand alone.

#### TIME queries

TIME fields are queried through part operands, each taking a numeric query, plus `$raw` which takes a string query against the stored value:

`$year`, `$month`, `$date`, `$hours`, `$minutes`, `$seconds`, `$dayOfWeek`, `$dayOfYear`, `$timestamp`, `$raw`

```typescript
{
  createdAt: {
    $year: { $eq: 2025 },
    $month: { $in: [1, 4] },
    $dayOfWeek: { $in: [0, 1] },
  },
}
```

Range filtering over an instant goes through `$timestamp`, e.g. `{ createdAt: { $timestamp: { $gte: 1750000000000 } } }`.

#### Spatial queries

```typescript
{ position: { $insideCircle: { center: { x: 1, y: 1 }, radius: 10 } } }
{ position: { $insidePolygon: [ { x: 1, y: 1 }, { x: 3, y: 0 }, { x: -1, y: -1 } ] } }
{ position: { $nearLines: { lines: [ [ { x: 1, y: 1 }, { x: 10, y: 10 } ] ], distance: 10 } } }
```

`$insideCircle`, `$outsideCircle`, `$insidePolygon`, `$outsidePolygon`, `$nearLines`, `$farFromLines`. Polygons need at least 3 points; each line is exactly 2 points and at least one line is required. `$nearLines.distance` must be positive; `$farFromLines.distance` may be 0.

#### Array queries

Array fields accept the same operands as their item type; the query is applied per element with OR semantics. To match documents whose array contains 1 and 2 but not 3:

```typescript
{ someArrayProperty: { $in: [1, 2], $nin: [3] } }
```

#### Reference sub-query

```typescript
{ user: { $subQuery: { name: { $eq: 'John' } } } }
```

### distinct / max / min

```typescript
const names = await userModel.distinct('name');
const best = await userModel.max('score');
const worst = await userModel.min('score');
```

### Transform Queries (Pipelines)

An array of stage objects, each holding exactly **one** key.

| Stage | Description |
|---|---|
| `$project` | Each key becomes a property of the result. Value may be boolean (keep / drop), a pointer string, or a projection operand object |
| `$group` | Same shape as `$project` plus a required `_id`, which may be `false` (merge into one), a pointer string, or an object of pointers |
| `$match` | Filters the current documents. Supports boolean, string, number, TIME and spatial queries — not `$subQuery` |
| `$lookup` | `{ collection (required), database, as (required), localField (required), foreignField (required), filter, sort, skip, limit, replaceRoot }`. Sets `as` to the array of found documents |
| `$unwind` | Pointer string. Deconstructs an array property into one document per item |
| `$replaceRoot` | Pointer string. Replaces each document with the value at that property |
| `$sort` | `{ field: 1 | -1 }`, multi-key |
| `$skip` | Integer >= 0 |
| `$limit` | Positive integer (>= 1) |

`$unwind` and `$replaceRoot` values must be pointers — they start with `$`.

### Projection Operands

Each projection object holds exactly one operand. Since Server 1.4.8, math operands never implicitly include the existing property value; reference it explicitly as a pointer: `{ $sum: ['$amount', 123] }`.

| Operand | Format | Description |
|---|---|---|
| `$max`, `$min`, `$avg` | pointer string | Maximum / minimum / average |
| `$sum`, `$subtract`, `$multiply`, `$divide` | array of numbers and pointers | Arithmetic. `$divide` rejects a `0` divisor |
| `$round` | integer 0–8 | Rounds to the given precision |
| `$ifNull` | array | First non-null / non-undefined value |
| `$push` | array of values and pointers | Creates an array property and pushes values |
| `$addToSet` | array of values and pointers | Like `$push` but unique |
| `$concatArray` | pointer to an array property | Concatenates arrays |
| `$first`, `$last` | pointer string | First / last value |
| `$concat` | `{ delimiter, parts[] }` | Concatenates strings; pointers in `parts` are evaluated |
| `$year`, `$month`, `$date`, `$hours`, `$minutes`, `$seconds`, `$dayOfWeek`, `$dayOfYear`, `$timestamp` | pointer to a TIME property | Extracts the time part as a number |

### Transform Query Example

```typescript
const result = await postModel.find(
  [{}],
  [
    { $group: { _id: '$user', user: { $first: '$user' }, postsCount: { $sum: [1] } } },
    {
      $lookup: {
        database: 'test',
        collection: 'user',
        localField: 'user',
        foreignField: '_id',
        as: 'user',
        filter: { isActive: { $eq: true } },
      },
    },
    { $unwind: '$user' },
    { $project: { _id: false, user: true, postsCount: true } },
    { $sort: { postsCount: -1 } },
    { $skip: 1 },
    { $limit: 1 },
  ],
);
```

---

## Update Documents

```typescript
await userModel.update(
  [{ name: { $eq: 'John' } }],
  { isActive: false },
);
```

Values may be plain values, or an operator object describing an update strategy for that field's type.

**String:**

| Operator | Type | Effect |
|---|---|---|
| `$concat` | `{ delimiter: string, parts: string[] }` | Concatenates `parts` with `delimiter` |
| `$replace` | `[search, replacement, flags]` | Replaces a substring; `flags` is a RegExp flag combination such as `'gm'` |

**Number:**

| Operator | Type | Effect |
|---|---|---|
| `$add` | number | Adds to the current value |
| `$subtract` | number | Subtracts from the current value |
| `$multiply` | number | Multiplies the current value |
| `$divide` | number, non-zero | Divides the current value |
| `$round` | integer 0–8 | Rounds to the given precision |

**Array** — exactly one operator per field:

| Operator | Type | Effect |
|---|---|---|
| `$addItem` | any | Appends an element if not already present |
| `$removeItem` | any | Removes an element if found |

Renamed in Server/ODM 1.9.0 from `$add` / `$remove` to avoid clashing with the numeric operators.

**TIME:**

| Operator | Structure | Effect |
|---|---|---|
| `$add` | `{ amount: positive integer, unit: UnitOfTime }` | Adds time |
| `$subtract` | `{ amount: positive integer, unit: UnitOfTime }` | Subtracts time |
| `$toStartOf` | `UnitOfTime` | Rounds down to the start of the unit |
| `$toEndOf` | `UnitOfTime` | Rounds up to the end of the unit |
| `$toNextWeekday` | `Weekday` | Jumps forward to the next occurrence of that weekday |
| `$toLastWeekday` | `Weekday` | Jumps back to the previous occurrence of that weekday |

`UnitOfTime`: `'year'`, `'month'`, `'week'`, `'day'`, `'hour'`, `'minute'`, `'second'`, `'millisecond'`.

`Weekday`: `'Sunday'`, `'Monday'`, `'Tuesday'`, `'Wednesday'`, `'Thursday'`, `'Friday'`, `'Saturday'`.

---

## Remove Documents

```typescript
const { found, removed } = await userModel.remove([{ isActive: { $eq: false } }]);
```

---

## Subscribe to Database Updates

```typescript
import { DatabaseUpdate } from '@diana-db/odm';

// 'user.post' = database 'user', collection 'post'
// 'user' alone = every collection in the 'user' database
const onPost = (databaseUpdate: DatabaseUpdate) => {
  // handle the update
};
await dianaDb.subscribe('user.post', onPost);
await dianaDb.unsubscribe('user.post', onPost); // or unsubscribe('user.post') for every subscriber of the key
```

Several subscribers per key are allowed; one that throws does not affect the others. Subscriptions use one dedicated connection per client that reconnects on its own and is closed by `disconnect()` or when the last subscriber is removed. Updates are delivered only for databases the user can read.

The subscriber connection authenticates with `user:<name>:subscriber:<token>`, where the token is AES-256-GCM(sha256(password), `user:<name>:subscriber:<UTC date YYYY-MM-DD>`) in base64. The ODM builds it on every (re)connect; the server accepts yesterday to tomorrow and closes subscriber connections without a valid token. ODM versions before 3.0.0 send no token, so Server 3.0.0 rejects their subscriptions.

### DatabaseUpdate Contract

The subscriber receives the update itself (since ODM 3.0.0; earlier versions passed the whole server message, with the update under `data`).

| Field | Type | Description |
|---|---|---|
| `database` | string | Database name |
| `collection` | string | Collection name |
| `action` | string | The operation that fired the update — `insert`, `update`, `remove` |
| `affectedIds` | string[] | Ids affected by the operation |
| `data` | object | Document snapshot for insert and update |

---

## Migrations

DianaDB has its own migration storage. Register a migration with `addMigration`:

```typescript
dianaDb.addMigration({
  index: 1,                 // unique integer index
  name: 'MyFirstMigration', // unique name
  up: async () => {
    // apply changes
  },
  down: async () => {
    // revert changes
  },
});

await dianaDb.connect(5000);
const applied = await dianaDb.migrateUp('test');        // [{ name: 'MyFirstMigration', status: 'up' }]
const reverted = await dianaDb.migrateDown('test', 1);  // revert only the last applied migration
const recorded = await dianaDb.getMigrations('test');   // names recorded as applied
```

A migration may span many databases and collections. There is a single list of applied migrations, so names and indexes must be unique — registering a duplicate index or name throws immediately. Never edit the code inside a migration after it has run.

- The server checks write access on a database for every migration call, so each call needs one: pass it, or set the `database` constructor option (or the connection string path). Without either, the ODM throws before sending anything.
- `migrateUp` runs pending migrations in ascending index order; `migrateDown` reverts applied ones in descending order — all, or the last `steps`.
- Both resolve with `MigrationInfo[]` — `{ name: string; status: 'up' | 'down' }` — the server's answer `{ migration: { name, status } }` for each migration, which the ODM checks. If the server doesn't confirm one, the call throws a migration error naming it; its `up()`/`down()` has already run at that point.

---

## Transactions

DianaDB supports non-blocking ACID transactions with document-level (row-level) locking.

- Transactions have higher priority than regular requests and can overwrite previously saved data.
- Affected documents are locked dynamically throughout execution until commit or rollback.
- Each transaction has its own scope; changes are invisible outside it until committed.

```typescript
await dianaDb.connect(5000);

const transactionId: string = await dianaDb.startTransaction({
  database: 'test',
  autoRollBackAfterMs: 60000,
});

const manageTransactionParameters = { database: 'test', transactionId };

try {
  await userModel.remove([{ ...filters }], transactionId);
  await dianaDb.commitTransaction(manageTransactionParameters);
} catch (e) {
  await dianaDb.rollbackTransaction(manageTransactionParameters);
}
```

`startTransaction` resolves to the transaction id string. `autoRollBackAfterMs` must lie between the server's `transactions_min_auto_rollback_value` and `transactions_max_auto_rollback_value` (defaults 1000 and 60000 ms); other values are rejected, and without it the maximum is used. A commit validates every collection before applying anything, so a failed commit changes nothing. Transactions need write access. Both commit and rollback resolve to a `TransactionInfo` — `{ transactionId, status }`, where status is `'committed'` or `'rolled-back'`.

---

## Materialized Views

A view is a stored aggregation that updates automatically when the underlying data changes. Views support cross-database lookups. A view made only of `$match`, `$project`, `$unwind`, `$replaceRoot` and `$lookup` is updated per document; a view with `$group`, `$sort`, `$skip` or `$limit`, and any change in a `$lookup` target, triggers a throttled full rebuild.

```typescript
const transactionModel = new Model({
  database: 'test',
  collection: 'transaction',
  name: 'Transaction',
  schema: {
    user: { type: Types.REFERENCE, reference: 'user', required: true, triggerRemove: true },
    currency: { type: Types.STRING, required: true },
    amount: { type: Types.NUMBER, default: 0, precision: 2 },
    status: { type: Types.STRING, default: 'pending' },
    created: { type: Types.TIME, default() { return new Date().toISOString(); } },
    updated: { type: Types.TIME, default() { return new Date().toISOString(); } },
  },
});

await transactionModel.init();

await transactionModel.createView('balance', [
  {
    $group: {
      _id: { user: '$user', currency: '$currency' },
      user: { $first: '$user' },
      amount: { $sum: ['$amount'] },
      currency: { $first: '$currency' },
    },
  },
]);

const balances = await transactionModel.findByView(
  'balance',
  { user: { $eq: someUser._id } },
  { amount: -1 },
);

const count = await transactionModel.countByView('balance', { user: { $eq: someUser._id } });

const views = await transactionModel.getViews();
await transactionModel.removeView('balance');
```

### Arguments

- `createView(name, transformQueries)` — both required. The name must not already be used as a collection or view name in that database.
- `removeView(name)` — required.
- `findByView(name, findQuery?, sortingQuery?, skip?, limit?)` — only the name is required. Keys in `findQuery` match the view's output shape, not the source schema.
- `countByView(name, findQuery?)` — only two arguments.

---

## Cook Book: A Mini Shop Example

An end-to-end example combining schemas, materialized views, seeding, and a transactional
order flow: users, items, prices, store inventory, adding money to a user's balance, and
creating orders against it.

### Setup

Before the client code below can connect, create a server user and grant it access to the
database this example uses:

```bash
sudo diana-db add-user -u admin -p password
sudo diana-db grant-access -u admin -db data:3
```

### Install

```bash
npm i @diana-db/odm
```

This example has no other dependencies -- the random data used to seed it (names, emails,
birthdates, coordinates, product copy) is generated by small local helper functions, not a
third-party library.

Full source and walkthrough: https://diana-db.com/cookbook/shop-example

### 1. Connect

Every collection shares one `database` name so they can all participate in the same
transaction later -- `startTransaction` / `commitTransaction` are scoped to a single
database. The `DianaDb` instance is constructed here, but the actual `connect()` call
happens in `initModels()` below, alongside every model's `.init()`.

```typescript
const DATABASE = 'data';

const dianaDb = new DianaDb({
  host: process.env.DIANA_DB_HOST || 'localhost',
  port: Number(process.env.DIANA_DB_PORT || 34567),
  user: process.env.DIANA_DB_USER || 'admin',
  password: process.env.DIANA_DB_PASSWORD || 'password',
  connectionPoolSize: 3,
  connectTimeoutValue: 1000,
});
```

### 2. Random data helpers

The seed step (further down) never hand-writes sample values -- it calls a small set of
helper functions instead. All of it is plain JS: names, emails, birthdates, coordinates,
and product copy are built from small local word pools and basic math, with no external
package involved.

```typescript
function rndInt(min, max) {
  return Math.floor(Math.random() * (max - min + 1)) + min;
}

function rndAmount(min, max) {
  return Math.round((Math.random() * (max - min) + min) * 100) / 100;
}

function rndSample(arr, count) {
  const n = Math.min(count, arr.length);
  const shuffled = [...arr].sort(() => Math.random() - 0.5);
  return shuffled.slice(0, n);
}

function rndPick(arr) {
  return arr[rndInt(0, arr.length - 1)];
}

function rndSex() {
  return Math.random() < 0.5 ? 'm' : 'f';
}

// A plausible date of birth for an 18-75 year old, as an ISO date string.
function rndDob() {
  const msPerYear = 365.25 * 24 * 60 * 60 * 1000;
  const minAgeMs = 18 * msPerYear;
  const maxAgeMs = 75 * msPerYear;
  const ts = Date.now() - (minAgeMs + Math.random() * (maxAgeMs - minAgeMs));
  return new Date(ts).toISOString().slice(0, 10);
}

function rndTags() {
  const pool = ['new', 'vip', 'wholesale', 'trial', 'referred', 'newsletter'];
  return rndSample(pool, rndInt(0, 3));
}

const FIRST_NAMES = [
  'James', 'Mary', 'John', 'Patricia', 'Robert', 'Jennifer', 'Michael', 'Linda',
  'William', 'Elizabeth', 'David', 'Barbara', 'Richard', 'Susan', 'Joseph', 'Jessica',
  'Thomas', 'Sarah', 'Charles', 'Karen', 'Somchai', 'Malee', 'Anong', 'Chai',
];

const LAST_NAMES = [
  'Smith', 'Johnson', 'Williams', 'Brown', 'Jones', 'Garcia', 'Miller', 'Davis',
  'Rodriguez', 'Martinez', 'Wilson', 'Anderson', 'Taylor', 'Moore', 'Jackson', 'Martin',
  'Saetang', 'Charoen', 'Suksawat', 'Wongsawat',
];

const EMAIL_DOMAINS = ['example.com', 'mail-test.dev', 'demo-inbox.io'];

function rndFullName() {
  return { firstname: rndPick(FIRST_NAMES), lastname: rndPick(LAST_NAMES) };
}

function rndEmail(firstname, lastname) {
  const domain = rndPick(EMAIL_DOMAINS);
  return `${firstname}.${lastname}${rndInt(1, 9999)}@${domain}`.toLowerCase();
}

function rndUser() {
  const { firstname, lastname } = rndFullName();
  return {
    email: rndEmail(firstname, lastname),
    firstname,
    lastname,
    dob: rndDob(),
    sex: rndSex(),
    tags: rndTags(),
  };
}

// A coordinate within `radiusKm` of an origin point, so demo users cluster
// around a real city instead of landing at random points on the globe.
// Samples uniformly over the disc (sqrt of a random distance, not a random
// distance directly) and offsets by a random bearing.
function rndPosition() {
  const originLat = 13.7563;
  const originLng = 100.5018;
  const radiusKm = 50;
  const earthRadiusKm = 6371;

  const distanceKm = radiusKm * Math.sqrt(Math.random());
  const bearing = Math.random() * 2 * Math.PI;
  const latRad = (originLat * Math.PI) / 180;

  const deltaLat = (distanceKm / earthRadiusKm) * Math.cos(bearing);
  const deltaLng = (distanceKm / (earthRadiusKm * Math.cos(latRad))) * Math.sin(bearing);

  const lat = originLat + (deltaLat * 180) / Math.PI;
  const lng = originLng + (deltaLng * 180) / Math.PI;

  return { x: lng, y: lat };
}

const PRODUCT_ADJECTIVES = [
  'Ergonomic', 'Rustic', 'Sleek', 'Handcrafted', 'Durable', 'Compact',
  'Premium', 'Lightweight', 'Vintage', 'Modern', 'Portable', 'Refined',
];

const PRODUCT_NOUNS = [
  'Chair', 'Lamp', 'Backpack', 'Keyboard', 'Mug', 'Jacket',
  'Shelf', 'Speaker', 'Wallet', 'Notebook', 'Bottle', 'Sofa',
];

function rndItem() {
  const adjective = rndPick(PRODUCT_ADJECTIVES);
  const noun = rndPick(PRODUCT_NOUNS);
  return {
    title: `${adjective} ${noun}`,
    description: `A ${adjective.toLowerCase()} ${noun.toLowerCase()} built to last.`,
  };
}

function rndSerialNumber(existing) {
  let candidate = rndInt(100000, 999999);
  while (existing.has(candidate)) {
    candidate = rndInt(100000, 999999);
  }
  existing.add(candidate);
  return candidate;
}
```

### 3. Define every schema and model

Seven collections make up the shop. Each one pairs a plain schema object with a `Model`
instance. None of these are initialized yet; that happens all together in
`initModels()`, the next step.

**User** -- the account itself. `email` is unique and lower-cased on write.

```typescript
const userSchema = {
  email: { type: Types.STRING, required: true, unique: true, lowercase: true },
  firstname: { type: Types.STRING, required: true },
  lastname: { type: Types.STRING, required: true },
  dob: { type: Types.TIME, required: true },
  sex: { type: Types.STRING, default: 'M' },
  created: { type: Types.TIME, required: true, default: () => new Date().toISOString() },
  active: { type: Types.BOOLEAN, default: true },
  tags: { type: Types.ARRAY, items: Types.STRING, default: () => [] },
};

const userModel = new Model({ database: DATABASE, collection: 'user', name: 'User', schema: userSchema });
```

**UserLocation** -- one row per user, using `Types.POSITION` (an `{ x, y }` pair) so the
user's location can be queried spatially, and looked up alongside their orders (see
`orderToPay` / `orderToShip` below).

```typescript
const userLocationSchema = {
  user: { type: Types.REFERENCE, reference: 'user', required: true, triggerRemove: true },
  location: { type: Types.POSITION, required: true },
};

const userLocationModel = new Model({ database: DATABASE, collection: 'userLocation', name: 'UserLocation', schema: userLocationSchema });
```

**Item** -- the catalog entry: a title and description, with no price or stock of its own.

```typescript
const itemSchema = {
  title: { type: Types.STRING, required: true },
  description: { type: Types.STRING, required: true },
  created: { type: Types.TIME, required: true, default: () => new Date().toISOString() },
};

const itemModel = new Model({ database: DATABASE, collection: 'item', name: 'Item', schema: itemSchema });
```

**Price** -- append-only: every price change inserts a new document rather than updating
an existing one, which is what lets the `currentPrice` view reconstruct "the latest price"
by sorting on `created`. The read helper for that view lives right here, next to the model:

```typescript
const priceSchema = {
  item: { type: Types.REFERENCE, reference: 'item', required: true, triggerRemove: true },
  amount: { type: Types.NUMBER, required: true, precision: 2 },
  currency: { type: Types.STRING, required: true, default: 'USD' },
  created: { type: Types.TIME, required: true, default: () => new Date().toISOString() },
};

const priceModel = new Model({ database: DATABASE, collection: 'price', name: 'Price', schema: priceSchema });

async function getCurrentPrice(itemId) {
  const [row] = await priceModel.findByView('currentPrice', { item: { $eq: itemId } });
  return row?.price;
}
```

**StoreItem** -- one document per physical unit in stock, each with its own
`serialNumber`. `reserved` starts `false` and flips to `true` the moment an order is placed
for it, so it stops counting as available stock in the `inventory` view. `reservedFor`
records which user holds that reservation. `default` here is given as a factory function
(`() => false`) rather than a bare value -- both forms are valid, the same way `tags` on
`User` uses `() => []`.

```typescript
const storeItemSchema = {
  item: { type: Types.REFERENCE, reference: 'item', required: true, triggerRemove: true },
  serialNumber: { type: Types.NUMBER, required: true, precision: 0 },
  reserved: { type: Types.BOOLEAN, default: () => false },
  reservedFor: { type: Types.REFERENCE, reference: 'user' },
  created: { type: Types.TIME, required: true, default: () => new Date().toISOString() },
};

const storeItemModel = new Model({ database: DATABASE, collection: 'storeItem', name: 'StoreItem', schema: storeItemSchema });

async function getAvailableCount(itemId) {
  const [row] = await storeItemModel.findByView('inventory', { item: { $eq: itemId } });
  return row?.count ?? 0;
}
```

**Transaction** -- every movement of money (deposits, withdrawals, order payments) is one
document. A positive `amount` is money coming in, negative is money going out; `status`
starts `'pending'` and moves to `'succeeded'` once settled.

```typescript
const transactionSchema = {
  user: { type: Types.REFERENCE, reference: 'user', required: true, triggerRemove: true },
  amount: { type: Types.NUMBER, required: true, precision: 2 },
  currency: { type: Types.STRING, default: 'USD' },
  created: { type: Types.TIME, default: () => new Date().toISOString() },
  status: { type: Types.STRING, required: true, default: 'pending' },
};

const transactionModel = new Model({ database: DATABASE, collection: 'transaction', name: 'Transaction', schema: transactionSchema });

async function getUserBalance(userId) {
  const [deposits] = await transactionModel.findByView('balancePendingDeposits', { user: { $eq: userId } });
  const [withdrawals] = await transactionModel.findByView('balancePendingWithdrawals', { user: { $eq: userId } });
  const [actual] = await transactionModel.findByView('balanceActual', { user: { $eq: userId } });
  return {
    user: userId,
    amount: {
      pending_deposits: deposits?.pending_deposits ?? 0,
      pending_withdrawals: withdrawals?.pending_withdrawals ?? 0,
      actual: actual?.actual ?? 0,
    },
  };
}
```

`getUserBalance` reads from three views (`balancePendingDeposits`,
`balancePendingWithdrawals`, `balanceActual`) that don't exist yet at this point in the
file -- that's fine, since this is a function definition, not a call. They're created in
`initModels()`, and only need to exist by the time this function is actually invoked.

**Order** -- references the specific `StoreItem` units purchased (not the `Item` catalog
entries), so each order is tied to the exact physical units that left the shelf. `status`
tracks where the order sits in its lifecycle: `'toPay'` (items are reserved but not yet
paid for), `'toShip'` (paid, waiting to go out), or `'toReview'` (delivered, waiting on the
buyer) -- this example only ever inserts an order as `'toPay'` or `'toShip'`.
`amount`/`currency` capture the order's total at creation time (the sum of its items'
current prices), so the price the buyer agreed to doesn't drift if prices change later.

```typescript
const orderSchema = {
  items: { type: Types.ARRAY, items: Types.REFERENCE, reference: 'storeItem', required: true },
  user: { type: Types.REFERENCE, reference: 'user', required: true, triggerRemove: true },
  status: { type: Types.STRING, required: true, default: 'toPay' },
  amount: { type: Types.NUMBER, required: true, precision: 2 },
  currency: { type: Types.STRING, required: true, default: 'USD' },
  created: { type: Types.TIME, required: true, default: () => new Date().toISOString() },
};

const orderModel = new Model({ database: DATABASE, collection: 'order', name: 'Order', schema: orderSchema });
```

### 4. Initialize every model and create its views

`initModels()` is where everything declared above actually gets registered with the
server: it opens the connection, calls `.init()` on all seven models, and then creates
every materialized view. It's called once, at the very start of `main()`, before
dispatching to either the `init` or `flow` mode below.

```typescript
async function initModels() {
  await dianaDb.connect(1000);
  await userModel.init();
  await userLocationModel.init();
  await itemModel.init();
  await priceModel.init();
  await storeItemModel.init();
  await transactionModel.init();
  await orderModel.init();

  await priceModel.createView('currentPrice', [
    { $sort: { created: 1 } },
    { $group: { _id: '$item', item: { $first: '$item' }, price: { $last: '$amount' } } },
  ]);

  // reserved: { $ne: true } matches both reserved: false and a
  // missing/undefined reserved field.
  await storeItemModel.createView('inventory', [
    { $match: { reserved: { $ne: true } } },
    { $group: { _id: '$item', item: { $first: '$item' }, count: { $sum: [1] } } },
  ]);

  await transactionModel.createView('balancePendingDeposits', [
    { $match: { status: { $eq: 'pending' }, amount: { $gt: 0 } } },
    { $group: { _id: '$user', user: { $first: '$user' }, pending_deposits: { $sum: ['$amount'] } } },
  ]);

  await transactionModel.createView('balancePendingWithdrawals', [
    { $match: { status: { $eq: 'pending' }, amount: { $lt: 0 } } },
    { $group: { _id: '$user', user: { $first: '$user' }, pending_withdrawals: { $sum: ['$amount'] } } },
  ]);

  await transactionModel.createView('balanceActual', [
    { $match: { status: { $eq: 'succeeded' } } },
    { $group: { _id: '$user', user: { $first: '$user' }, actual: { $sum: ['$amount'] } } },
  ]);

  await orderModel.createView('orderToPay', [
    { $match: { status: { $eq: 'toPay' } } },
    // order.user and userLocation.user both hold the same user _id, so this
    // join doesn't need the user lookup to run first
    { $lookup: { database: DATABASE, collection: 'userLocation', localField: 'user', foreignField: 'user', as: 'location' } },
    { $unwind: '$location' },
    { $lookup: { database: DATABASE, collection: 'user', localField: 'user', foreignField: '_id', as: 'user' } },
    { $unwind: '$user' },
    { $project: { _id: false, location: '$location', user: true, items: true, amount: true, currency: true } },
  ]);

  await orderModel.createView('orderToShip', [
    { $match: { status: { $eq: 'toShip' } } },
    // NOTE: this must look up the `userLocation` collection, same as
    // orderToPay above -- there is no `location` collection.
    { $lookup: { database: DATABASE, collection: 'userLocation', localField: 'user', foreignField: 'user', as: 'location' } },
    { $unwind: '$location' },
    { $lookup: { database: DATABASE, collection: 'user', localField: 'user', foreignField: '_id', as: 'user' } },
    { $unwind: '$user' },
    { $project: { _id: false, location: '$location', user: true, items: true, amount: true, currency: true } },
  ]);
}
```

`orderToPay` and `orderToShip` alias the `userLocation` lookup as `location` and project it
through as-is, so the `location` field on a row from either view is the whole
`UserLocation` document (its own `_id`, its `user` reference, and the actual coordinate) --
not just the bare `{ x, y }` point. The point itself is one level deeper, at
`location.location`.

#### Querying orderToShip by location

Because the coordinate lives at `location.location` on an `orderToShip` row, querying it
spatially means matching on that nested path rather than on `location` directly.
`$nearLines` matches documents within a given `distance` of one or more line segments --
useful for something like "which orders waiting to ship sit along today's delivery route":

```typescript
const shipmentsAlongRoute = await orderModel.findByView('orderToShip', {
  'location.location': {
    $nearLines: {
      lines: [
        [{ x: 1, y: 1 }, { x: 10, y: 10 }],
        [{ x: 2, y: 0 }, { x: 4, y: 20 }],
      ],
      distance: 10,
    },
  },
});
```

Each entry in `lines` is one segment as a pair of `{ x, y }` points; `distance` is the
maximum allowed distance from any of them, in the same units as the coordinates. See the
Find and Count Documents section above for the rest of the `POSITION` query operands
(`$insideCircle`, `$insidePolygon`, `$nearLines`, and their inverses).

### 5. Create users and their locations

```typescript
const USER_COUNT = 10;

async function seedUsers() {
  const users = [];
  for (let i = 0; i < USER_COUNT; i++) {
    const user = await userModel.insert(rndUser());
    users.push(user);
    await userLocationModel.insert({ user: user._id, location: rndPosition() });
  }
  return users;
}
```

### 6. Create items and their first prices

```typescript
const ITEM_COUNT = 10;

async function seedItems() {
  const items = [];
  for (let i = 0; i < ITEM_COUNT; i++) {
    const item = await itemModel.insert(rndItem());
    items.push(item);
    await priceModel.insert({ item: item._id, amount: rndAmount(10, 100), currency: 'USD' });
  }
  return items;
}
```

### 7. Add store items (stock)

Give each item 1-5 physical units, each with its own unique `serialNumber`. `reserved` is
passed explicitly as `false` here (its schema default would apply either way):

```typescript
async function seedStoreItems(items) {
  for (const item of items) {
    const usedSerials = new Set();
    const unitCount = rndInt(1, 5);
    for (let i = 0; i < unitCount; i++) {
      await storeItemModel.insert({ item: item._id, serialNumber: rndSerialNumber(usedSerials), reserved: false });
    }
  }
}
```

### 8. Add money to users (deposits)

A deposit is just a `Transaction` with a positive `amount` and `status: 'succeeded'`. This
is what the `balanceActual` view sums up per user to produce a spendable balance:

```typescript
async function seedDeposits(users) {
  for (const user of users) {
    await transactionModel.insert({ user: user._id, amount: rndAmount(10, 100), currency: 'USD', status: 'succeeded' });
  }
}

async function runInit() {
  const users = await seedUsers();
  const items = await seedItems();
  await seedStoreItems(items);
  await seedDeposits(users);
}
```

A deposit that should wait for a bank confirmation would instead be inserted with
`status: 'pending'` -- it would count toward `balance.amount.pending_deposits` but not
`balance.amount.actual` until it's updated to `'succeeded'`.

### 9. Create orders

The following runs once per user (for example inside a `processOrder(user, items)` step
called in a loop). Creating an order checks two things, not just one: does the user have
enough money, and is there actually stock left to sell. Start with availability -- for each
candidate item, ask the `inventory` view whether any unreserved units exist, then pull the
actual list of available `StoreItem` documents for the ones that do:

```typescript
async function pickAvailableStoreItems(candidateItems) {
  const chosen = [];

  for (const item of candidateItems) {
    // Availability check #1: does the inventory view show any unreserved stock at all?
    const [inventory] = await storeItemModel.findByView('inventory', { item: { $eq: item._id } });
    if (!inventory || inventory.count < 1) continue;

    // Pull the actual list of unreserved units for this item -- the view only
    // gives a count, not which units are free
    const [storeItem] = await storeItemModel.find(
      [{ item: { $eq: item._id }, reserved: { $ne: true } }],
      undefined, undefined, 0, 1,
    );
    if (storeItem) chosen.push(storeItem);
  }

  return chosen;
}

async function orderTotal(storeItems) {
  let total = 0;
  for (const storeItem of storeItems) {
    const price = await getCurrentPrice(storeItem.item);
    total += price ?? 0;
  }
  return Math.round(total * 100) / 100;
}

const candidateItems = rndSample(items, rndInt(1, 5));
const chosen = await pickAvailableStoreItems(candidateItems);
if (chosen.length === 0) return; // nothing available right now, try the next user

const total = await orderTotal(chosen);
```

Then open a transaction scoped to the shared database and reserve the chosen units before
the order references them: the update's filter requires `reserved: { $ne: true }`, so a
unit another order grabbed a moment earlier won't match. A follow-up read confirms every
unit really did flip to `reserved: true` for this user -- and this is the only condition
that rolls the transaction back. Money is handled differently: if the balance covers the
total, a debiting `Transaction` is inserted and the order goes out as `'toShip'`, carrying
its `amount`/`currency`; if it doesn't, the order is still created -- as `'toPay'`, with the
same `amount`/`currency` and its items still reserved -- and simply waits to be paid later:

```typescript
const transactionId = await dianaDb.startTransaction({ database: DATABASE, autoRollBackAfterMs: 60000 });
const txnParams = { database: DATABASE, transactionId };

try {
  // Reserve first. The reserved: { $ne: true } filter means this only
  // matches units nobody else has claimed since we read them above.
  await storeItemModel.update(
    [{ _id: { $in: chosen.map((si) => si._id) }, reserved: { $ne: true } }],
    { reserved: true, reservedFor: user._id },
    transactionId,
  );

  // Availability check #2: confirm the reservation actually stuck for every
  // chosen unit. If a concurrent order beat us to one of them, it will still
  // show reserved: false (or reservedFor someone else) and be missing here.
  // This is the only condition that rolls the transaction back.
  const confirmedReserved = await storeItemModel.find(
    [{ _id: { $in: chosen.map((si) => si._id) }, reserved: { $eq: true }, reservedFor: { $eq: user._id } }],
    undefined, undefined, 0, chosen.length,
    transactionId,
  );

  if (confirmedReserved.length !== chosen.length) {
    await dianaDb.rollbackTransaction(txnParams);
    return;
  }

  const balance = await getUserBalance(user._id);
  const orderItems = confirmedReserved.map((si) => si._id);

  if (balance.amount.actual >= total) {
    // Enough money: charge now, ship now. The items stay reserved -- they're
    // now owned by this order rather than up for grabs again.
    await transactionModel.insert(
      { user: user._id, amount: -total, currency: 'USD', status: 'succeeded' },
      transactionId,
    );
    await orderModel.insert(
      { items: orderItems, user: user._id, status: 'toShip', amount: total, currency: 'USD' },
      transactionId,
    );
  } else {
    // Not enough money yet: no transaction is created, but the order still
    // exists -- toPay -- holding the reservation until it's settled.
    await orderModel.insert(
      { items: orderItems, user: user._id, status: 'toPay', amount: total, currency: 'USD' },
      transactionId,
    );
  }

  await dianaDb.commitTransaction(txnParams);
} catch (err) {
  await dianaDb.rollbackTransaction(txnParams);
}
```

Reserving before inserting the `Order` means the reservation is the thing that's actually
contested between concurrent orders. Insufficient inventory is the only thing that rolls
this transaction back -- an insufficient balance still produces an order, just as `'toPay'`
instead of `'toShip'`, with its `StoreItem` units held in reserve rather than released. A
separate "settle payment" step (not shown here) would insert the debiting `Transaction`
later and update the order's `status` to `'toShip'`.
`autoRollBackAfterMs: 60000` is a safety net that rolls the transaction back automatically
if it's never explicitly committed or rolled back within 60 seconds.

---

## Changelog

### 2026-10-17 — Server v3.0.0 / ODM v3.0.0 (breaking)
Breaking:
- Server 3.0.0 requires ODM >= 3.0.0 for subscriptions: subscriber connections must send a token; ODM 3.0.0 also works with older servers, so upgrade it first
- Subscribers receive the `DatabaseUpdate` itself (`u.database`), not the whole server message (`u.data.database`)
- `migrateUp` / `migrateDown` need a database (argument, `database` option or connection string path) and resolve with `MigrationInfo[]`; `migrateDown` runs in descending index order
- Failed requests and `connect()` reject with an `Error` carrying a `code` (`SERVER`, `DURABILITY`, `TIMEOUT`, `CONNECTION`) instead of a string
- NUMBER fields without `precision` are stored as given; before, they were rounded to integers

Features:
- Write-ahead log (`wal`, `wal_fsync`) so acknowledged writes survive a crash; atomic snapshot writes; save on `SIGINT`/`SIGTERM`; first periodic backup one interval after start
- `precision` optional; server stores derived user keys only; logs contain request metadata only
- ODM: `unsubscribe`, several subscribers per key, `getMigrations`, `migrateDown(database, steps)`, `getCollectionSchemaLock`, `requestTimeoutValue` and `database` options, `dianaDb` model option, connection string path / URL-decoded credentials / `requestTimeoutValue` / `rejectUnauthorized`
- ODM: requests queue while all connections reconnect and fail at once when a connection drops or the client is disconnected; TLS options accept Buffers or strings and verify the certificate against `host`

Fixes:
- Message framing on both sides (a line split before its base64 padding broke the connection; the ODM dropped chunks without a newline); large messages are no longer quadratic — a 120 MB `find` response arrives in ~2.6 s
- TTL removed documents immediately; `$cn`/`$nc` treated the value as a regex; transactions without `autoRollBackAfterMs` expired immediately; a failed commit could apply part of a transaction; committed transaction updates stored precision values unscaled; update pointers were evaluated once for all matched documents; negative/fractional numbers and hour/weekday 0 stopped matching equality queries
- Publishes reached subscribers without read access; databases could be created without access rights
- ODM: static `Model.init()` didn't initialize models; concurrent `init()` raced; `startTransaction` resolved to an object; connection string parameters were ignored; missing `connectionPoolSize` created no connections; `$toNextWeekday`/`$toLastWeekday` were rejected; `disconnect()` left the subscriber connection and a timer open; a second client re-routed existing models

Performance (100k documents): removing 1,000 documents 61 s → 38 ms; updating a boolean on 10% 4 s → 51 ms; sort + limit 622 → 107 ms; per-document view updates; about 23% less memory per document

### 2026-08-17 — Server v2.2.8–v2.5.3 / ODM v2.2.8–v2.5.3
- Client-level database management: `createDatabase`, `removeDatabase`, `createCollection`, `removeCollection`
- Added `replaceRoot` to `LookupQuery`
- Core optimizations and refactoring

### 2026-07-14 — Server v2.2.7 / ODM v2.2.7
- Core optimization; refactored aggregation framework; refactored the Node.js ODM

### 2026-07-13 / 07-12 — Server v2.2.3–v2.2.6 / ODM v2.2.1
- Core optimization; refactored collection schema updating; refactored schema locking and unlocking

### 2026-07-10 — Server v2.2.2 / ODM v2.2.1
- Refactored `min`, `max` and `distinct` query performance; improved collection schema update handling; streamlined default value assignment

### 2026-07-09 — Server v2.2.1 / ODM v2.2.1
- Added `Client.getVersion()`; refactored collection schema updating

### 2026-07-06 — Server v2.1.3 / ODM v2.1.0
- Added `Model.removeView(view)`, `Model.getViews()`, `Client.getCollectionViews(db, collection)`
- Fixed return type mismatches between server and ODM in several methods

### 2026-07-06 — Server v2.1.2 / ODM v2.0.0 (breaking)
- TLS/mTLS support for client-server connections
- Storage strategies: `local` (previous default) and `s3`
- Collection schema locking and unlocking
- Document-level locking for transactions
- Storage engine optimization; added the missing `remove-backup` CLI command
- **Server >= 2.0.0 requires ODM >= 2.0.0**

### 2026-02-24 — Server v1.9.2 / ODM v1.9.2
- Added `getDatabases()` and `getHealth()` client methods

### 2026-02-23 — Server v1.9.1 / ODM v1.9.1
- Added `uppercase` and `lowercase` schema options for string fields

### 2026-02-19 — Server v1.9.0 / ODM v1.9.0
- Renamed array update operators `$add` → `$addItem` and `$remove` → `$removeItem` to avoid clashing with numeric `$add`
- Fixed ACID transaction consistency; refactored the migration engine; optimized temporal query execution

### 2026-02-18 — Server v1.8.3 / ODM v1.8.1
- Added `countByView`; added real-time server performance tracking; fixed broken projection in `$group`

### 2026-02-16 — Server v1.8.1–v1.8.2 / ODM v1.8.0
- Connection string support in the ODM; optimized encryption and internal processing; refactored data restoring

### 2026-02-07 — Server v1.6.0–v1.7.0 / ODM v1.6.0–v1.7.0 (breaking at 1.7.0)
- Materialized views updating on the fly; objects as `_id` in `$group`
- Access management and backup management in the CLI
- Fixed pointer evaluation inside transform queries; fixed `$match` for `$nc`, `$cn`, `$notStartsWith`, `$notEndsWith`
- **Server >= 1.7.0 requires ODM >= 1.7.0**

### 2026-01-17 — Server v1.5.0 / ODM v1.5.0
- Added `max` and `min` model methods; added `$max`, `$min`, `$avg` to `$group` and `$project`
- Fixed the data restoration mechanism

### 2026-01-13 / 01-16 — Server v1.4.8–v1.4.9 / ODM v1.4.11–v1.4.12
- Added the `distinct` model method; improved encryption; refactored connection management and FilterQuery validation

---

## Upcoming Features

- Desktop Management Tool — GUI for managing databases
- Query Language Improvements — making DiQL more expressive
- Golang, PHP, C# and Java clients
- Vectors — vector data type and similarity search
- Multi-Server Replication — real-time synchronization and high availability across nodes in different data centres

---

## License

MIT License with Usage Restriction. Copyright © 2025 Data Bikers Limited.

Permission is granted, free of charge, to use, copy, modify, merge, publish, distribute, sublicense and sell copies of the Software, subject to:

- The copyright notice and permission notice being included in all copies or substantial portions of the Software.
- **The Software may not be used to create or offer a competing product or service without prior written consent from Data Bikers Limited.**

The Software is provided "as is", without warranty of any kind.

---

## Links

- Website: https://diana-db.com
- npm: https://www.npmjs.com/package/@diana-db/odm
- GitHub: https://github.com/databikers/diana-db
- Issues: https://github.com/databikers/diana-db/issues
- Benchmark: https://github.com/databikers/db-benchmark
- Data Bikers Limited: https://databikers.com