# DianaDB

> DianaDB is a modern, column-oriented NoSQL DBMS written in Node.js. It stores documents in a columnar format where every field acts as an index, enabling exceptional query performance. It supports spatial queries, time-based queries, cross-database lookups, ACID transactions, subscriptions, migrations, and materialized views.

DianaDB is authored and maintained by Data Bikers Limited (Hong Kong). It is conceptually similar to MongoDB but architecturally distinct — columnar storage, schema-enforced, with a TypeScript-native ODM.

---

## What is DianaDB

DianaDB is a NoSQL, column-oriented database. Although it operates with documents (similar to objects or structs) and collections, it stores data in a highly optimized columnar format. Each document is decomposed into separate columns — each column tailored to a specific data type with its own dedicated logic for storage and processing. Data integrity is enforced through document schemas that define the expected structure within each collection. This architecture allows every field in a document to function like an index, enabling exceptional query performance.

DianaDB supports relationships between documents across different collections — including triggers similar to relational databases. It supports cross-database lookups on the same server. It provides a special TIME type with powerful utilities for calendar and scheduling queries, and a POSITION type for spatial/geometric queries. It supports ACID transactions, a built-in migration framework, and subscriptions to database changes per collection or globally.

---

## Server Installation

### Linux — Debian

```bash
sudo echo "deb [arch=amd64 trusted=yes] https://dist.databikers.com stable main" | sudo tee /etc/apt/sources.list.d/diana-db.list
sudo apt-get update
sudo apt-get install -y diana-db
```

```bash
sudo service diana-db [status|start|stop]
```

### Linux — Alpine

```bash
wget -q -O /etc/apk/keys/sgerrand.rsa.pub https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub
wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.35-r1/glibc-2.35-r1.apk
apk add glibc-2.35-r1.apk
wget https://dist.databikers.com/x86_64/diana-db-1.5.0-r0.apk
apk add --allow-untrusted ./diana-db-1.5.0-r0.apk
```

```bash
sudo rc-service diana-db [status|start|stop]
```

### CLI Commands

| Command       | Flags                          | Description                                      |
|---------------|--------------------------------|--------------------------------------------------|
| `start`       | `-c <conf>`                    | Stops any running instance, launches server      |
| `stop`        | `-c <conf>`                    | Kills running diana-db-server process            |
| `add-user`    | `-u <user> -p <pass> -c <conf>`| Creates a user with provided credentials         |
| `remove-user` | `-u <user> -c <conf>`          | Removes the given user                           |

### Config File (`/etc/diana-db/diana-db.conf`)

| Key                                    | Env Variable                            | Type         | Description                              |
|----------------------------------------|-----------------------------------------|--------------|------------------------------------------|
| `port`                                 | `DIANA_DB_PORT`                         | Integer      | Listening TCP port (default: 34567)      |
| `dump_create_interval`                 | `DIANA_DB_DUMP_CREATE_INTERVAL`         | Integer (ms) | How often to create DB snapshots         |
| `logs_directory`                       | `DIANA_DB_LOG_DIRECTORY`                | String path  | Directory for server logs                |
| `dump_directory`                       | `DIANA_DB_DUMP_DIRECTORY`               | String path  | Directory for DB dumps                   |
| `logs_ttl_value`                       | `DIANA_DB_LOG_TTL`                      | Integer (ms) | Log retention TTL                        |
| `current_dump_name`                    | `DIANA_DB_DUMP_CURRENT_NAME`            | String       | Filename for the latest dump             |
| `transactions_min_auto_rollback_value` | `DIANA_DB_TRANSACTION_AUTOROLLBACK_MIN` | Integer (ms) | Min time before auto-rollback            |
| `transactions_max_auto_rollback_value` | `DIANA_DB_TRANSACTION_AUTOROLLBACK_MAX` | Integer (ms) | Max time before auto-rollback            |

---

## Docker Installation

### Debian Dockerfile

```bash
# entrypoint.sh
#!/bin/sh
set -e
/usr/bin/diana-db add-user -u admin -p admin
/usr/bin/diana-db-server
```

```dockerfile
FROM databikers/diana-db:debian
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
```

### Alpine Dockerfile

```bash
# entrypoint.sh
#!/bin/sh
set -e
/usr/bin/diana-db add-user -u admin -p admin
exec /usr/bin/diana-db-server
```

```dockerfile
FROM databikers/diana-db:alpine
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
```

### Run

```bash
docker run -p 34567:34567 local-diana-db
```

### Docker Compose

```yaml
version: '3.8'
services:
  diana-db:
    image: local-diana-db
    ports:
      - "34567:34567"
    volumes:
      - ./data/etc:/etc/diana-db
      - ./data/lib:/var/lib/diana-db
      - ./data/log:/var/log/diana-db
    restart: unless-stopped
    container_name: diana-db
    entrypoint: ["/entrypoint.sh"]
```

---

## ODM Overview

Install:

```bash
npm i -s @diana-db/odm
```

Note: Diana DB Server 1.5.0 requires ODM version >= 1.5.0.

### Create a DianaDB Instance

```typescript
import { DianaDb } from '@diana-db/odm'

const dianaDb = new DianaDb({
  host: 'localhost',
  port: 34567,
  user: 'db-user',
  password: 'some-password',
  connectionPoolSize: 10,
  connectTimeoutValue: 5000,
});
```

You can create as many instances as you like. Each has its own models, migrations, and subscribers and can be connected/disconnected independently.

### Constructor Options

| Property             | Description                        | Type             | Default |
|----------------------|------------------------------------|------------------|---------|
| `host`               | Server host                        | String           | –       |
| `port`               | Server port                        | Positive integer | 34567   |
| `user`               | Username                           | String           | –       |
| `password`           | Password                           | String           | –       |
| `connectionPoolSize` | Max concurrent connections         | Positive integer | 10      |
| `connectTimeoutValue`| Connection timeout in milliseconds | Positive integer | 10000   |

### Client Methods

| Method                | Arguments                        | Description                              | Returns                        |
|-----------------------|----------------------------------|------------------------------------------|--------------------------------|
| `connect`             | `connectTimeoutValue`            | Connects to the server                   | `Promise<void>`                |
| `disconnect`          | –                                | Disconnects from the server              | `void`                         |
| `subscribe`           | `subscribeParameters`            | Sets a subscriber for DB changes         | `void`                         |
| `setMigration`        | `migrationOptions`               | Registers a migration script             | `void`                         |
| `startTransaction`    | `{ database, autoRollBackAfterMs }` | Starts a transaction                  | `Promise<string>` (transactionId) |
| `commitTransaction`   | `{ database, transactionId }`   | Commits an open transaction              | `Promise<TransactionInfo>`     |
| `rollbackTransaction` | `{ database, transactionId }`   | Rolls back a transaction                 | `Promise<TransactionInfo>`     |
| `migrateUp`           | –                                | Applies all pending migrations           | `Promise<void>`                |
| `migrateDown`         | –                                | Rolls back all applied migrations        | `Promise<void>`                |

---

## Document Schema

A schema is a configuration object that defines the structure of stored documents. Schema keys map to document properties, values are field configuration objects.

### Schema Field Options

| Option         | Type                                       | Description                                              | Required                          | Default |
|----------------|--------------------------------------------|----------------------------------------------------------|-----------------------------------|---------|
| `type`         | Types                                      | Field type                                               | Yes                               | –       |
| `required`     | boolean                                    | Property is required                                     | No                                | false   |
| `unique`       | boolean                                    | Property must be unique                                  | No                                | false   |
| `mutable`      | boolean                                    | Value can be changed after creation                      | No                                | true    |
| `lowercase`    | boolean                                    | Convert string to lowercase                              | No                                | false   |
| `uppercase`    | boolean                                    | Convert string to uppercase                              | No                                | false   |
| `default`      | `T[prop]`, `() => T[prop]`, or async fn   | Default value or factory function                        | No                                | –       |
| `items`        | Types                                      | Item type for ARRAY fields                               | Yes (if type is ARRAY)            | –       |
| `reference`    | string                                     | Name of referenced collection                            | Yes (if type is REFERENCE or items is REFERENCE) | – |
| `triggerRemove`| boolean                                    | Remove document when referenced item is deleted          | No                                | false   |
| `ttl`          | number                                     | Time to live in ms after insertion (TIME fields only)    | No                                | –       |

### Field Types

| Type              | Description                                              | Notes                                    |
|-------------------|----------------------------------------------------------|------------------------------------------|
| `Types.STRING`    | String value                                             | –                                        |
| `Types.NUMBER`    | Numeric value                                            | –                                        |
| `Types.BOOLEAN`   | Boolean value                                            | –                                        |
| `Types.POSITION`  | Object with `x` and `y` coordinates                     | Enables spatial/geometric queries        |
| `Types.OBJECT_ID` | Hex string document identifier                           | Not BSON; pattern: `/^[0-9a-f]{12}-[0-9a-f]{1,}-[0-9a-f]{11}-[0-9a-f]{9}$/` |
| `Types.REFERENCE` | Like OBJECT_ID but linked to another collection          | Requires `reference`. Enables sub-queries |
| `Types.TIME`      | ISO date/time string                                     | Supports time-based filters and TTL      |
| `Types.ARRAY`     | Homogeneous array; item type defined by `items`          | Mixed types not allowed; set-like (unique values only) |

### Schema Examples

```typescript
import { Types } from '@diana-db/odm';

export const userSchema = {
  name: {
    type: Types.STRING,
    unique: true,
    required: true,
    lowercase: true
  },
  isActive: {
    type: Types.BOOLEAN,
    default: true
  },
  position: {
    type: Types.POSITION
  },
  createdAt: {
    type: Types.TIME,
    default: () => new Date().toISOString()
  }
};

export const postSchema = {
  user: {
    type: Types.REFERENCE,
    reference: 'user',
    required: true,
    triggerRemove: true
  },
  title: { type: Types.STRING, required: true },
  content: { type: Types.STRING, required: true },
  isPublished: { type: Types.BOOLEAN, default: true },
  createdAt: { type: Types.TIME, required: true },
  publishedAt: { type: Types.TIME, required: true }
};
```

> Warning: Schema changes are automatically detected and applied on the server side. You may lose data if fields are removed. Store schemas in a single shared package to prevent accidental divergence.

---

## Model

```typescript
import { Model } from '@diana-db/odm';

const userModel = new Model<User>({
  database: 'test',
  collection: 'user',
  name: 'User',
  schema: userSchema
});
```

### Model Options

| Property     | Description                        |
|--------------|------------------------------------|
| `database`   | Database name                      |
| `collection` | Collection name                    |
| `name`       | Model name                         |
| `schema`     | Schema definition object           |

### Model Methods

| Method                                                    | Description                                      | Returns                                    |
|-----------------------------------------------------------|--------------------------------------------------|--------------------------------------------|
| `insert(data, transactionId?)`                            | Validates and inserts a document                 | `Promise<T & { _id: string }>`             |
| `find(filters, transforms?, sort?, skip?, limit?, transactionId?)` | Find documents                        | `Promise<T[]>`                             |
| `count(filters, transforms?, transactionId?)`             | Count matching documents                         | `Promise<number>`                          |
| `update(filters, updateData, transactionId?)`             | Update matching documents                        | `Promise<{ found: number, modified: number }>` |
| `remove(filters, transactionId?)`                         | Remove matching documents                        | `Promise<{ found: number, removed: number }>` |
| `distinct(key)`                                           | Get unique values for a key (ODM >= 1.4.10)      | `Promise<any[]>`                           |
| `max(key)`                                                | Documents with maximum value for key (ODM >= 1.5.0) | `Promise<T[]>`                          |
| `min(key)`                                                | Documents with minimum value for key (ODM >= 1.5.0) | `Promise<T[]>`                          |
| `createView(name, transforms)`                            | Create a materialized view                       | `Promise<void>`                            |
| `findByView(name, filter?, sort?, skip?, limit?)`         | Query a materialized view                        | `Promise<any[]>`                           |
| `countByView(name, filter?)`                              | Count results in a materialized view             | `Promise<number>`                          |

---

## Insert Documents

```typescript
const user = await userModel.insert({
  name: 'John',
  position: { x: 1, y: 1 },
  createdAt: new Date().toISOString()
});
// Returns: { _id: 'dc4628514b81-939f-198a25f76b7-bb1a9a777', name: 'john', position: { x: 1, y: 1 }, isActive: true, createdAt: '2025-06-18T06:30:24.977Z' }
```

The `_id` property is added automatically. The returned document is a plain object with no special methods.

---

## Find and Count Documents

```typescript
const [user] = await userModel.find(
  [{ name: { $eq: 'John' } }],    // FindQuery (OR array)
  [{ $project: { name: true } }], // Transform Queries
  { _id: -1 },                    // Sort
  0,                              // Skip
  1,                              // Limit
);
```

### Find Arguments

| Argument          | Description                                                             |
|-------------------|-------------------------------------------------------------------------|
| `FindQuery`       | Object or array of objects. Keys match schema fields. Array = OR logic. |
| `TransformQueries`| Array of transformation pipeline objects                                |
| `SortingQuery`    | `{ field: 1 }` ascending, `{ field: -1 }` descending                   |
| `Skip`            | Integer >= 0                                                            |
| `Limit`           | Positive integer                                                        |
| `transactionId`   | String — execute within a transaction                                   |

### Query Operands

| Operand                                                | Types          | Description                                      |
|--------------------------------------------------------|----------------|--------------------------------------------------|
| `$eq`                                                  | All            | Equals value (also: `field: value`)              |
| `$ne`                                                  | All            | Not equal                                        |
| `$in`                                                  | All            | Value in array (also: `field: [a, b]`)           |
| `$nin`                                                 | All            | Value not in array                               |
| `$gt`, `$gte`, `$lt`, `$lte`                           | Number, TIME   | Numeric/time comparisons                         |
| `$regex`                                               | String         | Regex or substring match                         |
| `$startsWith`, `$endsWith`, `$notStartsWith`, `$notEndsWith` | String  | String prefix/suffix matching                    |
| `$cn`, `$nc`                                           | String         | Contains / not contains                          |
| `$year`, `$month`, `$date`, `$week`, `$hours`, `$minutes`, `$dayOfYear`, `$dayOfWeek`, `$timeStamp` | TIME | Query on date parts |
| `$insideCircle`, `$outsideCircle`                      | POSITION       | Circle spatial query                             |
| `$insidePolygon`, `$outsidePolygon`                    | POSITION       | Polygon spatial query                            |
| `$nearLines`, `$farFromLines`                          | POSITION       | Proximity to line segments                       |
| `$subQuery`                                            | REFERENCE      | Nested query on referenced collection            |

#### Spatial query examples

```typescript
// Circle
{ position: { $insideCircle: { center: { x: 1, y: 1 }, radius: 10 } } }

// Polygon
{ position: { $insidePolygon: [{ x: 1, y: 1 }, { x: 3, y: 0 }, { x: -1, y: -1 }] } }

// Near lines
{ position: { $nearLines: { lines: [[{ x: 1, y: 1 }, { x: 10, y: 10 }]], distance: 10 } } }
```

#### TIME query example

```typescript
{ createdAt: { $year: { $eq: 2025 }, $month: { $in: [1, 4] }, $dayOfWeek: { $in: [0, 1] } } }
```

#### Reference sub-query example

```typescript
{ user: { $subQuery: { name: { $eq: 'John' } } } }
```

#### Array query example

```typescript
// Documents where someArrayProperty contains 1 and 2 but NOT 3
{ someArrayProperty: { $in: [1, 2], $nin: [3] } }
```

### Transform Queries (Pipelines)

| Pipeline       | Description                                                               |
|----------------|---------------------------------------------------------------------------|
| `$project`     | Shape/rename/compute fields on each document                              |
| `$group`       | Group documents by `_id` field; supports aggregation operands             |
| `$match`       | Filter documents mid-pipeline using Find Query syntax                     |
| `$lookup`      | Join from another collection/database (`database`, `collection`, `localField`, `foreignField`, `as`, `filter`) |
| `$unwind`      | Deconstruct array field — one doc per array item                          |
| `$replaceRoot` | Replace document with the value of a nested property                      |
| `$sort`        | Sort by field(s): `1` ascending, `-1` descending                          |
| `$skip`        | Skip N documents                                                          |
| `$limit`       | Limit to N documents                                                      |

### Projection Operands

| Operand          | Description                                                    |
|------------------|----------------------------------------------------------------|
| `$sum`           | Sum array of values/pointers                                   |
| `$subtract`      | Subtract values                                                |
| `$multiply`      | Multiply values                                                |
| `$divide`        | Divide values                                                  |
| `$round`         | Round to precision                                             |
| `$max`           | Maximum from pointer values                                    |
| `$min`           | Minimum from pointer values                                    |
| `$avg`           | Average from pointer values                                    |
| `$ifNull`        | Fallback if value is null/undefined                            |
| `$push`          | Append to array                                                |
| `$addToSet`      | Append unique to array                                         |
| `$concatArrays`  | Concatenate array property                                     |
| `$first`         | First value                                                    |
| `$last`          | Last value                                                     |
| `$concat`        | Concatenate strings with delimiter                             |
| `$year`, `$month`, `$date`, `$dayOfWeek`, `$dayOfYear`, `$week`, `$hours`, `$minutes`, `$seconds`, `$timestamp` | Extract TIME parts |

> Note (ODM >= 1.4.8): Math operands never implicitly use the existing property value. To include it, use a pointer: `{ $sum: ['$amount', 123] }`

### Transform Query Example

```typescript
const result = await postModel.find(
  [{}],
  [
    { $group: { _id: 'user', user: true, postsCount: { $sum: [1] } } },
    { $lookup: { database: 'test', collection: 'user', localField: 'user', foreignField: '_id', as: 'user', filter: { isActive: { $eq: true } } } },
    { $unwind: 'user' },
    { $project: { _id: false, user: true, postsCount: true } },
    { $sort: { postsCount: -1, name: 1 } },
    { $skip: 1 },
    { $limit: 1 },
  ],
);
```

---

## Update Documents

```typescript
await userModel.update(
  [{ name: { $eq: 'John' } }],
  { isActive: false }
);
```

### Update Operators by Field Type

**String:**

| Operator   | Type             | Effect                                  |
|------------|------------------|-----------------------------------------|
| `$concat`  | string           | Appends to current string value         |
| `$replace` | `[string, string]` | Replaces substring `[search, replacement]` |

**Number:**

| Operator    | Type   | Effect                      |
|-------------|--------|-----------------------------|
| `$add`      | number | Add to current value        |
| `$subtract` | number | Subtract from current value |
| `$multiply` | number | Multiply current value      |
| `$divide`   | number | Divide current value        |
| `$round`    | number | Round to precision          |

**Array:**

| Operator  | Type | Effect                                    |
|-----------|------|-------------------------------------------|
| `$add`    | any  | Append if not already present             |
| `$remove` | any  | Remove element if found                   |

**TIME:**

| Operator         | Structure               | Effect                              |
|------------------|-------------------------|-------------------------------------|
| `$add`           | `{ amount, unit }`      | Add time amount                     |
| `$subtract`      | `{ amount, unit }`      | Subtract time amount                |
| `$toStartOf`     | UnitOfTime              | Round down to start of unit         |
| `$toEndOf`       | UnitOfTime              | Round up to end of unit             |
| `$toNextWeekDay` | Weekday (0–6)           | Jump forward to next weekday        |
| `$toLastWeekDay` | Weekday (0–6)           | Jump back to previous weekday       |

UnitOfTime values: `"year"`, `"month"`, `"week"`, `"day"`, `"hour"`, `"minute"`, `"second"`, `"millisecond"`

Weekday values: `0` (Sunday) through `6` (Saturday)

---

## Remove Documents

```typescript
const { found, removed } = await userModel.remove([{ isActive: false }]);
```

---

## Subscribe to Database Updates

```typescript
// 'user.post' = database 'user', collection 'post'
// 'user' alone = all collections in 'user' database
await dianaDb.subscribe('user.post', (databaseUpdate) => {
  // handle update
});
```

### DatabaseUpdate Contract

| Field        | Type                          | Description                                |
|--------------|-------------------------------|--------------------------------------------|
| `database`   | string                        | Database name                              |
| `collection` | string                        | Collection name                            |
| `action`     | `'insert' \| 'update' \| 'remove'` | Operation type                        |
| `affectedIds`| string[]                      | ObjectIds affected by the operation        |
| `data`       | object                        | Document snapshot for insert/update        |

---

## Migrations

```typescript
dianaDb.setMigration({
  index: 1,            // unique integer index
  name: 'MyFirstMigration', // unique name
  up: async () => {
    // apply changes
  },
  down: async () => {
    // revert changes
  }
});

await dianaDb.connect();
await dianaDb.migrateUp();   // apply all pending
await dianaDb.migrateDown(); // rollback all applied
```

Migration names and indexes must be unique. Never modify the code inside an existing migration after it has run.

---

## Transactions

DianaDB supports non-blocking ACID transactions. Notes:
- Transactions have higher priority than regular requests and can overwrite previously saved data
- Each next transaction has a higher priority than the previous one — prefer update operators over setting explicit values
- Transaction changes are isolated until committed

```typescript
await dianaDb.connect();

const transactionId = await dianaDb.startTransaction({
  database: 'user',
  autoRollBackAfterMs: 60000
});

const manageTransactionParameters = {
  database: 'user',
  transactionId: transactionId
};

try {
  await userModel.remove([{ ...filters }], transactionId);
  await dianaDb.commitTransaction(manageTransactionParameters);
} catch (e) {
  await dianaDb.rollbackTransaction(manageTransactionParameters);
}
```

---

## Materialized Views

DianaDB allows creating views based on predefined aggregations that automatically update when the underlying collection data changes. Views support cross-database lookups.

```typescript
// Create the model
const transactionModel = new Model({
  database: 'test',
  collection: 'transaction',
  name: 'Transaction',
  schema: {
    user: { type: Types.REFERENCE, reference: 'user', required: true, triggerRemove: true },
    currency: { type: Types.STRING, required: true },
    amount: { type: Types.NUMBER, default: 0, precision: 0 },
    status: { type: Types.STRING, default: 'pending' },
    created: { type: Types.TIME, default() { return new Date().toISOString(); } },
    updated: { type: Types.TIME, default() { return new Date().toISOString(); } },
  },
});

await transactionModel.init();

// Create a materialized view
await transactionModel.createView('balance', [
  {
    $group: {
      _id: { user: '$user', currency: '$currency' },
      user: { $first: '$user' },
      amount: { $sum: ['$amount'] },
      currency: { $first: '$currency' },
    },
  },
]);

// Query the view
const balances = await transactionModel.findByView(
  'balance',
  { user: { $eq: someUser._id } },
  { amount: -1 }
);

const count = await transactionModel.countByView(
  'balance',
  { user: { $eq: someUser._id } }
);
```

### createView Arguments

| Argument           | Required | Description                                |
|--------------------|----------|--------------------------------------------|
| `name`             | Yes      | Unique view name (not used as collection)  |
| `TransformQueries` | Yes      | Aggregation pipeline defining the view     |

### findByView Arguments

| Argument       | Required | Description                           |
|----------------|----------|---------------------------------------|
| `name`         | Yes      | View name                             |
| `FindQuery`    | No       | Filter on view results                |
| `SortingQuery` | No       | Sort results                          |
| `Skip`         | No       | Skip N results                        |
| `Limit`        | No       | Limit to N results                    |

---

## Cook Book: A Mini Shop Example

An end-to-end example combining schemas, materialized views, seeding, and a transactional
order flow: users, items, prices, store inventory, adding money to a user's balance, and
creating orders against it.

### Setup

Before the client code below can connect, create a server user and grant it access to the
database this example uses:

```bash
sudo diana-db add-user -u admin -p password
sudo diana-db grant-access -u admin -db test:3
```

### Install

```bash
npm i @diana-db/odm
```

This example has no other dependencies -- the random data used to seed it (names, emails,
birthdates, coordinates, product copy) is generated by small local helper functions, not a
third-party library.

### 1. Connect

Every collection shares one `database` name so they can all participate in the same
transaction later -- `startTransaction` / `commitTransaction` are scoped to a single
database. The `DianaDb` instance is constructed here, but the actual `connect()` call
happens in `initModels()` below, alongside every model's `.init()`.

```typescript
const DATABASE = 'data';

const dianaDb = new DianaDb({
  host: process.env.DIANA_DB_HOST || 'localhost',
  port: Number(process.env.DIANA_DB_PORT || 34567),
  user: process.env.DIANA_DB_USER || 'admin',
  password: process.env.DIANA_DB_PASSWORD || 'password',
  connectionPoolSize: 3,
  connectTimeoutValue: 1000,
});
```

### 2. Random data helpers

The seed step (further down) never hand-writes sample values -- it calls a small set of
helper functions instead. All of it is plain JS: names, emails, birthdates, coordinates,
and product copy are built from small local word pools and basic math, with no external
package involved.

```typescript
function rndInt(min, max) {
  return Math.floor(Math.random() * (max - min + 1)) + min;
}

function rndAmount(min, max) {
  return Math.round((Math.random() * (max - min) + min) * 100) / 100;
}

function rndSample(arr, count) {
  const n = Math.min(count, arr.length);
  const shuffled = [...arr].sort(() => Math.random() - 0.5);
  return shuffled.slice(0, n);
}

function rndPick(arr) {
  return arr[rndInt(0, arr.length - 1)];
}

function rndSex() {
  return Math.random() < 0.5 ? 'm' : 'f';
}

// A plausible date of birth for an 18-75 year old, as an ISO date string.
function rndDob() {
  const msPerYear = 365.25 * 24 * 60 * 60 * 1000;
  const minAgeMs = 18 * msPerYear;
  const maxAgeMs = 75 * msPerYear;
  const ts = Date.now() - (minAgeMs + Math.random() * (maxAgeMs - minAgeMs));
  return new Date(ts).toISOString().slice(0, 10);
}

function rndTags() {
  const pool = ['new', 'vip', 'wholesale', 'trial', 'referred', 'newsletter'];
  return rndSample(pool, rndInt(0, 3));
}

const FIRST_NAMES = [
  'James', 'Mary', 'John', 'Patricia', 'Robert', 'Jennifer', 'Michael', 'Linda',
  'William', 'Elizabeth', 'David', 'Barbara', 'Richard', 'Susan', 'Joseph', 'Jessica',
  'Thomas', 'Sarah', 'Charles', 'Karen', 'Somchai', 'Malee', 'Anong', 'Chai',
];

const LAST_NAMES = [
  'Smith', 'Johnson', 'Williams', 'Brown', 'Jones', 'Garcia', 'Miller', 'Davis',
  'Rodriguez', 'Martinez', 'Wilson', 'Anderson', 'Taylor', 'Moore', 'Jackson', 'Martin',
  'Saetang', 'Charoen', 'Suksawat', 'Wongsawat',
];

const EMAIL_DOMAINS = ['example.com', 'mail-test.dev', 'demo-inbox.io'];

function rndFullName() {
  return { firstname: rndPick(FIRST_NAMES), lastname: rndPick(LAST_NAMES) };
}

function rndEmail(firstname, lastname) {
  const domain = rndPick(EMAIL_DOMAINS);
  return `${firstname}.${lastname}${rndInt(1, 9999)}@${domain}`.toLowerCase();
}

function rndUser() {
  const { firstname, lastname } = rndFullName();
  return {
    email: rndEmail(firstname, lastname),
    firstname,
    lastname,
    dob: rndDob(),
    sex: rndSex(),
    tags: rndTags(),
  };
}

// A coordinate within `radiusKm` of an origin point, so demo users cluster
// around a real city instead of landing at random points on the globe.
// Samples uniformly over the disc (sqrt of a random distance, not a random
// distance directly) and offsets by a random bearing.
function rndPosition() {
  const originLat = 13.7563;
  const originLng = 100.5018;
  const radiusKm = 50;
  const earthRadiusKm = 6371;

  const distanceKm = radiusKm * Math.sqrt(Math.random());
  const bearing = Math.random() * 2 * Math.PI;
  const latRad = (originLat * Math.PI) / 180;

  const deltaLat = (distanceKm / earthRadiusKm) * Math.cos(bearing);
  const deltaLng = (distanceKm / (earthRadiusKm * Math.cos(latRad))) * Math.sin(bearing);

  const lat = originLat + (deltaLat * 180) / Math.PI;
  const lng = originLng + (deltaLng * 180) / Math.PI;

  return { x: lng, y: lat };
}

const PRODUCT_ADJECTIVES = [
  'Ergonomic', 'Rustic', 'Sleek', 'Handcrafted', 'Durable', 'Compact',
  'Premium', 'Lightweight', 'Vintage', 'Modern', 'Portable', 'Refined',
];

const PRODUCT_NOUNS = [
  'Chair', 'Lamp', 'Backpack', 'Keyboard', 'Mug', 'Jacket',
  'Shelf', 'Speaker', 'Wallet', 'Notebook', 'Bottle', 'Sofa',
];

function rndItem() {
  const adjective = rndPick(PRODUCT_ADJECTIVES);
  const noun = rndPick(PRODUCT_NOUNS);
  return {
    title: `${adjective} ${noun}`,
    description: `A ${adjective.toLowerCase()} ${noun.toLowerCase()} built to last.`,
  };
}

function rndSerialNumber(existing) {
  let candidate = rndInt(100000, 999999);
  while (existing.has(candidate)) {
    candidate = rndInt(100000, 999999);
  }
  existing.add(candidate);
  return candidate;
}
```

### 3. Define every schema and model

Seven collections make up the shop. Each one pairs a plain schema object with a `Model`
instance. None of these are initialized yet; that happens all together in
`initModels()`, the next step.

**User** -- the account itself. `email` is unique and lower-cased on write.

```typescript
const userSchema = {
  email: { type: Types.STRING, required: true, unique: true, lowercase: true },
  firstname: { type: Types.STRING, required: true },
  lastname: { type: Types.STRING, required: true },
  dob: { type: Types.TIME, required: true },
  sex: { type: Types.STRING, default: 'M' },
  created: { type: Types.TIME, required: true, default: () => new Date().toISOString() },
  active: { type: Types.BOOLEAN, default: true },
  tags: { type: Types.ARRAY, items: Types.STRING, default: () => [] },
};

const userModel = new Model({ database: DATABASE, collection: 'user', name: 'User', schema: userSchema });
```

**UserLocation** -- one row per user, using `Types.POSITION` (an `{ x, y }` pair) so the
user's location can be queried spatially, and looked up alongside their orders (see
`orderToPay` / `orderToShip` below).

```typescript
const userLocationSchema = {
  user: { type: Types.REFERENCE, reference: 'user', required: true, triggerRemove: true },
  location: { type: Types.POSITION, required: true },
};

const userLocationModel = new Model({ database: DATABASE, collection: 'userLocation', name: 'UserLocation', schema: userLocationSchema });
```

**Item** -- the catalog entry: a title and description, with no price or stock of its own.

```typescript
const itemSchema = {
  title: { type: Types.STRING, required: true },
  description: { type: Types.STRING, required: true },
  created: { type: Types.TIME, required: true, default: () => new Date().toISOString() },
};

const itemModel = new Model({ database: DATABASE, collection: 'item', name: 'Item', schema: itemSchema });
```

**Price** -- append-only: every price change inserts a new document rather than updating
an existing one, which is what lets the `currentPrice` view reconstruct "the latest price"
by sorting on `created`. The read helper for that view lives right here, next to the model:

```typescript
const priceSchema = {
  item: { type: Types.REFERENCE, reference: 'item', required: true, triggerRemove: true },
  amount: { type: Types.NUMBER, required: true, precision: 2 },
  currency: { type: Types.STRING, required: true, default: 'USD' },
  created: { type: Types.TIME, required: true, default: () => new Date().toISOString() },
};

const priceModel = new Model({ database: DATABASE, collection: 'price', name: 'Price', schema: priceSchema });

async function getCurrentPrice(itemId) {
  const [row] = await priceModel.findByView('currentPrice', { item: { $eq: itemId } });
  return row?.price;
}
```

**StoreItem** -- one document per physical unit in stock, each with its own
`serialNumber`. `reserved` starts `false` and flips to `true` the moment an order is placed
for it, so it stops counting as available stock in the `inventory` view. `reservedFor`
records which user holds that reservation. `default` here is given as a factory function
(`() => false`) rather than a bare value -- both forms are valid, the same way `tags` on
`User` uses `() => []`.

```typescript
const storeItemSchema = {
  item: { type: Types.REFERENCE, reference: 'item', required: true, triggerRemove: true },
  serialNumber: { type: Types.NUMBER, required: true, precision: 0 },
  reserved: { type: Types.BOOLEAN, default: () => false },
  reservedFor: { type: Types.REFERENCE, reference: 'user' },
  created: { type: Types.TIME, required: true, default: () => new Date().toISOString() },
};

const storeItemModel = new Model({ database: DATABASE, collection: 'storeItem', name: 'StoreItem', schema: storeItemSchema });

async function getAvailableCount(itemId) {
  const [row] = await storeItemModel.findByView('inventory', { item: { $eq: itemId } });
  return row?.count ?? 0;
}
```

**Transaction** -- every movement of money (deposits, withdrawals, order payments) is one
document. A positive `amount` is money coming in, negative is money going out; `status`
starts `'pending'` and moves to `'succeeded'` once settled.

```typescript
const transactionSchema = {
  user: { type: Types.REFERENCE, reference: 'user', required: true, triggerRemove: true },
  amount: { type: Types.NUMBER, required: true, precision: 2 },
  currency: { type: Types.STRING, default: 'USD' },
  created: { type: Types.TIME, default: () => new Date().toISOString() },
  status: { type: Types.STRING, required: true, default: 'pending' },
};

const transactionModel = new Model({ database: DATABASE, collection: 'transaction', name: 'Transaction', schema: transactionSchema });

async function getUserBalance(userId) {
  const [deposits] = await transactionModel.findByView('balancePendingDeposits', { user: { $eq: userId } });
  const [withdrawals] = await transactionModel.findByView('balancePendingWithdrawals', { user: { $eq: userId } });
  const [actual] = await transactionModel.findByView('balanceActual', { user: { $eq: userId } });
  return {
    user: userId,
    amount: {
      pending_deposits: deposits?.pending_deposits ?? 0,
      pending_withdrawals: withdrawals?.pending_withdrawals ?? 0,
      actual: actual?.actual ?? 0,
    },
  };
}
```

`getUserBalance` reads from three views (`balancePendingDeposits`,
`balancePendingWithdrawals`, `balanceActual`) that don't exist yet at this point in the
file -- that's fine, since this is a function definition, not a call. They're created in
`initModels()`, and only need to exist by the time this function is actually invoked.

**Order** -- references the specific `StoreItem` units purchased (not the `Item` catalog
entries), so each order is tied to the exact physical units that left the shelf. `status`
tracks where the order sits in its lifecycle: `'toPay'` (items are reserved but not yet
paid for), `'toShip'` (paid, waiting to go out), or `'toReview'` (delivered, waiting on the
buyer) -- this example only ever inserts an order as `'toPay'` or `'toShip'`.
`amount`/`currency` capture the order's total at creation time (the sum of its items'
current prices), so the price the buyer agreed to doesn't drift if prices change later.

```typescript
const orderSchema = {
  items: { type: Types.ARRAY, items: Types.REFERENCE, reference: 'storeItem', required: true },
  user: { type: Types.REFERENCE, reference: 'user', required: true, triggerRemove: true },
  status: { type: Types.STRING, required: true, default: 'toPay' },
  amount: { type: Types.NUMBER, required: true, precision: 2 },
  currency: { type: Types.STRING, required: true, default: 'USD' },
  created: { type: Types.TIME, required: true, default: () => new Date().toISOString() },
};

const orderModel = new Model({ database: DATABASE, collection: 'order', name: 'Order', schema: orderSchema });
```

### 4. Initialize every model and create its views

`initModels()` is where everything declared above actually gets registered with the
server: it opens the connection, calls `.init()` on all seven models, and then creates
every materialized view. It's called once, at the very start of `main()`, before
dispatching to either the `init` or `flow` mode below.

```typescript
async function initModels() {
  await dianaDb.connect(1000);
  await userModel.init();
  await userLocationModel.init();
  await itemModel.init();
  await priceModel.init();
  await storeItemModel.init();
  await transactionModel.init();
  await orderModel.init();

  await priceModel.createView('currentPrice', [
    { $sort: { created: 1 } },
    { $group: { _id: '$item', item: { $first: '$item' }, price: { $last: '$amount' } } },
  ]);

  // reserved: { $ne: true } matches both reserved: false and a
  // missing/undefined reserved field.
  await storeItemModel.createView('inventory', [
    { $match: { reserved: { $ne: true } } },
    { $group: { _id: '$item', item: { $first: '$item' }, count: { $sum: [1] } } },
  ]);

  await transactionModel.createView('balancePendingDeposits', [
    { $match: { status: { $eq: 'pending' }, amount: { $gt: 0 } } },
    { $group: { _id: '$user', user: { $first: '$user' }, pending_deposits: { $sum: ['$amount'] } } },
  ]);

  await transactionModel.createView('balancePendingWithdrawals', [
    { $match: { status: { $eq: 'pending' }, amount: { $lt: 0 } } },
    { $group: { _id: '$user', user: { $first: '$user' }, pending_withdrawals: { $sum: ['$amount'] } } },
  ]);

  await transactionModel.createView('balanceActual', [
    { $match: { status: { $eq: 'succeeded' } } },
    { $group: { _id: '$user', user: { $first: '$user' }, actual: { $sum: ['$amount'] } } },
  ]);

  await orderModel.createView('orderToPay', [
    { $match: { status: { $eq: 'toPay' } } },
    // order.user and userLocation.user both hold the same user _id, so this
    // join doesn't need the user lookup to run first
    { $lookup: { database: DATABASE, collection: 'userLocation', localField: 'user', foreignField: 'user', as: 'location' } },
    { $unwind: '$location' },
    { $lookup: { database: DATABASE, collection: 'user', localField: 'user', foreignField: '_id', as: 'user' } },
    { $unwind: '$user' },
    { $project: { _id: false, location: '$location', user: true, items: true, amount: true, currency: true } },
  ]);

  await orderModel.createView('orderToShip', [
    { $match: { status: { $eq: 'toShip' } } },
    // NOTE: this must look up the `userLocation` collection, same as
    // orderToPay above -- there is no `location` collection.
    { $lookup: { database: DATABASE, collection: 'userLocation', localField: 'user', foreignField: 'user', as: 'location' } },
    { $unwind: '$location' },
    { $lookup: { database: DATABASE, collection: 'user', localField: 'user', foreignField: '_id', as: 'user' } },
    { $unwind: '$user' },
    { $project: { _id: false, location: '$location', user: true, items: true, amount: true, currency: true } },
  ]);
}
```

`orderToPay` and `orderToShip` alias the `userLocation` lookup as `location` and project it
through as-is, so the `location` field on a row from either view is the whole
`UserLocation` document (its own `_id`, its `user` reference, and the actual coordinate) --
not just the bare `{ x, y }` point. The point itself is one level deeper, at
`location.location`.

#### Querying orderToShip by location

Because the coordinate lives at `location.location` on an `orderToShip` row, querying it
spatially means matching on that nested path rather than on `location` directly.
`$nearLines` matches documents within a given `distance` of one or more line segments --
useful for something like "which orders waiting to ship sit along today's delivery route":

```typescript
const shipmentsAlongRoute = await orderModel.findByView('orderToShip', {
  'location.location': {
    $nearLines: {
      lines: [
        [{ x: 1, y: 1 }, { x: 10, y: 10 }],
        [{ x: 2, y: 0 }, { x: 4, y: 20 }],
      ],
      distance: 10,
    },
  },
});
```

Each entry in `lines` is one segment as a pair of `{ x, y }` points; `distance` is the
maximum allowed distance from any of them, in the same units as the coordinates. See the
Find and Count Documents section above for the rest of the `POSITION` query operands
(`$insideCircle`, `$insidePolygon`, `$nearLines`, and their inverses).

### 5. Create users and their locations

```typescript
const USER_COUNT = 10;

async function seedUsers() {
  const users = [];
  for (let i = 0; i < USER_COUNT; i++) {
    const user = await userModel.insert(rndUser());
    users.push(user);
    await userLocationModel.insert({ user: user._id, location: rndPosition() });
  }
  return users;
}
```

### 6. Create items and their first prices

```typescript
const ITEM_COUNT = 10;

async function seedItems() {
  const items = [];
  for (let i = 0; i < ITEM_COUNT; i++) {
    const item = await itemModel.insert(rndItem());
    items.push(item);
    await priceModel.insert({ item: item._id, amount: rndAmount(10, 100), currency: 'USD' });
  }
  return items;
}
```

### 7. Add store items (stock)

Give each item 1-5 physical units, each with its own unique `serialNumber`. `reserved` is
passed explicitly as `false` here (its schema default would apply either way):

```typescript
async function seedStoreItems(items) {
  for (const item of items) {
    const usedSerials = new Set();
    const unitCount = rndInt(1, 5);
    for (let i = 0; i < unitCount; i++) {
      await storeItemModel.insert({ item: item._id, serialNumber: rndSerialNumber(usedSerials), reserved: false });
    }
  }
}
```

### 8. Add money to users (deposits)

A deposit is just a `Transaction` with a positive `amount` and `status: 'succeeded'`. This
is what the `balanceActual` view sums up per user to produce a spendable balance:

```typescript
async function seedDeposits(users) {
  for (const user of users) {
    await transactionModel.insert({ user: user._id, amount: rndAmount(10, 100), currency: 'USD', status: 'succeeded' });
  }
}

async function runInit() {
  const users = await seedUsers();
  const items = await seedItems();
  await seedStoreItems(items);
  await seedDeposits(users);
}
```

A deposit that should wait for a bank confirmation would instead be inserted with
`status: 'pending'` -- it would count toward `balance.amount.pending_deposits` but not
`balance.amount.actual` until it's updated to `'succeeded'`.

### 9. Create orders

The following runs once per user (for example inside a `processOrder(user, items)` step
called in a loop). Creating an order checks two things, not just one: does the user have
enough money, and is there actually stock left to sell. Start with availability -- for each
candidate item, ask the `inventory` view whether any unreserved units exist, then pull the
actual list of available `StoreItem` documents for the ones that do:

```typescript
async function pickAvailableStoreItems(candidateItems) {
  const chosen = [];

  for (const item of candidateItems) {
    // Availability check #1: does the inventory view show any unreserved stock at all?
    const [inventory] = await storeItemModel.findByView('inventory', { item: { $eq: item._id } });
    if (!inventory || inventory.count < 1) continue;

    // Pull the actual list of unreserved units for this item -- the view only
    // gives a count, not which units are free
    const [storeItem] = await storeItemModel.find(
      [{ item: { $eq: item._id }, reserved: { $ne: true } }],
      undefined, undefined, 0, 1,
    );
    if (storeItem) chosen.push(storeItem);
  }

  return chosen;
}

async function orderTotal(storeItems) {
  let total = 0;
  for (const storeItem of storeItems) {
    const price = await getCurrentPrice(storeItem.item);
    total += price ?? 0;
  }
  return Math.round(total * 100) / 100;
}

const candidateItems = rndSample(items, rndInt(1, 5));
const chosen = await pickAvailableStoreItems(candidateItems);
if (chosen.length === 0) return; // nothing available right now, try the next user

const total = await orderTotal(chosen);
```

Then open a transaction scoped to the shared database and reserve the chosen units before
the order references them: the update's filter requires `reserved: { $ne: true }`, so a
unit another order grabbed a moment earlier won't match. A follow-up read confirms every
unit really did flip to `reserved: true` for this user -- and this is the only condition
that rolls the transaction back. Money is handled differently: if the balance covers the
total, a debiting `Transaction` is inserted and the order goes out as `'toShip'`, carrying
its `amount`/`currency`; if it doesn't, the order is still created -- as `'toPay'`, with the
same `amount`/`currency` and its items still reserved -- and simply waits to be paid later:

```typescript
const transactionId = await dianaDb.startTransaction({ database: DATABASE, autoRollBackAfterMs: 60000 });
const txnParams = { database: DATABASE, transactionId };

try {
  // Reserve first. The reserved: { $ne: true } filter means this only
  // matches units nobody else has claimed since we read them above.
  await storeItemModel.update(
    [{ _id: { $in: chosen.map((si) => si._id) }, reserved: { $ne: true } }],
    { reserved: true, reservedFor: user._id },
    transactionId,
  );

  // Availability check #2: confirm the reservation actually stuck for every
  // chosen unit. If a concurrent order beat us to one of them, it will still
  // show reserved: false (or reservedFor someone else) and be missing here.
  // This is the only condition that rolls the transaction back.
  const confirmedReserved = await storeItemModel.find(
    [{ _id: { $in: chosen.map((si) => si._id) }, reserved: { $eq: true }, reservedFor: { $eq: user._id } }],
    undefined, undefined, 0, chosen.length,
    transactionId,
  );

  if (confirmedReserved.length !== chosen.length) {
    await dianaDb.rollbackTransaction(txnParams);
    return;
  }

  const balance = await getUserBalance(user._id);
  const orderItems = confirmedReserved.map((si) => si._id);

  if (balance.amount.actual >= total) {
    // Enough money: charge now, ship now. The items stay reserved -- they're
    // now owned by this order rather than up for grabs again.
    await transactionModel.insert(
      { user: user._id, amount: -total, currency: 'USD', status: 'succeeded' },
      transactionId,
    );
    await orderModel.insert(
      { items: orderItems, user: user._id, status: 'toShip', amount: total, currency: 'USD' },
      transactionId,
    );
  } else {
    // Not enough money yet: no transaction is created, but the order still
    // exists -- toPay -- holding the reservation until it's settled.
    await orderModel.insert(
      { items: orderItems, user: user._id, status: 'toPay', amount: total, currency: 'USD' },
      transactionId,
    );
  }

  await dianaDb.commitTransaction(txnParams);
} catch (err) {
  await dianaDb.rollbackTransaction(txnParams);
}
```

Reserving before inserting the `Order` means the reservation is the thing that's actually
contested between concurrent orders. Insufficient inventory is the only thing that rolls
this transaction back -- an insufficient balance still produces an order, just as `'toPay'`
instead of `'toShip'`, with its `StoreItem` units held in reserve rather than released. A
separate "settle payment" step (not shown here) would insert the debiting `Transaction`
later and update the order's `status` to `'toShip'`.
`autoRollBackAfterMs: 60000` is a safety net that rolls the transaction back automatically
if it's never explicitly committed or rolled back within 60 seconds.

---

## Changelog

### 2026-01-17 — Server v1.5.0 / ODM v1.5.0 (Stable)

**Features:**
- Added `max` and `min` model methods — return documents with maximum or minimum value for a specified property
- Added `$max`, `$min`, `$avg` operators to `$group` and `$project` transform pipelines

**Fixes:**
- Fixed the data restoration mechanism

### 2026-01-16 — Server v1.4.9 / ODM v1.4.11

**Refactoring:**
- Refactored FilterQuery validation and processing

### 2026-01-13 — Server v1.4.8 / ODM v1.4.10

**Features:**
- Added `distinct` model method
- Improved encryption implementation

**Refactoring:**
- Refactored connection management

---

## Upcoming Features

- Multi-Server Replication — real-time sync and high availability across distributed nodes
- Advanced Access Management — granular user roles and permission controls
- TLS Support — encrypted client-server communication
- RPM, ARM + Docker Support — official packages for additional platforms
- Desktop Management Tool — cross-platform GUI for browsing, querying, and managing data
- Golang, PHP & Java Clients — official language clients

---

## License

MIT License with Usage Restriction. Copyright © 2025 Data Bikers Limited.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files to deal in the Software without restriction, subject to the following conditions:

- The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
- **The Software may not be used to create or offer a competing product or service without prior written consent from Data Bikers Limited.**

---

## Links

- Website: https://diana-db.com
- npm: https://www.npmjs.com/package/@diana-db/odm
- GitHub: https://github.com/databikers/diana-db
- Issues: https://github.com/databikers/diana-db/issues
- Data Bikers Limited: https://databikers.com