jormun-db/README.md

<p align="center">
  <img src="https://artifacts.ewr1.vultrobjects.com/jormundb.png" alt="JormunDB logo" width="220" />
</p>

<h1 align="center">JormunDB</h1>

<p align="center">
  A high-performance, DynamoDB-compatible database server written in Odin, backed by RocksDB.
  <br />
  <strong>DynamoDB-Compatible Database</strong> · Powered by <strong>RocksDB</strong> + <strong>Odin</strong>
</p>

---

## What is JormunDB?

JormunDB is an obnoxiously fast, single-tenant DynamoDB replacement designed for teams who need something under their control.
It speaks the DynamoDB wire protocol, so any existing AWS SDK client works against it without modification.

## Features

- ✅ **DynamoDB Wire Protocol**: Works with AWS SDKs and CLI out of the box
- ✅ **Binary Storage**: Efficient TLV encoding for items, varint-prefixed keys
- ✅ **Arena-per-Request**: Zero explicit memory management in business logic
- ✅ **Table Operations**: CreateTable, DeleteTable, DescribeTable, ListTables
- ✅ **Item Operations**: PutItem, GetItem, DeleteItem
- ✅ **Query & Scan**: With pagination support (Limit, ExclusiveStartKey)
- ✅ **Expression Parsing**: KeyConditionExpression for Query operations
- ✅ **Persistent Storage**: RocksDB-backed with full ACID guarantees
- ✅ **Concurrency**: Table-level RW locks for safe concurrent access

## Quick Start

### Prerequisites

- Odin compiler (latest)
- RocksDB development libraries
- Standard compression libraries (snappy, lz4, zstd, etc.)

#### macOS (Homebrew)

```bash
brew install rocksdb odin
```

#### Ubuntu/Debian

```bash
sudo apt install librocksdb-dev libsnappy-dev liblz4-dev libzstd-dev libbz2-dev
# Install Odin from https://odin-lang.org/docs/install/
```

### Build & Run

```bash
# Build the server
make build

# Run with default settings (localhost:8002, ./data directory)
make run

# Run with custom port
make run PORT=9000

# Run with custom data directory
make run DATA_DIR=/tmp/jormundb
```

### Test with AWS CLI

```bash
# Create a table
aws dynamodb create-table \
    --endpoint-url http://localhost:8002 \
    --table-name Users \
    --key-schema AttributeName=id,KeyType=HASH \
    --attribute-definitions AttributeName=id,AttributeType=S \
    --billing-mode PAY_PER_REQUEST

# Put an item
aws dynamodb put-item \
    --endpoint-url http://localhost:8002 \
    --table-name Users \
    --item '{"id":{"S":"user123"},"name":{"S":"Alice"},"age":{"N":"30"}}'

# Get an item
aws dynamodb get-item \
    --endpoint-url http://localhost:8002 \
    --table-name Users \
    --key '{"id":{"S":"user123"}}'

# Query items
aws dynamodb query \
    --endpoint-url http://localhost:8002 \
    --table-name Users \
    --key-condition-expression "id = :id" \
    --expression-attribute-values '{":id":{"S":"user123"}}'

# Scan table
aws dynamodb scan \
    --endpoint-url http://localhost:8002 \
    --table-name Users
```

## Architecture

```
HTTP Request (POST /)
  ↓
X-Amz-Target header → Operation routing
  ↓
JSON body → DynamoDB types
  ↓
Storage engine → RocksDB operations
  ↓
Binary encoding → Disk
  ↓
JSON response → Client
```

### Storage Format

**Keys** (varint-length-prefixed segments):
```
Meta:  [0x01][len][table_name]
Data:  [0x02][len][table_name][len][pk_value][len][sk_value]?
GSI:   [0x03][len][table_name][len][index_name][len][gsi_pk][len][gsi_sk]?
LSI:   [0x04][len][table_name][len][index_name][len][pk][len][lsi_sk]
```

**Values** (TLV binary encoding):
```
[attr_count:varint]
  [name_len:varint][name:bytes][type_tag:u8][value_encoded:bytes]...

Type tags:
  String=0x01, Number=0x02, Binary=0x03, Bool=0x04, Null=0x05
  SS=0x10, NS=0x11, BS=0x12
  List=0x20, Map=0x21
```

## Memory Management

JormunDB uses Odin's context allocator system for elegant memory management:

```odin
// Request handler entry point
handle_request :: proc(conn: net.TCP_Socket) {
    arena: mem.Arena
    mem.arena_init(&arena, make([]byte, mem.Megabyte * 4))
    defer mem.arena_destroy(&arena)

    context.allocator = mem.arena_allocator(&arena)

    // Everything below uses the arena automatically
    request := parse_request()  // Uses context.allocator
    response := process(request) // Uses context.allocator
    send_response(response)      // Uses context.allocator

    // Arena is freed automatically with some exceptions
}
```

Long-lived data (table metadata, locks) uses the default allocator. Request-scoped data uses the arena.

## Development

```bash
# Build debug version
make build

# Build optimized release
make release

# Run tests
make test

# Format code
make fmt

# Clean build artifacts
make clean

# Run with custom settings
make run PORT=9000 DATA_DIR=/tmp/db
```

## Performance

Benchmarked on single node localhost, 1000 iterations per test.

### Basic Operations

| Operation | Throughput | Avg Latency | P95 Latency | P99 Latency |
|-----------|------------|-------------|-------------|-------------|
| **PutItem** | 1,021 ops/sec | 0.98ms | 1.02ms | 1.64ms |
| **GetItem** | 1,207 ops/sec | 0.83ms | 0.90ms | 1.14ms |
| **Query** | 1,002 ops/sec | 1.00ms | 1.11ms | 1.85ms |
| **Scan** (100 items) | 18,804 ops/sec | 0.05ms | - | - |
| **DeleteItem** | 1,254 ops/sec | 0.80ms | - | - |

### Batch Operations

| Operation | Throughput | Batch Size |
|-----------|------------|------------|
| **BatchWriteItem** | 9,297 ops/sec | 25 items |
| **BatchGetItem** | 9,113 ops/sec | 25 items |

### Concurrent Operations

| Workers | Throughput | Avg Latency | P95 Latency | P99 Latency |
|---------|------------|-------------|-------------|-------------|
| **10 concurrent** | 1,286 ops/sec | 7.70ms | 15.16ms | 19.72ms |

### Large Payloads

| Payload Size | Throughput | Avg Latency |
|--------------|------------|-------------|
| **10KB** | 522 ops/sec | 1.91ms |
| **50KB** | 166 ops/sec | 6.01ms |
| **100KB** | 96 ops/sec | 10.33ms |

## API Compatibility

### Supported Operations

- ✅ CreateTable
- ✅ DeleteTable
- ✅ DescribeTable
- ✅ ListTables
- ✅ PutItem
- ✅ GetItem
- ✅ DeleteItem
- ✅ Query (with KeyConditionExpression)
- ✅ Scan (with pagination)
- ✅ ConditionExpression
- ✅ FilterExpression
- ✅ ProjectionExpression
- ✅ BatchWriteItem
- ✅ BatchGetItem
- ✅ Global Secondary Indexes

### Coming Soon

- ⏳ UpdateItem (works but needs UPDATED_NEW/UPDATED_OLD response filtering to work for full Dynamo Parity)
- ⏳ Local Secondary Indexes
- ⏳ Read-Only replicas via WAL
- ⏳ A "Rebuild Index" Tool

## Configuration

### Environment Variables

```bash
JORMUN_PORT=8002           # Server port (I have something locally on port 8000 so now everyone has to use port 8002)
JORMUN_HOST=0.0.0.0        # Bind address
JORMUN_DATA_DIR=./data     # RocksDB data directory
```

### Command Line Arguments

```bash
./jormundb --port 9000 --host 127.0.0.1 --data-dir /var/db
```

## Design Philosophy

**One instance, one application.** JormunDB is intentionally single-tenant. Rather than running one massive shared database for all your applications, the model is to spin up a dedicated JormunDB instance per application. Instances are cheap. Coordination is not. You can run multiple tables per instance if you want but that's not what it's optimized for, and it's not a replacement for a full RDBMS. There's no query optimizer, no planner, none of the real crazy stuff that PostgreSQL or real DynamoDB has built up over years to solve specific bottlenecks. It's a fast, durable key-value store that speaks a protocol you already know.

**Local disk is fast.** RocksDB is exceptional at squeezing performance out of local storage. JormunDB is designed to run on bare metal with NVMe drives, or on VMs where the disk is local at the host level. Networked storage probably won't see much benefit from Jormun. The closer the data is to the CPU, the better.

**Minimal config, maximal durability and portability.** RocksDB handles durability. JormunDB handles the DynamoDB protocol. Compiles to a single binary.

**It's an excellent write-through cache.** The original use case was a write-through cache sitting in front of a production RDBMS. If you already have the AWS SDK in your stack for S3 or other services, you have a DynamoDB client sitting there doing nothing. Put it to work.

**HTTPS is optional and handled externally.** If you need TLS, put Caddy in front of it. JormunDB is designed to run inside a VPC with no public IP. The optional access key check is not a cryptographic auth mechanism.

---

## Why Not Just Use DynamoDB Local?

DynamoDB Local is a development tool and not built for production workloads. Under the hood it's a Java application backed by SQLite where every operation goes through two layers of translation: DynamoDB semantics get mapped into SQL, SQLite executes it as a relational query, and the result gets mapped back. Serializing a key-value workload into a relational engine and then back out again was what I was trying to avoid in the first place.

RocksDB is a native LSM-tree key-value store so it's architecturally closer to what DynamoDB actually is under the hood. JormunDB skips the translation entirely. The data path is the Request, Odin, RocksDB, and Disk.

---

## Why Odin?

This project went through a few languages before landing on Odin, and the journey is worth explaining.

**C++** was the obvious choice but the ecosystem overhead wasn't worth it for a focused project like this.

**Rust** came next, as is trendy. The problem I learned is that a database is an imperative, stateful thing that needs mutation and memory management. Rust's functional programming model fights at every turn, and binding RocksDB without sprinkling `unsafe` everywhere turned out to be essentially impossible.

**Zig** got further than anything else, and the original implementation was solid. The dealbreaker was allocator threading, every function that allocates needs an explicit `allocator` parameter, and every allocation needs `errdefer` cleanup. For a request-handling database, this meant every function in the call stack had an allocator bolted on. It works, but it's ceremonious in a way that compounds and became fatigue.

**Odin** solved the exact problem Zig introduced. Odin has an implicit context system where the allocator lives in a thread-local context object. Set `context.allocator = request_arena` once at the top of your request handler, and every allocation downstream automatically uses the arena. It feels closer to working with `ctx` in Go instead of filling out tax forms. The entire request lifetime is managed by a single growing arena that gets thrown away when the request completes.

That model is a natural fit for a request-handling server, and it's a big part of why JormunDB is as fast as it is.

## Note:

Complex auth systems tend to become an attack surface. Issues like leaked keys, token forgery, and privilege escalation are problems that come with the complexity of a robust auth implementation rather than the underlying data layer. I did not want to deal with multi-tenant credential stores, token refresh logic, and signature verification edge cases. I focused on this doing 1 job because I can leverage the strength in that. More robust auth will come in the future. For now, simple is safe.


## Credits

- Inspired by DynamoDB
- Built with [Odin](https://odin-lang.org/)
- Powered by [RocksDB](https://rocksdb.org/)


---

**Why "Jormun"?** Jörmungandr, the World Serpent from Norse mythology, which I found fitting for something built in a language called Odin. Also, it sounds cool. Think of deploying a "Jormun Cluster", just rolls off the tongue and sounds crazy.