jormun-db/README.md

<p align="center">
  <img src="https://artifacts.ewr1.vultrobjects.com/jormundb.png" alt="JormunDB logo" width="220" />
</p>

<h1 align="center">JormunDB</h1>

<p align="center">
  A high-performance, DynamoDB-compatible database server written in Odin, backed by RocksDB.
  <br />
  <strong>DynamoDB-Compatible Database</strong> · Powered by <strong>RocksDB</strong> + <strong>Odin</strong>
</p>

---

## What is JormunDB?

JormunDB is a Self-Hosted DynamoDB replacement that speaks the DynamoDB wire protocol. Point your AWS SDK or CLI at it and use it as a drop-in replacement.

**Why Odin?** The original Zig implementation suffered from explicit allocator threading. Where every function ended up needing an `allocator` parameter and every allocation needed `errdefer` cleanup. Odin's implicit context allocator system eliminates this ceremony. Just one `context.allocator = arena_allocator` at the request handler entry and it feels more like working with ctx in Go instead of filling out tax forms.

## Features

- ✅ **DynamoDB Wire Protocol**: Works with AWS SDKs and CLI out of the box
- ✅ **Binary Storage**: Efficient TLV encoding for items, varint-prefixed keys
- ✅ **Arena-per-Request**: Zero explicit memory management in business logic
- ✅ **Table Operations**: CreateTable, DeleteTable, DescribeTable, ListTables
- ✅ **Item Operations**: PutItem, GetItem, DeleteItem
- ✅ **Query & Scan**: With pagination support (Limit, ExclusiveStartKey)
- ✅ **Expression Parsing**: KeyConditionExpression for Query operations
- ✅ **Persistent Storage**: RocksDB-backed with full ACID guarantees
- ✅ **Concurrency**: Table-level RW locks for safe concurrent access

## Quick Start

### Prerequisites

- Odin compiler (latest)
- RocksDB development libraries
- Standard compression libraries (snappy, lz4, zstd, etc.)

#### macOS (Homebrew)

```bash
brew install rocksdb odin
```

#### Ubuntu/Debian

```bash
sudo apt install librocksdb-dev libsnappy-dev liblz4-dev libzstd-dev libbz2-dev
# Install Odin from https://odin-lang.org/docs/install/
```

### Build & Run

```bash
# Build the server
make build

# Run with default settings (localhost:8002, ./data directory)
make run

# Run with custom port
make run PORT=9000

# Run with custom data directory
make run DATA_DIR=/tmp/jormundb
```

### Test with AWS CLI

```bash
# Create a table
aws dynamodb create-table \
    --endpoint-url http://localhost:8002 \
    --table-name Users \
    --key-schema AttributeName=id,KeyType=HASH \
    --attribute-definitions AttributeName=id,AttributeType=S \
    --billing-mode PAY_PER_REQUEST

# Put an item
aws dynamodb put-item \
    --endpoint-url http://localhost:8002 \
    --table-name Users \
    --item '{"id":{"S":"user123"},"name":{"S":"Alice"},"age":{"N":"30"}}'

# Get an item
aws dynamodb get-item \
    --endpoint-url http://localhost:8002 \
    --table-name Users \
    --key '{"id":{"S":"user123"}}'

# Query items
aws dynamodb query \
    --endpoint-url http://localhost:8002 \
    --table-name Users \
    --key-condition-expression "id = :id" \
    --expression-attribute-values '{":id":{"S":"user123"}}'

# Scan table
aws dynamodb scan \
    --endpoint-url http://localhost:8002 \
    --table-name Users
```

## Architecture

```
HTTP Request (POST /)
  ↓
X-Amz-Target header → Operation routing
  ↓
JSON body → DynamoDB types
  ↓
Storage engine → RocksDB operations
  ↓
Binary encoding → Disk
  ↓
JSON response → Client
```

### Storage Format

**Keys** (varint-length-prefixed segments):
```
Meta:  [0x01][len][table_name]
Data:  [0x02][len][table_name][len][pk_value][len][sk_value]?
GSI:   [0x03][len][table_name][len][index_name][len][gsi_pk][len][gsi_sk]?
LSI:   [0x04][len][table_name][len][index_name][len][pk][len][lsi_sk]
```

**Values** (TLV binary encoding):
```
[attr_count:varint]
  [name_len:varint][name:bytes][type_tag:u8][value_encoded:bytes]...

Type tags:
  String=0x01, Number=0x02, Binary=0x03, Bool=0x04, Null=0x05
  SS=0x10, NS=0x11, BS=0x12
  List=0x20, Map=0x21
```

## Memory Management

JormunDB uses Odin's context allocator system for elegant memory management:

```odin
// Request handler entry point
handle_request :: proc(conn: net.TCP_Socket) {
    arena: mem.Arena
    mem.arena_init(&arena, make([]byte, mem.Megabyte * 4))
    defer mem.arena_destroy(&arena)

    context.allocator = mem.arena_allocator(&arena)

    // Everything below uses the arena automatically
    // No manual frees, no errdefer cleanup needed
    request := parse_request()  // Uses context.allocator
    response := process(request) // Uses context.allocator
    send_response(response)      // Uses context.allocator

    // Arena is freed here automatically
}
```

Long-lived data (table metadata, locks) uses the default allocator. Request-scoped data uses the arena.

## Development

```bash
# Build debug version
make build

# Build optimized release
make release

# Run tests
make test

# Format code
make fmt

# Clean build artifacts
make clean

# Run with custom settings
make run PORT=9000 DATA_DIR=/tmp/db VERBOSE=1
```

## Performance

Benchmarked on single node localhost, 1000 iterations per test.

### Basic Operations

| Operation | Throughput | Avg Latency | P95 Latency | P99 Latency |
|-----------|------------|-------------|-------------|-------------|
| **PutItem** | 1,021 ops/sec | 0.98ms | 1.02ms | 1.64ms |
| **GetItem** | 1,207 ops/sec | 0.83ms | 0.90ms | 1.14ms |
| **Query** | 1,002 ops/sec | 1.00ms | 1.11ms | 1.85ms |
| **Scan** (100 items) | 18,804 ops/sec | 0.05ms | - | - |
| **DeleteItem** | 1,254 ops/sec | 0.80ms | - | - |

### Batch Operations

| Operation | Throughput | Batch Size |
|-----------|------------|------------|
| **BatchWriteItem** | 9,297 ops/sec | 25 items |
| **BatchGetItem** | 9,113 ops/sec | 25 items |

### Concurrent Operations

| Workers | Throughput | Avg Latency | P95 Latency | P99 Latency |
|---------|------------|-------------|-------------|-------------|
| **10 concurrent** | 1,286 ops/sec | 7.70ms | 15.16ms | 19.72ms |

### Large Payloads

| Payload Size | Throughput | Avg Latency |
|--------------|------------|-------------|
| **10KB** | 522 ops/sec | 1.91ms |
| **50KB** | 166 ops/sec | 6.01ms |
| **100KB** | 96 ops/sec | 10.33ms |

## API Compatibility

### Supported Operations

- ✅ CreateTable
- ✅ DeleteTable
- ✅ DescribeTable
- ✅ ListTables
- ✅ PutItem
- ✅ GetItem
- ✅ DeleteItem
- ✅ Query (with KeyConditionExpression)
- ✅ Scan (with pagination)
- ✅ ConditionExpression
- ✅ FilterExpression
- ✅ ProjectionExpression
- ✅ BatchWriteItem
- ✅ BatchGetItem
- ✅ Global Secondary Indexes

### Coming Soon

- ⏳ UpdateItem (works but needs UPDATED_NEW/UPDATED_OLD response filtering to work for full Dynamo Parity)
- ⏳ Local Secondary Indexes

## Configuration

### Environment Variables

```bash
JORMUN_PORT=8002           # Server port (I have something locally on port 8000 so now everyone has to use port 8002)
JORMUN_HOST=0.0.0.0        # Bind address
JORMUN_DATA_DIR=./data     # RocksDB data directory
JORMUN_VERBOSE=1           # Enable verbose logging
```

### Command Line Arguments

```bash
./jormundb --port 9000 --host 127.0.0.1 --data-dir /var/db --verbose
```

## Troubleshooting

### "Cannot open RocksDB"

Ensure RocksDB libraries are installed and the data directory is writable:

```bash
# Check RocksDB installation
pkg-config --libs rocksdb

# Check permissions
mkdir -p ./data
chmod 755 ./data
```

### "Connection refused"

Check if the port is already in use:

```bash
lsof -i :8002
```

### "Invalid JSON" errors

Ensure you're using the correct DynamoDB JSON format:

```json
{
  "TableName": "Users",
  "Item": {
    "id": {"S": "user123"},
    "age": {"N": "30"}
  }
}
```

## Credits

- Inspired by DynamoDB
- Built with [Odin](https://odin-lang.org/)
- Powered by [RocksDB](https://rocksdb.org/)
- Originally implemented as ZynamoDB in Zig

## Contributing

Contributions welcome! Please:

1. Format code with `make fmt`
2. Run tests with `make test`
3. Update documentation as needed
4. Follow Odin idioms (context allocators, explicit returns, etc.)

---

**Why "Jormun"?** Jörmungandr, the World Serpent from Norse mythology, which I found fitting for something built in a language called Odin. Also, it sounds cool.