jormun-db/README.md

<p align="center">
  <img src="https://artifacts.ewr1.vultrobjects.com/jormundb.png" alt="JormunDB logo" width="220" />
</p>

<h1 align="center">JormunDB</h1>

<p align="center">
  A high-performance, DynamoDB-compatible database server written in Odin, backed by RocksDB.
  <br />
  <strong>DynamoDB-Compatible Database</strong> · Powered by <strong>RocksDB</strong> + <strong>Odin</strong>
</p>

---

## What is JormunDB?

JormunDB (formerly ZynamoDB) is a Self-Hosted DynamoDB replacement that speaks the DynamoDB wire protocol. Point your AWS SDK or CLI at it and use it as a drop-in replacement.

**Why Odin?** The original Zig implementation suffered from explicit allocator threading—every function taking an `allocator` parameter, every allocation needing `errdefer` cleanup. Odin's implicit context allocator system eliminates this ceremony: one `context.allocator = arena_allocator` at the request handler entry and everything downstream just works.

## Features

- ✅ **DynamoDB Wire Protocol**: Works with AWS SDKs and CLI out of the box
- ✅ **Binary Storage**: Efficient TLV encoding for items, varint-prefixed keys
- ✅ **Arena-per-Request**: Zero explicit memory management in business logic
- ✅ **Table Operations**: CreateTable, DeleteTable, DescribeTable, ListTables
- ✅ **Item Operations**: PutItem, GetItem, DeleteItem
- ✅ **Query & Scan**: With pagination support (Limit, ExclusiveStartKey)
- ✅ **Expression Parsing**: KeyConditionExpression for Query operations
- ✅ **Persistent Storage**: RocksDB-backed with full ACID guarantees
- ✅ **Concurrency**: Table-level RW locks for safe concurrent access

## Quick Start

### Prerequisites

- Odin compiler (latest)
- RocksDB development libraries
- Standard compression libraries (snappy, lz4, zstd, etc.)

#### macOS (Homebrew)

```bash
brew install rocksdb odin
```

#### Ubuntu/Debian

```bash
sudo apt install librocksdb-dev libsnappy-dev liblz4-dev libzstd-dev libbz2-dev
# Install Odin from https://odin-lang.org/docs/install/
```

### Build & Run

```bash
# Build the server
make build

# Run with default settings (localhost:8002, ./data directory)
make run

# Run with custom port
make run PORT=9000

# Run with custom data directory
make run DATA_DIR=/tmp/jormundb
```

### Test with AWS CLI

```bash
# Create a table
aws dynamodb create-table \
    --endpoint-url http://localhost:8002 \
    --table-name Users \
    --key-schema AttributeName=id,KeyType=HASH \
    --attribute-definitions AttributeName=id,AttributeType=S \
    --billing-mode PAY_PER_REQUEST

# Put an item
aws dynamodb put-item \
    --endpoint-url http://localhost:8002 \
    --table-name Users \
    --item '{"id":{"S":"user123"},"name":{"S":"Alice"},"age":{"N":"30"}}'

# Get an item
aws dynamodb get-item \
    --endpoint-url http://localhost:8002 \
    --table-name Users \
    --key '{"id":{"S":"user123"}}'

# Query items
aws dynamodb query \
    --endpoint-url http://localhost:8002 \
    --table-name Users \
    --key-condition-expression "id = :id" \
    --expression-attribute-values '{":id":{"S":"user123"}}'

# Scan table
aws dynamodb scan \
    --endpoint-url http://localhost:8002 \
    --table-name Users
```

## Architecture

```
HTTP Request (POST /)
  ↓
X-Amz-Target header → Operation routing
  ↓
JSON body → DynamoDB types
  ↓
Storage engine → RocksDB operations
  ↓
Binary encoding → Disk
  ↓
JSON response → Client
```

### Module Structure

```
jormundb/
├── rocksdb/            - C FFI bindings to librocksdb
├── rocksdb_shim/       - C++ FFI bindings to librocksdb (so we can use the WAL helper functions)
├── dynamodb/           - Core types and operations
│   ├── types.odin      - AttributeValue, Item, Key, etc.
│   ├── json.odin       - DynamoDB JSON serialization
│   ├── storage.odin    - Storage engine with RocksDB
│   ├── key_codec.odin  - Binary key encoding
│   ├── item_codec.odin - Binary TLV item encoding
│   └── handler.odin    - HTTP request handlers
├── http.odin           - HTTP Server
└── main.odin           - HTTP Router and handler (entry point)
```

### Storage Format

**Keys** (varint-length-prefixed segments):
```
Meta:  [0x01][len][table_name]
Data:  [0x02][len][table_name][len][pk_value][len][sk_value]?
GSI:   [0x03][len][table_name][len][index_name][len][gsi_pk][len][gsi_sk]?
LSI:   [0x04][len][table_name][len][index_name][len][pk][len][lsi_sk]
```

**Values** (TLV binary encoding):
```
[attr_count:varint]
  [name_len:varint][name:bytes][type_tag:u8][value_encoded:bytes]...

Type tags:
  String=0x01, Number=0x02, Binary=0x03, Bool=0x04, Null=0x05
  SS=0x10, NS=0x11, BS=0x12
  List=0x20, Map=0x21
```

## Memory Management

JormunDB uses Odin's context allocator system for elegant memory management:

```odin
// Request handler entry point
handle_request :: proc(conn: net.TCP_Socket) {
    arena: mem.Arena
    mem.arena_init(&arena, make([]byte, mem.Megabyte * 4))
    defer mem.arena_destroy(&arena)

    context.allocator = mem.arena_allocator(&arena)

    // Everything below uses the arena automatically
    // No manual frees, no errdefer cleanup needed
    request := parse_request()  // Uses context.allocator
    response := process(request) // Uses context.allocator
    send_response(response)      // Uses context.allocator

    // Arena is freed here automatically
}
```

Long-lived data (table metadata, locks) uses the default allocator. Request-scoped data uses the arena.

## Development

```bash
# Build debug version
make build

# Build optimized release
make release

# Run tests
make test

# Format code
make fmt

# Clean build artifacts
make clean

# Run with custom settings
make run PORT=9000 DATA_DIR=/tmp/db VERBOSE=1
```

## Performance

From benchmarks on the original Zig version (Odin expected to be similar or better):

```
Sequential Writes     |      10000 ops |   245.32 ms |    40765 ops/sec
Random Reads          |      10000 ops |   312.45 ms |    32006 ops/sec
Batch Writes          |      10000 ops |    89.23 ms |   112071 ops/sec
PutItem               |       5000 ops |   892.34 ms |     5604 ops/sec
GetItem               |       5000 ops |   678.91 ms |     7365 ops/sec
Scan (full table)     |       5000 ops |   234.56 ms |    21320 ops/sec
```

## API Compatibility

### Supported Operations

- ✅ CreateTable
- ✅ DeleteTable
- ✅ DescribeTable
- ✅ ListTables
- ✅ PutItem
- ✅ GetItem
- ✅ DeleteItem
- ✅ Query (with KeyConditionExpression)
- ✅ Scan (with pagination)

### Coming Soon

- ⏳ UpdateItem (with UpdateExpression)
- ⏳ BatchWriteItem
- ⏳ BatchGetItem
- ⏳ Global Secondary Indexes
- ⏳ Local Secondary Indexes
- ⏳ ConditionExpression
- ⏳ FilterExpression
- ⏳ ProjectionExpression

## Configuration

### Environment Variables

```bash
JORMUN_PORT=8002           # Server port (I have something locally on port 8000 so now everyone has to use port 8002)
JORMUN_HOST=0.0.0.0        # Bind address
JORMUN_DATA_DIR=./data     # RocksDB data directory
JORMUN_VERBOSE=1           # Enable verbose logging
```

### Command Line Arguments

```bash
./jormundb --port 9000 --host 127.0.0.1 --data-dir /var/db --verbose
```

## Troubleshooting

### "Cannot open RocksDB"

Ensure RocksDB libraries are installed and the data directory is writable:

```bash
# Check RocksDB installation
pkg-config --libs rocksdb

# Check permissions
mkdir -p ./data
chmod 755 ./data
```

### "Connection refused"

Check if the port is already in use:

```bash
lsof -i :8002
```

### "Invalid JSON" errors

Ensure you're using the correct DynamoDB JSON format:

```json
{
  "TableName": "Users",
  "Item": {
    "id": {"S": "user123"},
    "age": {"N": "30"}
  }
}
```

## Credits

- Inspired by DynamoDB
- Built with [Odin](https://odin-lang.org/)
- Powered by [RocksDB](https://rocksdb.org/)
- Originally implemented as ZynamoDB in Zig

## Contributing

Contributions welcome! Please:

1. Format code with `make fmt`
2. Run tests with `make test`
3. Update documentation as needed
4. Follow Odin idioms (context allocators, explicit returns, etc.)

---

**Why "Jormun"?** Jörmungandr, the World Serpent from Norse mythology, which I found fitting for something built in a language called Odin. Also, it sounds cool.