first commit

2026-02-15 08:55:22 -05:00
commit 677bbb4028
21 changed files with 7320 additions and 0 deletions
--- a/README.md
+++ b/README.md
@@ -0,0 +1,317 @@
+# JormunDB
+
+A high-performance, DynamoDB-compatible database server written in Odin, backed by RocksDB.
+
+```
+     ╦╔═╗╦═╗╔╦╗╦ ╦╔╗╔╔╦╗╔╗ 
+     ║║ ║╠╦╝║║║║ ║║║║ ║║╠╩╗
+    ╚╝╚═╝╩╚═╩ ╩╚═╝╝╚╝═╩╝╚═╝
+    DynamoDB-Compatible Database
+    Powered by RocksDB + Odin
+```
+
+## What is JormunDB?
+
+JormunDB (formerly ZynamoDB) is a local DynamoDB replacement that speaks the DynamoDB wire protocol. Point your AWS SDK or CLI at it and use it as a drop-in development database.
+
+**Why Odin?** The original Zig implementation suffered from explicit allocator threading—every function taking an `allocator` parameter, every allocation needing `errdefer` cleanup. Odin's implicit context allocator system eliminates this ceremony: one `context.allocator = arena_allocator` at the request handler entry and everything downstream just works.
+
+## Features
+
+- ✅ **DynamoDB Wire Protocol**: Works with AWS SDKs and CLI out of the box
+- ✅ **Binary Storage**: Efficient TLV encoding for items, varint-prefixed keys
+- ✅ **Arena-per-Request**: Zero explicit memory management in business logic
+- ✅ **Table Operations**: CreateTable, DeleteTable, DescribeTable, ListTables
+- ✅ **Item Operations**: PutItem, GetItem, DeleteItem
+- ✅ **Query & Scan**: With pagination support (Limit, ExclusiveStartKey)
+- ✅ **Expression Parsing**: KeyConditionExpression for Query operations
+- ✅ **Persistent Storage**: RocksDB-backed with full ACID guarantees
+- ✅ **Concurrency**: Table-level RW locks for safe concurrent access
+
+## Quick Start
+
+### Prerequisites
+
+- Odin compiler (latest)
+- RocksDB development libraries
+- Standard compression libraries (snappy, lz4, zstd, etc.)
+
+#### macOS (Homebrew)
+
+```bash
+brew install rocksdb odin
+```
+
+#### Ubuntu/Debian
+
+```bash
+sudo apt install librocksdb-dev libsnappy-dev liblz4-dev libzstd-dev libbz2-dev
+# Install Odin from https://odin-lang.org/docs/install/
+```
+
+### Build & Run
+
+```bash
+# Build the server
+make build
+
+# Run with default settings (localhost:8000, ./data directory)
+make run
+
+# Run with custom port
+make run PORT=9000
+
+# Run with custom data directory
+make run DATA_DIR=/tmp/jormundb
+```
+
+### Test with AWS CLI
+
+```bash
+# Create a table
+aws dynamodb create-table \
+    --endpoint-url http://localhost:8000 \
+    --table-name Users \
+    --key-schema AttributeName=id,KeyType=HASH \
+    --attribute-definitions AttributeName=id,AttributeType=S \
+    --billing-mode PAY_PER_REQUEST
+
+# Put an item
+aws dynamodb put-item \
+    --endpoint-url http://localhost:8000 \
+    --table-name Users \
+    --item '{"id":{"S":"user123"},"name":{"S":"Alice"},"age":{"N":"30"}}'
+
+# Get an item
+aws dynamodb get-item \
+    --endpoint-url http://localhost:8000 \
+    --table-name Users \
+    --key '{"id":{"S":"user123"}}'
+
+# Query items
+aws dynamodb query \
+    --endpoint-url http://localhost:8000 \
+    --table-name Users \
+    --key-condition-expression "id = :id" \
+    --expression-attribute-values '{":id":{"S":"user123"}}'
+
+# Scan table
+aws dynamodb scan \
+    --endpoint-url http://localhost:8000 \
+    --table-name Users
+```
+
+## Architecture
+
+```
+HTTP Request (POST /)
+  ↓
+X-Amz-Target header → Operation routing
+  ↓
+JSON body → DynamoDB types
+  ↓
+Storage engine → RocksDB operations
+  ↓
+Binary encoding → Disk
+  ↓
+JSON response → Client
+```
+
+### Module Structure
+
+```
+jormundb/
+├── rocksdb/         - C FFI bindings to librocksdb
+├── dynamodb/        - Core types and operations
+│   ├── types.odin   - AttributeValue, Item, Key, etc.
+│   ├── json.odin    - DynamoDB JSON serialization
+│   ├── storage.odin - Storage engine with RocksDB
+│   └── handler.odin - HTTP request handlers
+├── key_codec/       - Binary key encoding (varint-prefixed)
+├── item_codec/      - Binary TLV item encoding
+└── main.odin        - HTTP server and entry point
+```
+
+### Storage Format
+
+**Keys** (varint-length-prefixed segments):
+```
+Meta:  [0x01][len][table_name]
+Data:  [0x02][len][table_name][len][pk_value][len][sk_value]?
+GSI:   [0x03][len][table_name][len][index_name][len][gsi_pk][len][gsi_sk]?
+LSI:   [0x04][len][table_name][len][index_name][len][pk][len][lsi_sk]
+```
+
+**Values** (TLV binary encoding):
+```
+[attr_count:varint]
+  [name_len:varint][name:bytes][type_tag:u8][value_encoded:bytes]...
+
+Type tags:
+  String=0x01, Number=0x02, Binary=0x03, Bool=0x04, Null=0x05
+  SS=0x10, NS=0x11, BS=0x12
+  List=0x20, Map=0x21
+```
+
+## Memory Management
+
+JormunDB uses Odin's context allocator system for elegant memory management:
+
+```odin
+// Request handler entry point
+handle_request :: proc(conn: net.TCP_Socket) {
+    arena: mem.Arena
+    mem.arena_init(&arena, make([]byte, mem.Megabyte * 4))
+    defer mem.arena_destroy(&arena)
+    
+    context.allocator = mem.arena_allocator(&arena)
+    
+    // Everything below uses the arena automatically
+    // No manual frees, no errdefer cleanup needed
+    request := parse_request()  // Uses context.allocator
+    response := process(request) // Uses context.allocator
+    send_response(response)      // Uses context.allocator
+    
+    // Arena is freed here automatically
+}
+```
+
+Long-lived data (table metadata, locks) uses the default allocator. Request-scoped data uses the arena.
+
+## Development
+
+```bash
+# Build debug version
+make build
+
+# Build optimized release
+make release
+
+# Run tests
+make test
+
+# Format code
+make fmt
+
+# Clean build artifacts
+make clean
+
+# Run with custom settings
+make run PORT=9000 DATA_DIR=/tmp/db VERBOSE=1
+```
+
+## Performance
+
+From benchmarks on the original Zig version (Odin expected to be similar or better):
+
+```
+Sequential Writes     |      10000 ops |   245.32 ms |    40765 ops/sec
+Random Reads          |      10000 ops |   312.45 ms |    32006 ops/sec
+Batch Writes          |      10000 ops |    89.23 ms |   112071 ops/sec
+PutItem               |       5000 ops |   892.34 ms |     5604 ops/sec
+GetItem               |       5000 ops |   678.91 ms |     7365 ops/sec
+Scan (full table)     |       5000 ops |   234.56 ms |    21320 ops/sec
+```
+
+## API Compatibility
+
+### Supported Operations
+
+- ✅ CreateTable
+- ✅ DeleteTable
+- ✅ DescribeTable
+- ✅ ListTables
+- ✅ PutItem
+- ✅ GetItem
+- ✅ DeleteItem
+- ✅ Query (with KeyConditionExpression)
+- ✅ Scan (with pagination)
+
+### Coming Soon
+
+- ⏳ UpdateItem (with UpdateExpression)
+- ⏳ BatchWriteItem
+- ⏳ BatchGetItem
+- ⏳ Global Secondary Indexes
+- ⏳ Local Secondary Indexes
+- ⏳ ConditionExpression
+- ⏳ FilterExpression
+- ⏳ ProjectionExpression
+
+## Configuration
+
+### Environment Variables
+
+```bash
+JORMUN_PORT=8000           # Server port
+JORMUN_HOST=0.0.0.0        # Bind address
+JORMUN_DATA_DIR=./data     # RocksDB data directory
+JORMUN_VERBOSE=1           # Enable verbose logging
+```
+
+### Command Line Arguments
+
+```bash
+./jormundb --port 9000 --host 127.0.0.1 --data-dir /var/db --verbose
+```
+
+## Troubleshooting
+
+### "Cannot open RocksDB"
+
+Ensure RocksDB libraries are installed and the data directory is writable:
+
+```bash
+# Check RocksDB installation
+pkg-config --libs rocksdb
+
+# Check permissions
+mkdir -p ./data
+chmod 755 ./data
+```
+
+### "Connection refused"
+
+Check if the port is already in use:
+
+```bash
+lsof -i :8000
+```
+
+### "Invalid JSON" errors
+
+Ensure you're using the correct DynamoDB JSON format:
+
+```json
+{
+  "TableName": "Users",
+  "Item": {
+    "id": {"S": "user123"},
+    "age": {"N": "30"}
+  }
+}
+```
+
+## License
+
+MIT License - see LICENSE file for details.
+
+## Credits
+
+- Inspired by DynamoDB Local
+- Built with [Odin](https://odin-lang.org/)
+- Powered by [RocksDB](https://rocksdb.org/)
+- Originally implemented as ZynamoDB in Zig
+
+## Contributing
+
+Contributions welcome! Please:
+
+1. Format code with `make fmt`
+2. Run tests with `make test`
+3. Update documentation as needed
+4. Follow Odin idioms (context allocators, explicit returns, etc.)
+
+---
+
+**Why "Jormun"?** Jörmungandr, the World Serpent from Norse mythology—a fitting name for something that wraps around your data. Also, it sounds cool.