diff --git a/README.md b/README.md index 1d41bf5..901e824 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,16 @@ -# JormunDB +

+ JormunDB logo +

-A high-performance, DynamoDB-compatible database server written in Odin, backed by RocksDB. +

JormunDB

-``` - ╦╔═╗╦═╗╔╦╗╦ ╦╔╗╔╔╦╗╔╗ - ║║ ║╠╦╝║║║║ ║║║║ ║║╠╩╗ - ╚╝╚═╝╩╚═╩ ╩╚═╝╝╚╝═╩╝╚═╝ - DynamoDB-Compatible Database - Powered by RocksDB + Odin -``` +

+ A high-performance, DynamoDB-compatible database server written in Odin, backed by RocksDB. +
+ DynamoDB-Compatible Database · Powered by RocksDB + Odin +

+ +--- ## What is JormunDB? @@ -28,286 +30,3 @@ JormunDB (formerly ZynamoDB) is a Self-Hosted DynamoDB replacement that speaks t - ✅ **Persistent Storage**: RocksDB-backed with full ACID guarantees - ✅ **Concurrency**: Table-level RW locks for safe concurrent access -## Quick Start - -### Prerequisites - -- Odin compiler (latest) -- RocksDB development libraries -- Standard compression libraries (snappy, lz4, zstd, etc.) - -#### macOS (Homebrew) - -```bash -brew install rocksdb odin -``` - -#### Ubuntu/Debian - -```bash -sudo apt install librocksdb-dev libsnappy-dev liblz4-dev libzstd-dev libbz2-dev -# Install Odin from https://odin-lang.org/docs/install/ -``` - -### Build & Run - -```bash -# Build the server -make build - -# Run with default settings (localhost:8002, ./data directory) -make run - -# Run with custom port -make run PORT=9000 - -# Run with custom data directory -make run DATA_DIR=/tmp/jormundb -``` - -### Test with AWS CLI - -```bash -# Create a table -aws dynamodb create-table \ - --endpoint-url http://localhost:8002 \ - --table-name Users \ - --key-schema AttributeName=id,KeyType=HASH \ - --attribute-definitions AttributeName=id,AttributeType=S \ - --billing-mode PAY_PER_REQUEST - -# Put an item -aws dynamodb put-item \ - --endpoint-url http://localhost:8002 \ - --table-name Users \ - --item '{"id":{"S":"user123"},"name":{"S":"Alice"},"age":{"N":"30"}}' - -# Get an item -aws dynamodb get-item \ - --endpoint-url http://localhost:8002 \ - --table-name Users \ - --key '{"id":{"S":"user123"}}' - -# Query items -aws dynamodb query \ - --endpoint-url http://localhost:8002 \ - --table-name Users \ - --key-condition-expression "id = :id" \ - --expression-attribute-values '{":id":{"S":"user123"}}' - -# Scan table -aws dynamodb scan \ - --endpoint-url http://localhost:8002 \ - --table-name Users -``` - -## Architecture - -``` -HTTP Request (POST /) - ↓ -X-Amz-Target header → Operation routing - ↓ -JSON body → DynamoDB types - ↓ -Storage engine → RocksDB operations - ↓ -Binary encoding → Disk - ↓ -JSON response → Client -``` - -### Module Structure - -``` -jormundb/ -├── rocksdb/ - C FFI bindings to librocksdb -├── dynamodb/ - Core types and operations -│ ├── types.odin - AttributeValue, Item, Key, etc. -│ ├── json.odin - DynamoDB JSON serialization -│ ├── storage.odin - Storage engine with RocksDB -│ └── handler.odin - HTTP request handlers -├── key_codec/ - Binary key encoding (varint-prefixed) -├── item_codec/ - Binary TLV item encoding -└── main.odin - HTTP server and entry point -``` - -### Storage Format - -**Keys** (varint-length-prefixed segments): -``` -Meta: [0x01][len][table_name] -Data: [0x02][len][table_name][len][pk_value][len][sk_value]? -GSI: [0x03][len][table_name][len][index_name][len][gsi_pk][len][gsi_sk]? -LSI: [0x04][len][table_name][len][index_name][len][pk][len][lsi_sk] -``` - -**Values** (TLV binary encoding): -``` -[attr_count:varint] - [name_len:varint][name:bytes][type_tag:u8][value_encoded:bytes]... - -Type tags: - String=0x01, Number=0x02, Binary=0x03, Bool=0x04, Null=0x05 - SS=0x10, NS=0x11, BS=0x12 - List=0x20, Map=0x21 -``` - -## Memory Management - -JormunDB uses Odin's context allocator system for elegant memory management: - -```odin -// Request handler entry point -handle_request :: proc(conn: net.TCP_Socket) { - arena: mem.Arena - mem.arena_init(&arena, make([]byte, mem.Megabyte * 4)) - defer mem.arena_destroy(&arena) - - context.allocator = mem.arena_allocator(&arena) - - // Everything below uses the arena automatically - // No manual frees, no errdefer cleanup needed - request := parse_request() // Uses context.allocator - response := process(request) // Uses context.allocator - send_response(response) // Uses context.allocator - - // Arena is freed here automatically -} -``` - -Long-lived data (table metadata, locks) uses the default allocator. Request-scoped data uses the arena. - -## Development - -```bash -# Build debug version -make build - -# Build optimized release -make release - -# Run tests -make test - -# Format code -make fmt - -# Clean build artifacts -make clean - -# Run with custom settings -make run PORT=9000 DATA_DIR=/tmp/db VERBOSE=1 -``` - -## Performance - -From benchmarks on the original Zig version (Odin expected to be similar or better): - -``` -Sequential Writes | 10000 ops | 245.32 ms | 40765 ops/sec -Random Reads | 10000 ops | 312.45 ms | 32006 ops/sec -Batch Writes | 10000 ops | 89.23 ms | 112071 ops/sec -PutItem | 5000 ops | 892.34 ms | 5604 ops/sec -GetItem | 5000 ops | 678.91 ms | 7365 ops/sec -Scan (full table) | 5000 ops | 234.56 ms | 21320 ops/sec -``` - -## API Compatibility - -### Supported Operations - -- ✅ CreateTable -- ✅ DeleteTable -- ✅ DescribeTable -- ✅ ListTables -- ✅ PutItem -- ✅ GetItem -- ✅ DeleteItem -- ✅ Query (with KeyConditionExpression) -- ✅ Scan (with pagination) - -### Coming Soon - -- ⏳ UpdateItem (with UpdateExpression) -- ⏳ BatchWriteItem -- ⏳ BatchGetItem -- ⏳ Global Secondary Indexes -- ⏳ Local Secondary Indexes -- ⏳ ConditionExpression -- ⏳ FilterExpression -- ⏳ ProjectionExpression - -## Configuration - -### Environment Variables - -```bash -JORMUN_PORT=8002 # Server port -JORMUN_HOST=0.0.0.0 # Bind address -JORMUN_DATA_DIR=./data # RocksDB data directory -JORMUN_VERBOSE=1 # Enable verbose logging -``` - -### Command Line Arguments - -```bash -./jormundb --port 9000 --host 127.0.0.1 --data-dir /var/db --verbose -``` - -## Troubleshooting - -### "Cannot open RocksDB" - -Ensure RocksDB libraries are installed and the data directory is writable: - -```bash -# Check RocksDB installation -pkg-config --libs rocksdb - -# Check permissions -mkdir -p ./data -chmod 755 ./data -``` - -### "Connection refused" - -Check if the port is already in use: - -```bash -lsof -i :8002 -``` - -### "Invalid JSON" errors - -Ensure you're using the correct DynamoDB JSON format: - -```json -{ - "TableName": "Users", - "Item": { - "id": {"S": "user123"}, - "age": {"N": "30"} - } -} -``` - -## Credits - -- Inspired by DynamoDB -- Built with [Odin](https://odin-lang.org/) -- Powered by [RocksDB](https://rocksdb.org/) -- Originally implemented as ZynamoDB in Zig - -## Contributing - -Contributions welcome! Please: - -1. Format code with `make fmt` -2. Run tests with `make test` -3. Update documentation as needed -4. Follow Odin idioms (context allocators, explicit returns, etc.) - ---- - -**Why "Jormun"?** Jörmungandr, the World Serpent from Norse mythology—a fitting name for something built in a language called Odin. Also, it sounds cool.