diff --git a/README.md b/README.md index 901e824..0c0e3ce 100644 --- a/README.md +++ b/README.md @@ -30,3 +30,288 @@ JormunDB (formerly ZynamoDB) is a Self-Hosted DynamoDB replacement that speaks t - ✅ **Persistent Storage**: RocksDB-backed with full ACID guarantees - ✅ **Concurrency**: Table-level RW locks for safe concurrent access +## Quick Start + +### Prerequisites + +- Odin compiler (latest) +- RocksDB development libraries +- Standard compression libraries (snappy, lz4, zstd, etc.) + +#### macOS (Homebrew) + +```bash +brew install rocksdb odin +``` + +#### Ubuntu/Debian + +```bash +sudo apt install librocksdb-dev libsnappy-dev liblz4-dev libzstd-dev libbz2-dev +# Install Odin from https://odin-lang.org/docs/install/ +``` + +### Build & Run + +```bash +# Build the server +make build + +# Run with default settings (localhost:8002, ./data directory) +make run + +# Run with custom port +make run PORT=9000 + +# Run with custom data directory +make run DATA_DIR=/tmp/jormundb +``` + +### Test with AWS CLI + +```bash +# Create a table +aws dynamodb create-table \ + --endpoint-url http://localhost:8002 \ + --table-name Users \ + --key-schema AttributeName=id,KeyType=HASH \ + --attribute-definitions AttributeName=id,AttributeType=S \ + --billing-mode PAY_PER_REQUEST + +# Put an item +aws dynamodb put-item \ + --endpoint-url http://localhost:8002 \ + --table-name Users \ + --item '{"id":{"S":"user123"},"name":{"S":"Alice"},"age":{"N":"30"}}' + +# Get an item +aws dynamodb get-item \ + --endpoint-url http://localhost:8002 \ + --table-name Users \ + --key '{"id":{"S":"user123"}}' + +# Query items +aws dynamodb query \ + --endpoint-url http://localhost:8002 \ + --table-name Users \ + --key-condition-expression "id = :id" \ + --expression-attribute-values '{":id":{"S":"user123"}}' + +# Scan table +aws dynamodb scan \ + --endpoint-url http://localhost:8002 \ + --table-name Users +``` + +## Architecture + +``` +HTTP Request (POST /) + ↓ +X-Amz-Target header → Operation routing + ↓ +JSON body → DynamoDB types + ↓ +Storage engine → RocksDB operations + ↓ +Binary encoding → Disk + ↓ +JSON response → Client +``` + +### Module Structure + +``` +jormundb/ +├── rocksdb/ - C FFI bindings to librocksdb +├── rocksdb_shim/ - C++ FFI bindings to librocksdb (so we can use the WAL helper functions) +├── dynamodb/ - Core types and operations +│ ├── types.odin - AttributeValue, Item, Key, etc. +│ ├── json.odin - DynamoDB JSON serialization +│ ├── storage.odin - Storage engine with RocksDB +│ ├── key_codec.odin - Binary key encoding +│ ├── item_codec.odin - Binary TLV item encoding +│ └── handler.odin - HTTP request handlers +├── http.odin - HTTP Server +└── main.odin - HTTP Router and handler (entry point) +``` + +### Storage Format + +**Keys** (varint-length-prefixed segments): +``` +Meta: [0x01][len][table_name] +Data: [0x02][len][table_name][len][pk_value][len][sk_value]? +GSI: [0x03][len][table_name][len][index_name][len][gsi_pk][len][gsi_sk]? +LSI: [0x04][len][table_name][len][index_name][len][pk][len][lsi_sk] +``` + +**Values** (TLV binary encoding): +``` +[attr_count:varint] + [name_len:varint][name:bytes][type_tag:u8][value_encoded:bytes]... + +Type tags: + String=0x01, Number=0x02, Binary=0x03, Bool=0x04, Null=0x05 + SS=0x10, NS=0x11, BS=0x12 + List=0x20, Map=0x21 +``` + +## Memory Management + +JormunDB uses Odin's context allocator system for elegant memory management: + +```odin +// Request handler entry point +handle_request :: proc(conn: net.TCP_Socket) { + arena: mem.Arena + mem.arena_init(&arena, make([]byte, mem.Megabyte * 4)) + defer mem.arena_destroy(&arena) + + context.allocator = mem.arena_allocator(&arena) + + // Everything below uses the arena automatically + // No manual frees, no errdefer cleanup needed + request := parse_request() // Uses context.allocator + response := process(request) // Uses context.allocator + send_response(response) // Uses context.allocator + + // Arena is freed here automatically +} +``` + +Long-lived data (table metadata, locks) uses the default allocator. Request-scoped data uses the arena. + +## Development + +```bash +# Build debug version +make build + +# Build optimized release +make release + +# Run tests +make test + +# Format code +make fmt + +# Clean build artifacts +make clean + +# Run with custom settings +make run PORT=9000 DATA_DIR=/tmp/db VERBOSE=1 +``` + +## Performance + +From benchmarks on the original Zig version (Odin expected to be similar or better): + +``` +Sequential Writes | 10000 ops | 245.32 ms | 40765 ops/sec +Random Reads | 10000 ops | 312.45 ms | 32006 ops/sec +Batch Writes | 10000 ops | 89.23 ms | 112071 ops/sec +PutItem | 5000 ops | 892.34 ms | 5604 ops/sec +GetItem | 5000 ops | 678.91 ms | 7365 ops/sec +Scan (full table) | 5000 ops | 234.56 ms | 21320 ops/sec +``` + +## API Compatibility + +### Supported Operations + +- ✅ CreateTable +- ✅ DeleteTable +- ✅ DescribeTable +- ✅ ListTables +- ✅ PutItem +- ✅ GetItem +- ✅ DeleteItem +- ✅ Query (with KeyConditionExpression) +- ✅ Scan (with pagination) + +### Coming Soon + +- ⏳ UpdateItem (with UpdateExpression) +- ⏳ BatchWriteItem +- ⏳ BatchGetItem +- ⏳ Global Secondary Indexes +- ⏳ Local Secondary Indexes +- ⏳ ConditionExpression +- ⏳ FilterExpression +- ⏳ ProjectionExpression + +## Configuration + +### Environment Variables + +```bash +JORMUN_PORT=8002 # Server port (I have something locally on port 8000 so now everyone has to use port 8002) +JORMUN_HOST=0.0.0.0 # Bind address +JORMUN_DATA_DIR=./data # RocksDB data directory +JORMUN_VERBOSE=1 # Enable verbose logging +``` + +### Command Line Arguments + +```bash +./jormundb --port 9000 --host 127.0.0.1 --data-dir /var/db --verbose +``` + +## Troubleshooting + +### "Cannot open RocksDB" + +Ensure RocksDB libraries are installed and the data directory is writable: + +```bash +# Check RocksDB installation +pkg-config --libs rocksdb + +# Check permissions +mkdir -p ./data +chmod 755 ./data +``` + +### "Connection refused" + +Check if the port is already in use: + +```bash +lsof -i :8002 +``` + +### "Invalid JSON" errors + +Ensure you're using the correct DynamoDB JSON format: + +```json +{ + "TableName": "Users", + "Item": { + "id": {"S": "user123"}, + "age": {"N": "30"} + } +} +``` + +## Credits + +- Inspired by DynamoDB +- Built with [Odin](https://odin-lang.org/) +- Powered by [RocksDB](https://rocksdb.org/) +- Originally implemented as ZynamoDB in Zig + +## Contributing + +Contributions welcome! Please: + +1. Format code with `make fmt` +2. Run tests with `make test` +3. Update documentation as needed +4. Follow Odin idioms (context allocators, explicit returns, etc.) + +--- + +**Why "Jormun"?** Jörmungandr, the World Serpent from Norse mythology, which I found fitting for something built in a language called Odin. Also, it sounds cool.