biondizzle/jormun-db
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix docs and todo

Browse Source
This commit is contained in:
biondizzle
2026-02-15 11:42:43 -05:00
parent 37b423ce1a
commit ad599a0af7
5 changed files with 53 additions and 3373 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
Makefile
View File

@@ -41,7 +41,7 @@ COMMON_FLAGS := -vet -strict-style
EXTRA_LINKER_FLAGS := $(LIB_PATH) $(SHIM_LIB) $(ROCKSDB_LIBS) EXTRA_LINKER_FLAGS := $(LIB_PATH) $(SHIM_LIB) $(ROCKSDB_LIBS)
# Runtime configuration # Runtime configuration
PORT ?= 8000 PORT ?= 8002
HOST ?= 0.0.0.0 HOST ?= 0.0.0.0
DATA_DIR ?= ./data DATA_DIR ?= ./data
VERBOSE ?= 0 VERBOSE ?= 0
@@ -191,7 +191,7 @@ help:
@echo " make clean - Remove build artifacts" @echo " make clean - Remove build artifacts"
@echo "" @echo ""
@echo "$(GREEN)Run Commands:$(NC)" @echo "$(GREEN)Run Commands:$(NC)"
@echo " make run - Build and run server (default: localhost:8000)" @echo " make run - Build and run server (default: localhost:8002)"
@echo " make run PORT=9000 - Run on custom port" @echo " make run PORT=9000 - Run on custom port"
@echo " make dev - Clean, build, and run" @echo " make dev - Clean, build, and run"
@echo " make quick - Fast rebuild and run" @echo " make quick - Fast rebuild and run"

36
QUICKSTART.md
View File

@@ -101,7 +101,7 @@ export PATH=$PATH:/path/to/odin
### Basic Usage ### Basic Usage
```bash ```bash
# Run with defaults (localhost:8000, ./data directory) # Run with defaults (localhost:8002, ./data directory)
make run make run
``` ```
@@ -118,10 +118,10 @@ You should see:
║ ║ ║ ║
╚═══════════════════════════════════════════════╝ ╚═══════════════════════════════════════════════╝
Port: 8000 | Data Dir: ./data Port: 8002 | Data Dir: ./data
Storage engine initialized at ./data Storage engine initialized at ./data
Starting DynamoDB-compatible server on 0.0.0.0:8000 Starting DynamoDB-compatible server on 0.0.0.0:8002
Ready to accept connections! Ready to accept connections!
``` ```
@@ -186,7 +186,7 @@ aws configure
**Create a Table:** **Create a Table:**
```bash ```bash
aws dynamodb create-table \ aws dynamodb create-table \
--endpoint-url http://localhost:8000 \ --endpoint-url http://localhost:8002 \
--table-name Users \ --table-name Users \
--key-schema \ --key-schema \
AttributeName=id,KeyType=HASH \ AttributeName=id,KeyType=HASH \
@@ -197,13 +197,13 @@ aws dynamodb create-table \
**List Tables:** **List Tables:**
```bash ```bash
aws dynamodb list-tables --endpoint-url http://localhost:8000 aws dynamodb list-tables --endpoint-url http://localhost:8002
``` ```
**Put an Item:** **Put an Item:**
```bash ```bash
aws dynamodb put-item \ aws dynamodb put-item \
--endpoint-url http://localhost:8000 \ --endpoint-url http://localhost:8002 \
--table-name Users \ --table-name Users \
--item '{ --item '{
"id": {"S": "user123"}, "id": {"S": "user123"},
@@ -216,7 +216,7 @@ aws dynamodb put-item \
**Get an Item:** **Get an Item:**
```bash ```bash
aws dynamodb get-item \ aws dynamodb get-item \
--endpoint-url http://localhost:8000 \ --endpoint-url http://localhost:8002 \
--table-name Users \ --table-name Users \
--key '{"id": {"S": "user123"}}' --key '{"id": {"S": "user123"}}'
``` ```
@@ -224,7 +224,7 @@ aws dynamodb get-item \
**Query Items:** **Query Items:**
```bash ```bash
aws dynamodb query \ aws dynamodb query \
--endpoint-url http://localhost:8000 \ --endpoint-url http://localhost:8002 \
--table-name Users \ --table-name Users \
--key-condition-expression "id = :id" \ --key-condition-expression "id = :id" \
--expression-attribute-values '{ --expression-attribute-values '{
@@ -235,14 +235,14 @@ aws dynamodb query \
**Scan Table:** **Scan Table:**
```bash ```bash
aws dynamodb scan \ aws dynamodb scan \
--endpoint-url http://localhost:8000 \ --endpoint-url http://localhost:8002 \
--table-name Users --table-name Users
``` ```
**Delete an Item:** **Delete an Item:**
```bash ```bash
aws dynamodb delete-item \ aws dynamodb delete-item \
--endpoint-url http://localhost:8000 \ --endpoint-url http://localhost:8002 \
--table-name Users \ --table-name Users \
--key '{"id": {"S": "user123"}}' --key '{"id": {"S": "user123"}}'
``` ```
@@ -250,7 +250,7 @@ aws dynamodb delete-item \
**Delete a Table:** **Delete a Table:**
```bash ```bash
aws dynamodb delete-table \ aws dynamodb delete-table \
--endpoint-url http://localhost:8000 \ --endpoint-url http://localhost:8002 \
--table-name Users --table-name Users
``` ```
@@ -262,7 +262,7 @@ aws dynamodb delete-table \
const { DynamoDBClient, PutItemCommand, GetItemCommand } = require("@aws-sdk/client-dynamodb"); const { DynamoDBClient, PutItemCommand, GetItemCommand } = require("@aws-sdk/client-dynamodb");
const client = new DynamoDBClient({ const client = new DynamoDBClient({
endpoint: "http://localhost:8000", endpoint: "http://localhost:8002",
region: "us-east-1", region: "us-east-1",
credentials: { credentials: {
accessKeyId: "dummy", accessKeyId: "dummy",
@@ -279,13 +279,13 @@ async function test() {
name: { S: "Alice" } name: { S: "Alice" }
} }
})); }));
// Get the item // Get the item
const result = await client.send(new GetItemCommand({ const result = await client.send(new GetItemCommand({
TableName: "Users", TableName: "Users",
Key: { id: { S: "user123" } } Key: { id: { S: "user123" } }
})); }));
console.log(result.Item); console.log(result.Item);
} }
@@ -299,7 +299,7 @@ import boto3
dynamodb = boto3.client( dynamodb = boto3.client(
'dynamodb', 'dynamodb',
endpoint_url='http://localhost:8000', endpoint_url='http://localhost:8002',
region_name='us-east-1', region_name='us-east-1',
aws_access_key_id='dummy', aws_access_key_id='dummy',
aws_secret_access_key='dummy' aws_secret_access_key='dummy'
@@ -364,8 +364,8 @@ make fmt
### Port Already in Use ### Port Already in Use
```bash ```bash
# Check what's using port 8000 # Check what's using port 8002
lsof -i :8000 lsof -i :8002
# Use a different port # Use a different port
make run PORT=9000 make run PORT=9000
@@ -426,7 +426,7 @@ make profile
# Load test # Load test
ab -n 10000 -c 100 -p item.json -T application/json \ ab -n 10000 -c 100 -p item.json -T application/json \
http://localhost:8000/ http://localhost:8002/
``` ```
## Production Deployment ## Production Deployment

24
README.md
View File

@@ -3,7 +3,7 @@
A high-performance, DynamoDB-compatible database server written in Odin, backed by RocksDB. A high-performance, DynamoDB-compatible database server written in Odin, backed by RocksDB.
``` ```
╦╔═╗╦═╗╔╦╗╦ ╦╔╗╔╔╦╗╔╗ ╦╔═╗╦═╗╔╦╗╦ ╦╔╗╔╔╦╗╔╗
║║ ║╠╦╝║║║║ ║║║║ ║║╠╩╗ ║║ ║╠╦╝║║║║ ║║║║ ║║╠╩╗
╚╝╚═╝╩╚═╩ ╩╚═╝╝╚╝═╩╝╚═╝ ╚╝╚═╝╩╚═╩ ╩╚═╝╝╚╝═╩╝╚═╝
DynamoDB-Compatible Database DynamoDB-Compatible Database
@@ -55,7 +55,7 @@ sudo apt install librocksdb-dev libsnappy-dev liblz4-dev libzstd-dev libbz2-dev
# Build the server # Build the server
make build make build
# Run with default settings (localhost:8000, ./data directory) # Run with default settings (localhost:8002, ./data directory)
make run make run
# Run with custom port # Run with custom port
@@ -70,7 +70,7 @@ make run DATA_DIR=/tmp/jormundb
```bash ```bash
# Create a table # Create a table
aws dynamodb create-table \ aws dynamodb create-table \
--endpoint-url http://localhost:8000 \ --endpoint-url http://localhost:8002 \
--table-name Users \ --table-name Users \
--key-schema AttributeName=id,KeyType=HASH \ --key-schema AttributeName=id,KeyType=HASH \
--attribute-definitions AttributeName=id,AttributeType=S \ --attribute-definitions AttributeName=id,AttributeType=S \
@@ -78,26 +78,26 @@ aws dynamodb create-table \
# Put an item # Put an item
aws dynamodb put-item \ aws dynamodb put-item \
--endpoint-url http://localhost:8000 \ --endpoint-url http://localhost:8002 \
--table-name Users \ --table-name Users \
--item '{"id":{"S":"user123"},"name":{"S":"Alice"},"age":{"N":"30"}}' --item '{"id":{"S":"user123"},"name":{"S":"Alice"},"age":{"N":"30"}}'
# Get an item # Get an item
aws dynamodb get-item \ aws dynamodb get-item \
--endpoint-url http://localhost:8000 \ --endpoint-url http://localhost:8002 \
--table-name Users \ --table-name Users \
--key '{"id":{"S":"user123"}}' --key '{"id":{"S":"user123"}}'
# Query items # Query items
aws dynamodb query \ aws dynamodb query \
--endpoint-url http://localhost:8000 \ --endpoint-url http://localhost:8002 \
--table-name Users \ --table-name Users \
--key-condition-expression "id = :id" \ --key-condition-expression "id = :id" \
--expression-attribute-values '{":id":{"S":"user123"}}' --expression-attribute-values '{":id":{"S":"user123"}}'
# Scan table # Scan table
aws dynamodb scan \ aws dynamodb scan \
--endpoint-url http://localhost:8000 \ --endpoint-url http://localhost:8002 \
--table-name Users --table-name Users
``` ```
@@ -163,15 +163,15 @@ handle_request :: proc(conn: net.TCP_Socket) {
arena: mem.Arena arena: mem.Arena
mem.arena_init(&arena, make([]byte, mem.Megabyte * 4)) mem.arena_init(&arena, make([]byte, mem.Megabyte * 4))
defer mem.arena_destroy(&arena) defer mem.arena_destroy(&arena)
context.allocator = mem.arena_allocator(&arena) context.allocator = mem.arena_allocator(&arena)
// Everything below uses the arena automatically // Everything below uses the arena automatically
// No manual frees, no errdefer cleanup needed // No manual frees, no errdefer cleanup needed
request := parse_request() // Uses context.allocator request := parse_request() // Uses context.allocator
response := process(request) // Uses context.allocator response := process(request) // Uses context.allocator
send_response(response) // Uses context.allocator send_response(response) // Uses context.allocator
// Arena is freed here automatically // Arena is freed here automatically
} }
``` ```
@@ -243,7 +243,7 @@ Scan (full table) | 5000 ops | 234.56 ms | 21320 ops/sec
### Environment Variables ### Environment Variables
```bash ```bash
JORMUN_PORT=8000 # Server port JORMUN_PORT=8002 # Server port
JORMUN_HOST=0.0.0.0 # Bind address JORMUN_HOST=0.0.0.0 # Bind address
JORMUN_DATA_DIR=./data # RocksDB data directory JORMUN_DATA_DIR=./data # RocksDB data directory
JORMUN_VERBOSE=1 # Enable verbose logging JORMUN_VERBOSE=1 # Enable verbose logging
@@ -275,7 +275,7 @@ chmod 755 ./data
Check if the port is already in use: Check if the port is already in use:
```bash ```bash
lsof -i :8000 lsof -i :8002
``` ```
### "Invalid JSON" errors ### "Invalid JSON" errors

25
TODO.md
View File

@@ -12,8 +12,8 @@ This tracks the rewrite from Zig to Odin and remaining features.
- [x] Core types (dynamodb/types.odin) - [x] Core types (dynamodb/types.odin)
- [x] Key codec with varint encoding (key_codec/key_codec.odin) - [x] Key codec with varint encoding (key_codec/key_codec.odin)
- [x] Main entry point with arena pattern demo - [x] Main entry point with arena pattern demo
- [x] LICENSE file
- [x] .gitignore - [x] .gitignore
- [x] HTTP Server Scaffolding
## 🚧 In Progress (Need to Complete) ## 🚧 In Progress (Need to Complete)
@@ -23,7 +23,7 @@ This tracks the rewrite from Zig to Odin and remaining features.
- Parse `{"S": "value"}` format - Parse `{"S": "value"}` format
- Serialize AttributeValue to DynamoDB JSON - Serialize AttributeValue to DynamoDB JSON
- Parse request bodies (PutItem, GetItem, etc.) - Parse request bodies (PutItem, GetItem, etc.)
- [ ] **item_codec/item_codec.odin** - Binary TLV encoding for items - [ ] **item_codec/item_codec.odin** - Binary TLV encoding for items
- Encode Item to binary TLV format - Encode Item to binary TLV format
- Decode binary TLV back to Item - Decode binary TLV back to Item
@@ -50,7 +50,7 @@ This tracks the rewrite from Zig to Odin and remaining features.
- Read JSON bodies - Read JSON bodies
- Send HTTP responses with headers - Send HTTP responses with headers
- Keep-alive support - Keep-alive support
- Options: - Options (Why we haven't checked this off yet, we need to make sure we chose the right option as the project grows, might make more sense to impliment different option):
- Use `core:net` directly - Use `core:net` directly
- Use C FFI with libmicrohttpd - Use C FFI with libmicrohttpd
- Use Odin's vendor:microui (if suitable) - Use Odin's vendor:microui (if suitable)
@@ -69,6 +69,23 @@ This tracks the rewrite from Zig to Odin and remaining features.
- ADD operations - ADD operations
- DELETE operations - DELETE operations
### Replication Support (Priority 4)
- [ ] **Build C++ Shim in order to use RocksDB's WAL replication helpers**
- [ ] **Add configurator to set instance as a master or slave node and point to proper Target and Destination IPs**
- [ ] **Leverage C++ helpers from shim**
### Subscribe To Changes Feature (Priority LAST [But keep in mind because semantics we decide now will make this easier later])
- [ ] **Best-effort notifications (Postgres-ish LISTEN/NOTIFY [in-memory pub/sub fanout. If you’re not connected, you miss it.])**
- Add an in-process “event bus” channels: table-wide, partition-key, item-key, “all”.
- When putItem/deleteItem/updateItem/createTable/... commits successfully publish {op, table, key, timestamp, item?}
- [ ] **Durable change streams (Mongo-ish [append every mutation to a persistent log and let consumers read it with resume tokens.])**
- Create a “changelog” keyspace
- Generate a monotonically increasing sequence by using a stable Per-partition sequence cursor
- Expose via an API (I prefer publishing to MQTT or SSE)
## 📋 Testing ## 📋 Testing
- [ ] Unit tests for key_codec - [ ] Unit tests for key_codec
@@ -182,5 +199,5 @@ make test
make run make run
# Test with AWS CLI # Test with AWS CLI
aws dynamodb list-tables --endpoint-url http://localhost:8000 aws dynamodb list-tables --endpoint-url http://localhost:8002
``` ```

3337
project_context.txt
View File

File diff suppressed because it is too large Load Diff