jormun-db/TODO.md

JormunDB Implementation TODO

This tracks the rewrite from Zig to Odin and remaining features.

✅ Completed

• Project structure
• Makefile with build/run/test targets
• README with usage instructions
• ARCHITECTURE documentation
• RocksDB FFI bindings (rocksdb/rocksdb.odin)
• Core types (dynamodb/types.odin)
• Key codec with varint encoding (key_codec/key_codec.odin)
• Main entry point with arena pattern demo
• .gitignore
• HTTP Server Scaffolding
• JSON Parser
• Item_codec
• Storage

🚧 In Progress (Need to Complete)

Core Modules

• dynamodb/json.odin - DynamoDB JSON parsing and serialization

  • Parse {"S": "value"} format
  • Serialize AttributeValue to DynamoDB JSON
  • Parse request bodies (PutItem, GetItem, etc.)

• item_codec/item_codec.odin - Binary TLV encoding for items

  • Encode Item to binary TLV format
  • Decode binary TLV back to Item
  • Type tag handling for all DynamoDB types

• dynamodb/storage.odin - Storage engine with RocksDB

  • Table metadata management
  • create_table, delete_table, describe_table, list_tables
  • put_item, get_item, delete_item
  • scan, query with pagination
  • Table-level RW locks

HTTP Server

• HTTP server implementation (MOSTLY DONE CONSOLIDATED HANDLER INTO MAIN AND HTTO FILES. NO NEED FOR A STAND ALONE HANDLER LIKE WE DID IN ZIG! JUST PLEASE GO OVER WHAT IS THERE!!!)
  • Accept TCP connections
  • Parse HTTP POST requests
  • Read JSON bodies
  • Send HTTP responses with headers
  • Keep-alive support
  • Route X-Amz-Target functions (this was the handler in zig but no need for that crap in odin land)
  • handle_create_table, handle_put_item, etc. (this was the handler in zig but no need for that crap in odin land)
  • Build responses with proper error handling (this was the handler in zig but no need for that crap in odin land)
  • Arena allocator integration
  • Options (Why we haven't checked this off yet, we need to make sure we chose the right option as the project grows, might make more sense to impliment different option):
    • Use core:net directly
    • Use C FFI with libmicrohttpd
    • Use Odin's vendor:microui (if suitable)

Expression Parsers (Priority 3)

• KeyConditionExpression parser

  • Tokenizer for expressions
  • Parse pk = :pk AND sk > :sk
  • Support begins_with, BETWEEN
  • ExpressionAttributeNames/Values

• UpdateExpression parser (later)

  • SET operations
  • REMOVE operations
  • ADD operations
  • DELETE operations

Credential Support (Priority 4)

• Support a way to configure AWS compatible credentials.
  • This is very important because remember when mongo didn't come with a root password by default and everyone who had the port open to the world got their DB ransomed? Yeah, we don't want that to happen

Replication Support (Priority 5)

• Build C++ Shim in order to use RocksDB's WAL replication helpers
• Add configurator to set instance as a master or slave node and point to proper Target and Destination IPs
• Leverage C++ helpers from shim

Subscribe To Changes Feature (Priority LAST [But keep in mind because semantics we decide now will make this easier later])

• Best-effort notifications (Postgres-ish LISTEN/NOTIFY [in-memory pub/sub fanout. If you’re not connected, you miss it.])

  • Add an in-process “event bus” channels: table-wide, partition-key, item-key, “all”.
  • When putItem/deleteItem/updateItem/createTable/... commits successfully publish {op, table, key, timestamp, item?}

• Durable change streams (Mongo-ish [append every mutation to a persistent log and let consumers read it with resume tokens.])

  • Create a “changelog” keyspace
  • Generate a monotonically increasing sequence by using a stable Per-partition sequence cursor
  • Expose via an API (I prefer publishing to MQTT or SSE)

📋 Testing

• Unit tests for key_codec
• Unit tests for item_codec
• Unit tests for JSON parsing
• Integration tests with AWS CLI
• Benchmark suite

🔧 Build & Tooling

• Verify Makefile works on macOS
• Verify Makefile works on Linux
• Add Docker support (optional)
• Add install script

📚 Documentation

• Code comments for public APIs
• Usage examples in README
• API compatibility matrix
• Performance tuning guide

🎯 Priority Order

1. HTTP Server - Need this to accept requests
2. JSON Parsing - Need this to understand DynamoDB format
3. Storage Engine - Core CRUD operations
4. Handlers - Wire everything together
5. Item Codec - Efficient binary storage
6. Expression Parsers - Query functionality

📝 Notes

Zig → Odin Translation Patterns

Memory Management:

// Zig
const item = try allocator.create(Item);
defer allocator.destroy(item);

// Odin
item := new(Item)
// No defer needed if using arena

Error Handling:

// Zig
fn foo() !Result {
    return error.Failed;
}
const x = try foo();

// Odin
foo :: proc() -> (Result, bool) {
    return {}, false
}
x := foo() or_return

Slices:

// Zig
const slice: []const u8 = data;

// Odin
slice: []byte = data

Maps:

// Zig
var map = std.StringHashMap(Value).init(allocator);
defer map.deinit();

// Odin
map := make(map[string]Value)
defer delete(map)

Key Decisions

1. Use Maybe(T) instead of ?T - Odin's optional type
2. Use or_return instead of try - Odin's error propagation
3. Use context.allocator - Implicit allocator from context
4. Use #partial switch - For union type checking
5. Use transmute - For zero-cost type conversions

Reference Zig Files

When implementing, reference these Zig files:

• src/dynamodb/json.zig - 400 lines, DynamoDB JSON format
• src/dynamodb/storage.zig - 460 lines, storage engine
• src/dynamodb/handler.zig - 500+ lines, request handlers
• src/item_codec.zig - 350 lines, TLV encoding
• src/http.zig - 250 lines, HTTP server

Quick Test Commands

# Build and test
make build
make test

# Run server
make run

# Test with AWS CLI
aws dynamodb list-tables --endpoint-url http://localhost:8002