jormun-db/TODO.md

# JormunDB Implementation TODO

This tracks the rewrite from Zig to Odin and remaining features.

## ✅ Completed

- [x] Project structure
- [x] Makefile with build/run/test targets
- [x] README with usage instructions
- [x] ARCHITECTURE documentation
- [x] RocksDB FFI bindings (rocksdb/rocksdb.odin)
- [x] Core types (dynamodb/types.odin)
- [x] Key codec with varint encoding (key_codec/key_codec.odin)
- [x] Main entry point with arena pattern demo
- [x] .gitignore
- [x] HTTP Server Scaffolding

## 🚧 In Progress (Need to Complete)

### Core Modules

- [ ] **dynamodb/json.odin** - DynamoDB JSON parsing and serialization
  - Parse `{"S": "value"}` format
  - Serialize AttributeValue to DynamoDB JSON
  - Parse request bodies (PutItem, GetItem, etc.)

- [ ] **item_codec/item_codec.odin** - Binary TLV encoding for items
  - Encode Item to binary TLV format
  - Decode binary TLV back to Item
  - Type tag handling for all DynamoDB types

- [ ] **dynamodb/storage.odin** - Storage engine with RocksDB
  - Table metadata management
  - create_table, delete_table, describe_table, list_tables
  - put_item, get_item, delete_item
  - scan, query with pagination
  - Table-level RW locks

- [ ] **dynamodb/handler.odin** - HTTP request handlers
  - Route X-Amz-Target to handler functions
  - handle_create_table, handle_put_item, etc.
  - Build responses with proper error handling
  - Arena allocator integration

### HTTP Server

- [ ] **HTTP server implementation**
  - Accept TCP connections
  - Parse HTTP POST requests
  - Read JSON bodies
  - Send HTTP responses with headers
  - Keep-alive support
  - Options (Why we haven't checked this off yet, we need to make sure we chose the right option as the project grows, might make more sense to impliment different option):
    - Use `core:net` directly
    - Use C FFI with libmicrohttpd
    - Use Odin's vendor:microui (if suitable)

### Expression Parsers (Priority 3)

- [ ] **KeyConditionExpression parser**
  - Tokenizer for expressions
  - Parse `pk = :pk AND sk > :sk`
  - Support begins_with, BETWEEN
  - ExpressionAttributeNames/Values

- [ ] **UpdateExpression parser** (later)
  - SET operations
  - REMOVE operations
  - ADD operations
  - DELETE operations

### Replication Support (Priority 4)

  - [ ] **Build C++ Shim in order to use RocksDB's WAL replication helpers**
  - [ ] **Add configurator to set instance as a master or slave node and point to proper Target and Destination IPs**
  - [ ] **Leverage C++ helpers from shim**

### Subscribe To Changes Feature (Priority LAST [But keep in mind because semantics we decide now will make this easier later])

  - [ ] **Best-effort notifications (Postgres-ish LISTEN/NOTIFY [in-memory pub/sub fanout. If you’re not connected, you miss it.])**
    - Add an in-process “event bus” channels: table-wide, partition-key, item-key, “all”.
    - When putItem/deleteItem/updateItem/createTable/... commits successfully publish {op, table, key, timestamp, item?}

  - [ ] **Durable change streams (Mongo-ish [append every mutation to a persistent log and let consumers read it with resume tokens.])**
    - Create a “changelog” keyspace
    - Generate a monotonically increasing sequence by using a stable Per-partition sequence cursor
    - Expose via an API (I prefer publishing to MQTT or SSE)

## 📋 Testing

- [ ] Unit tests for key_codec
- [ ] Unit tests for item_codec
- [ ] Unit tests for JSON parsing
- [ ] Integration tests with AWS CLI
- [ ] Benchmark suite

## 🔧 Build & Tooling

- [ ] Verify Makefile works on macOS
- [ ] Verify Makefile works on Linux
- [ ] Add Docker support (optional)
- [ ] Add install script

## 📚 Documentation

- [ ] Code comments for public APIs
- [ ] Usage examples in README
- [ ] API compatibility matrix
- [ ] Performance tuning guide

## 🎯 Priority Order

1. **HTTP Server** - Need this to accept requests
2. **JSON Parsing** - Need this to understand DynamoDB format
3. **Storage Engine** - Core CRUD operations
4. **Handlers** - Wire everything together
5. **Item Codec** - Efficient binary storage
6. **Expression Parsers** - Query functionality

## 📝 Notes

### Zig → Odin Translation Patterns

**Memory Management:**
```zig
// Zig
const item = try allocator.create(Item);
defer allocator.destroy(item);
```
```odin
// Odin
item := new(Item)
// No defer needed if using arena
```

**Error Handling:**
```zig
// Zig
fn foo() !Result {
    return error.Failed;
}
const x = try foo();
```
```odin
// Odin
foo :: proc() -> (Result, bool) {
    return {}, false
}
x := foo() or_return
```

**Slices:**
```zig
// Zig
const slice: []const u8 = data;
```
```odin
// Odin
slice: []byte = data
```

**Maps:**
```zig
// Zig
var map = std.StringHashMap(Value).init(allocator);
defer map.deinit();
```
```odin
// Odin
map := make(map[string]Value)
defer delete(map)
```

### Key Decisions

1. **Use `Maybe(T)` instead of `?T`** - Odin's optional type
2. **Use `or_return` instead of `try`** - Odin's error propagation
3. **Use `context.allocator`** - Implicit allocator from context
4. **Use `#partial switch`** - For union type checking
5. **Use `transmute`** - For zero-cost type conversions

### Reference Zig Files

When implementing, reference these Zig files:
- `src/dynamodb/json.zig` - 400 lines, DynamoDB JSON format
- `src/dynamodb/storage.zig` - 460 lines, storage engine
- `src/dynamodb/handler.zig` - 500+ lines, request handlers
- `src/item_codec.zig` - 350 lines, TLV encoding
- `src/http.zig` - 250 lines, HTTP server

### Quick Test Commands

```bash
# Build and test
make build
make test

# Run server
make run

# Test with AWS CLI
aws dynamodb list-tables --endpoint-url http://localhost:8002
```