# Project: jormun-db
# Generated: Sun Feb 15 11:44:33 AM EST 2026

================================================================================

================================================================================
FILE: ./ARCHITECTURE.md
================================================================================

## JormunDB Architecture

This document explains the internal architecture of JormunDB, including design decisions, storage formats, and the arena-per-request memory management pattern.

## Table of Contents

- [Overview](#overview)
- [Why Odin?](#why-odin)
- [Memory Management](#memory-management)
- [Storage Format](#storage-format)
- [Module Structure](#module-structure)
- [Request Flow](#request-flow)
- [Concurrency Model](#concurrency-model)

## Overview

JormunDB is a DynamoDB-compatible database server that speaks the DynamoDB wire protocol. It uses RocksDB for persistent storage and is written in Odin for elegant memory management.

### Key Design Goals

1. **Zero allocation ceremony** - No explicit `defer free()` or error handling for every allocation
2. **Binary storage** - Efficient TLV encoding instead of JSON
3. **API compatibility** - Drop-in replacement for DynamoDB Local
4. **Performance** - RocksDB-backed with efficient key encoding

## Why Odin?

The original implementation in Zig suffered from explicit allocator threading:

```zig
// Zig version - explicit allocator everywhere
fn handleRequest(allocator: std.mem.Allocator, request: []const u8) !Response {
    const parsed = try parseJson(allocator, request);
    defer parsed.deinit(allocator);
    
    const item = try storage.getItem(allocator, parsed.table_name, parsed.key);
    defer if (item) |i| freeItem(allocator, i);
    
    const response = try serializeResponse(allocator, item);
    defer allocator.free(response);
    
    return response; // Wait, we deferred the free!
}
```

Odin's context allocator system eliminates this:

```odin
// Odin version - implicit context allocator
handle_request :: proc(request: []byte) -> Response {
    // All allocations use context.allocator automatically
    parsed := parse_json(request)
    item := storage_get_item(parsed.table_name, parsed.key)
    response := serialize_response(item)
    
    return response
    // Everything freed when arena is destroyed
}
```

## Memory Management

JormunDB uses a two-allocator strategy:

### 1. Arena Allocator (Request-Scoped)

Every HTTP request gets its own arena:

```odin
handle_connection :: proc(conn: net.TCP_Socket) {
    // Create arena for this request (4MB)
    arena: mem.Arena
    mem.arena_init(&arena, make([]byte, mem.Megabyte * 4))
    defer mem.arena_destroy(&arena)
    
    // Set context allocator
    context.allocator = mem.arena_allocator(&arena)
    
    // All downstream code uses context.allocator
    request := parse_http_request(conn)    // uses arena
    response := handle_request(request)     // uses arena
    send_response(conn, response)           // uses arena
    
    // Arena is freed here - everything cleaned up automatically
}
```

**Benefits:**
- No individual `free()` calls needed
- No `errdefer` cleanup
- No use-after-free bugs
- No memory leaks from forgotten frees
- Predictable performance (no GC pauses)

### 2. Default Allocator (Long-Lived Data)

The default allocator (typically `context.allocator` at program start) is used for:

- Table metadata
- Table locks (sync.RW_Mutex)
- Engine state
- Items returned from storage layer (copied to request arena when needed)

## Storage Format

### Binary Keys (Varint-Prefixed Segments)

All keys use varint length prefixes for space efficiency:

```
Meta key:  [0x01][len][table_name]
Data key:  [0x02][len][table_name][len][pk_value][len][sk_value]?
GSI key:   [0x03][len][table_name][len][index_name][len][gsi_pk][len][gsi_sk]?
LSI key:   [0x04][len][table_name][len][index_name][len][pk][len][lsi_sk]
```

**Example Data Key:**
```
Table: "Users"
PK: "user:123"
SK: "profile"

Encoded:
[0x02]          // Entity type (Data)
[0x05]          // Table name length (5)
Users           // Table name bytes
[0x08]          // PK length (8)
user:123        // PK bytes
[0x07]          // SK length (7)
profile         // SK bytes
```

### Item Encoding (TLV Format)

Items use Tag-Length-Value encoding for space efficiency:

```
Format:
[attr_count:varint]
  [name_len:varint][name:bytes][type_tag:u8][value_len:varint][value:bytes]...

Type Tags:
  String  = 0x01    Number = 0x02    Binary = 0x03
  Bool    = 0x04    Null   = 0x05
  SS      = 0x10    NS     = 0x11    BS     = 0x12
  List    = 0x20    Map    = 0x21
```

**Example Item:**
```json
{
  "id": {"S": "user123"},
  "age": {"N": "30"}
}
```

Encoded as:
```
[0x02]              // 2 attributes
  [0x02]            // name length (2)
  id                // name bytes
  [0x01]            // type tag (String)
  [0x07]            // value length (7)
  user123           // value bytes
  
  [0x03]            // name length (3)
  age               // name bytes
  [0x02]            // type tag (Number)
  [0x02]            // value length (2)
  30                // value bytes (stored as string)
```

## Module Structure

```
jormundb/
├── main.odin           # Entry point, HTTP server
├── rocksdb/            # RocksDB C FFI bindings
│   └── rocksdb.odin    # db_open, db_put, db_get, etc.
├── dynamodb/           # DynamoDB protocol implementation
│   ├── types.odin      # Core types (Attribute_Value, Item, Key, etc.)
│   ├── json.odin       # DynamoDB JSON parsing/serialization
│   ├── storage.odin    # Storage engine (CRUD, scan, query)
│   └── handler.odin    # HTTP request handlers
├── key_codec/          # Binary key encoding
│   └── key_codec.odin  # build_data_key, decode_data_key, etc.
└── item_codec/         # Binary TLV item encoding
    └── item_codec.odin # encode, decode
```

## Request Flow

```
1. HTTP POST / arrives
   ↓
2. Create arena allocator (4MB)
   Set context.allocator = arena_allocator
   ↓
3. Parse HTTP headers
   Extract X-Amz-Target → Operation
   ↓
4. Parse JSON body
   Convert DynamoDB JSON → internal types
   ↓
5. Route to handler (e.g., handle_put_item)
   ↓
6. Storage engine operation
   - Build binary key
   - Encode item to TLV
   - RocksDB put/get/delete
   ↓
7. Build response
   - Serialize item to DynamoDB JSON
   - Format HTTP response
   ↓
8. Send response
   ↓
9. Destroy arena
   All request memory freed automatically
```

## Concurrency Model

### Table-Level RW Locks

Each table has a reader-writer lock:

```odin
Storage_Engine :: struct {
    db:                 rocksdb.DB,
    table_locks:        map[string]^sync.RW_Mutex,
    table_locks_mutex:  sync.Mutex,
}
```

**Read Operations** (GetItem, Query, Scan):
- Acquire shared lock
- Multiple readers can run concurrently
- Writers are blocked

**Write Operations** (PutItem, DeleteItem, UpdateItem):
- Acquire exclusive lock
- Only one writer at a time
- All readers are blocked

### Thread Safety

- RocksDB handles are thread-safe (column family-based)
- Table metadata is protected by locks
- Request arenas are thread-local (no sharing)

## Error Handling

Odin uses explicit error returns via `or_return`:

```odin
// Odin error handling
parse_json :: proc(data: []byte) -> (Item, bool) {
    parsed := json.parse(data) or_return
    item := json_to_item(parsed) or_return
    return item, true
}

// Usage
item := parse_json(request.body) or_else {
    return error_response(.ValidationException, "Invalid JSON")
}
```

No exceptions, no panic-recover patterns. Every error path is explicit.

## DynamoDB Wire Protocol

### Request Format

```
POST / HTTP/1.1
X-Amz-Target: DynamoDB_20120810.PutItem
Content-Type: application/x-amz-json-1.0

{
  "TableName": "Users",
  "Item": {
    "id": {"S": "user123"},
    "name": {"S": "Alice"}
  }
}
```

### Response Format

```
HTTP/1.1 200 OK
Content-Type: application/x-amz-json-1.0
x-amzn-RequestId: local-request-id

{}
```

### Error Format

```json
{
  "__type": "com.amazonaws.dynamodb.v20120810#ResourceNotFoundException",
  "message": "Table not found"
}
```

## Performance Characteristics

### Time Complexity

| Operation | Complexity | Notes |
|-----------|-----------|-------|
| PutItem | O(log n) | RocksDB LSM tree insert |
| GetItem | O(log n) | RocksDB point lookup |
| DeleteItem | O(log n) | RocksDB deletion |
| Query | O(log n + m) | n = items in table, m = result set |
| Scan | O(n) | Full table scan |

### Space Complexity

- Binary keys: ~20-100 bytes (vs 50-200 bytes JSON)
- Binary items: ~30% smaller than JSON
- Varint encoding saves space on small integers

### Benchmarks (Expected)

Based on Zig version performance:

```
Operation          Throughput      Latency (p50)
PutItem            ~5,000/sec      ~0.2ms
GetItem            ~7,000/sec      ~0.14ms
Query (1 item)     ~8,000/sec      ~0.12ms
Scan (1000 items)  ~20/sec         ~50ms
```

## Future Enhancements

### Planned Features

1. **UpdateExpression** - SET/REMOVE/ADD/DELETE operations
2. **FilterExpression** - Post-query filtering
3. **ProjectionExpression** - Return subset of attributes
4. **Global Secondary Indexes** - Query by non-key attributes
5. **Local Secondary Indexes** - Alternate sort keys
6. **BatchWriteItem** - Batch mutations
7. **BatchGetItem** - Batch reads
8. **Transactions** - ACID multi-item operations

### Optimization Opportunities

1. **Connection pooling** - Reuse HTTP connections
2. **Bloom filters** - Faster negative lookups
3. **Compression** - LZ4/Zstd on large items
4. **Caching layer** - Hot item cache
5. **Parallel scan** - Segment-based scanning

## Debugging

### Enable Verbose Logging

```bash
make run VERBOSE=1
```

### Inspect RocksDB

```bash
# Use ldb tool to inspect database
ldb --db=./data scan
ldb --db=./data get <key_hex>
```

### Memory Profiling

Odin's tracking allocator can detect leaks:

```odin
when ODIN_DEBUG {
    track: mem.Tracking_Allocator
    mem.tracking_allocator_init(&track, context.allocator)
    context.allocator = mem.tracking_allocator(&track)
    
    defer {
        for _, leak in track.allocation_map {
            fmt.printfln("Leaked %d bytes at %p", leak.size, leak.location)
        }
    }
}
```

## Migration from Zig Version

The Zig version (ZynamoDB) used the same binary storage format, so existing RocksDB databases can be read by JormunDB without migration.

### Compatibility

- ✅ Binary key format (byte-compatible)
- ✅ Binary item format (byte-compatible)
- ✅ Table metadata (JSON, compatible)
- ✅ HTTP wire protocol (identical)

### Breaking Changes

None - JormunDB can open ZynamoDB databases directly.

---

## Contributing

When contributing to JormunDB:

1. **Use the context allocator** - All request-scoped allocations should use `context.allocator`
2. **Avoid manual frees** - Let the arena handle it
3. **Long-lived data** - Use the default allocator explicitly
4. **Test thoroughly** - Run `make test` before committing
5. **Format code** - Run `make fmt` before committing

## References

- [Odin Language](https://odin-lang.org/)
- [RocksDB Wiki](https://github.com/facebook/rocksdb/wiki)
- [DynamoDB API Reference](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/)
- [Varint Encoding](https://developers.google.com/protocol-buffers/docs/encoding#varints)


================================================================================
FILE: ./dynamodb/types.odin
================================================================================

package dynamodb

import "core:fmt"
import "core:strings"

// DynamoDB AttributeValue - the core data type
Attribute_Value :: union {
	String,        // S
	Number,        // N (stored as string)
	Binary,        // B (base64)
	Bool,          // BOOL
	Null,          // NULL
	String_Set,    // SS
	Number_Set,    // NS
	Binary_Set,    // BS
	List,          // L
	Map,           // M
}

String :: distinct string
Number :: distinct string
Binary :: distinct string
Bool :: distinct bool
Null :: distinct bool

String_Set :: distinct []string
Number_Set :: distinct []string
Binary_Set :: distinct []string
List :: distinct []Attribute_Value
Map :: distinct map[string]Attribute_Value

// Item is a map of attribute names to values
Item :: map[string]Attribute_Value

// Key represents a DynamoDB key (partition key + optional sort key)
Key :: struct {
	pk: Attribute_Value,
	sk: Maybe(Attribute_Value),
}

// Free a key
key_destroy :: proc(key: ^Key) {
	attr_value_destroy(&key.pk)
	if sk, ok := key.sk.?; ok {
		sk_copy := sk
		attr_value_destroy(&sk_copy)
	}
}

// Extract key from item based on key schema
key_from_item :: proc(item: Item, key_schema: []Key_Schema_Element) -> (Key, bool) {
	pk_value: Attribute_Value
	sk_value: Maybe(Attribute_Value)
	
	for schema_elem in key_schema {
		attr, ok := item[schema_elem.attribute_name]
		if !ok {
			return {}, false
		}
		
		// Validate that key is a scalar type (S, N, or B)
		#partial switch _ in attr {
		case String, Number, Binary:
			// Valid key type
		case:
			return {}, false
		}
		
		// Deep copy the attribute value
		copied := attr_value_deep_copy(attr)
		
		switch schema_elem.key_type {
		case .HASH:
			pk_value = copied
		case .RANGE:
			sk_value = copied
		}
	}
	
	return Key{pk = pk_value, sk = sk_value}, true
}

// Convert key to item
key_to_item :: proc(key: Key, key_schema: []Key_Schema_Element) -> Item {
	item := make(Item)
	
	for schema_elem in key_schema {
		attr_value: Attribute_Value
		
		switch schema_elem.key_type {
		case .HASH:
			attr_value = key.pk
		case .RANGE:
			if sk, ok := key.sk.?; ok {
				attr_value = sk
			} else {
				continue
			}
		}
		
		item[schema_elem.attribute_name] = attr_value_deep_copy(attr_value)
	}
	
	return item
}

// Extract raw byte values from key
Key_Values :: struct {
	pk: []byte,
	sk: Maybe([]byte),
}

key_get_values :: proc(key: ^Key) -> (Key_Values, bool) {
	pk_bytes: []byte
	
	switch v in key.pk {
	case String:
		pk_bytes = transmute([]byte)string(v)
	case Number:
		pk_bytes = transmute([]byte)string(v)
	case Binary:
		pk_bytes = transmute([]byte)string(v)
	case:
		return {}, false
	}
	
	sk_bytes: Maybe([]byte)
	if sk, ok := key.sk.?; ok {
		switch v in sk {
		case String:
			sk_bytes = transmute([]byte)string(v)
		case Number:
			sk_bytes = transmute([]byte)string(v)
		case Binary:
			sk_bytes = transmute([]byte)string(v)
		case:
			return {}, false
		}
	}
	
	return Key_Values{pk = pk_bytes, sk = sk_bytes}, true
}

// Key type
Key_Type :: enum {
	HASH,
	RANGE,
}

key_type_to_string :: proc(kt: Key_Type) -> string {
	switch kt {
	case .HASH:  return "HASH"
	case .RANGE: return "RANGE"
	}
	return "HASH"
}

key_type_from_string :: proc(s: string) -> (Key_Type, bool) {
	switch s {
	case "HASH":  return .HASH, true
	case "RANGE": return .RANGE, true
	}
	return .HASH, false
}

// Scalar attribute type
Scalar_Attribute_Type :: enum {
	S, // String
	N, // Number
	B, // Binary
}

scalar_type_to_string :: proc(t: Scalar_Attribute_Type) -> string {
	switch t {
	case .S: return "S"
	case .N: return "N"
	case .B: return "B"
	}
	return "S"
}

scalar_type_from_string :: proc(s: string) -> (Scalar_Attribute_Type, bool) {
	switch s {
	case "S": return .S, true
	case "N": return .N, true
	case "B": return .B, true
	}
	return .S, false
}

// Key schema element
Key_Schema_Element :: struct {
	attribute_name: string,
	key_type:       Key_Type,
}

// Attribute definition
Attribute_Definition :: struct {
	attribute_name: string,
	attribute_type: Scalar_Attribute_Type,
}

// Projection type for indexes
Projection_Type :: enum {
	ALL,
	KEYS_ONLY,
	INCLUDE,
}

// Projection
Projection :: struct {
	projection_type:     Projection_Type,
	non_key_attributes:  Maybe([]string),
}

// Global secondary index
Global_Secondary_Index :: struct {
	index_name:  string,
	key_schema:  []Key_Schema_Element,
	projection:  Projection,
}

// Local secondary index
Local_Secondary_Index :: struct {
	index_name:  string,
	key_schema:  []Key_Schema_Element,
	projection:  Projection,
}

// Table status
Table_Status :: enum {
	CREATING,
	UPDATING,
	DELETING,
	ACTIVE,
	INACCESSIBLE_ENCRYPTION_CREDENTIALS,
	ARCHIVING,
	ARCHIVED,
}

table_status_to_string :: proc(status: Table_Status) -> string {
	switch status {
	case .CREATING: return "CREATING"
	case .UPDATING: return "UPDATING"
	case .DELETING: return "DELETING"
	case .ACTIVE:   return "ACTIVE"
	case .INACCESSIBLE_ENCRYPTION_CREDENTIALS: return "INACCESSIBLE_ENCRYPTION_CREDENTIALS"
	case .ARCHIVING: return "ARCHIVING"
	case .ARCHIVED:  return "ARCHIVED"
	}
	return "ACTIVE"
}

// Table description
Table_Description :: struct {
	table_name:                 string,
	key_schema:                 []Key_Schema_Element,
	attribute_definitions:      []Attribute_Definition,
	table_status:               Table_Status,
	creation_date_time:         i64,
	item_count:                 u64,
	table_size_bytes:           u64,
	global_secondary_indexes:   Maybe([]Global_Secondary_Index),
	local_secondary_indexes:    Maybe([]Local_Secondary_Index),
}

// DynamoDB operation types
Operation :: enum {
	CreateTable,
	DeleteTable,
	DescribeTable,
	ListTables,
	UpdateTable,
	PutItem,
	GetItem,
	DeleteItem,
	UpdateItem,
	Query,
	Scan,
	BatchGetItem,
	BatchWriteItem,
	TransactGetItems,
	TransactWriteItems,
	Unknown,
}

operation_from_target :: proc(target: string) -> Operation {
	prefix :: "DynamoDB_20120810."
	if !strings.has_prefix(target, prefix) {
		return .Unknown
	}
	
	op_name := target[len(prefix):]
	
	switch op_name {
	case "CreateTable":      return .CreateTable
	case "DeleteTable":      return .DeleteTable
	case "DescribeTable":    return .DescribeTable
	case "ListTables":       return .ListTables
	case "UpdateTable":      return .UpdateTable
	case "PutItem":          return .PutItem
	case "GetItem":          return .GetItem
	case "DeleteItem":       return .DeleteItem
	case "UpdateItem":       return .UpdateItem
	case "Query":            return .Query
	case "Scan":             return .Scan
	case "BatchGetItem":     return .BatchGetItem
	case "BatchWriteItem":   return .BatchWriteItem
	case "TransactGetItems": return .TransactGetItems
	case "TransactWriteItems": return .TransactWriteItems
	}
	
	return .Unknown
}

// DynamoDB error types
DynamoDB_Error_Type :: enum {
	ValidationException,
	ResourceNotFoundException,
	ResourceInUseException,
	ConditionalCheckFailedException,
	ProvisionedThroughputExceededException,
	ItemCollectionSizeLimitExceededException,
	InternalServerError,
	SerializationException,
}

error_to_response :: proc(err_type: DynamoDB_Error_Type, message: string) -> string {
	type_str: string
	
	switch err_type {
	case .ValidationException:
		type_str = "com.amazonaws.dynamodb.v20120810#ValidationException"
	case .ResourceNotFoundException:
		type_str = "com.amazonaws.dynamodb.v20120810#ResourceNotFoundException"
	case .ResourceInUseException:
		type_str = "com.amazonaws.dynamodb.v20120810#ResourceInUseException"
	case .ConditionalCheckFailedException:
		type_str = "com.amazonaws.dynamodb.v20120810#ConditionalCheckFailedException"
	case .ProvisionedThroughputExceededException:
		type_str = "com.amazonaws.dynamodb.v20120810#ProvisionedThroughputExceededException"
	case .ItemCollectionSizeLimitExceededException:
		type_str = "com.amazonaws.dynamodb.v20120810#ItemCollectionSizeLimitExceededException"
	case .InternalServerError:
		type_str = "com.amazonaws.dynamodb.v20120810#InternalServerError"
	case .SerializationException:
		type_str = "com.amazonaws.dynamodb.v20120810#SerializationException"
	}
	
	return fmt.aprintf(`{{"__type":"%s","message":"%s"}}`, type_str, message)
}

// Deep copy an attribute value
attr_value_deep_copy :: proc(attr: Attribute_Value) -> Attribute_Value {
	switch v in attr {
	case String:
		return String(strings.clone(string(v)))
	case Number:
		return Number(strings.clone(string(v)))
	case Binary:
		return Binary(strings.clone(string(v)))
	case Bool:
		return v
	case Null:
		return v
	case String_Set:
		ss := make([]string, len(v))
		for s, i in v {
			ss[i] = strings.clone(s)
		}
		return String_Set(ss)
	case Number_Set:
		ns := make([]string, len(v))
		for n, i in v {
			ns[i] = strings.clone(n)
		}
		return Number_Set(ns)
	case Binary_Set:
		bs := make([]string, len(v))
		for b, i in v {
			bs[i] = strings.clone(b)
		}
		return Binary_Set(bs)
	case List:
		list := make([]Attribute_Value, len(v))
		for item, i in v {
			list[i] = attr_value_deep_copy(item)
		}
		return List(list)
	case Map:
		m := make(map[string]Attribute_Value)
		for key, val in v {
			m[strings.clone(key)] = attr_value_deep_copy(val)
		}
		return Map(m)
	}
	return nil
}

// Free an attribute value
attr_value_destroy :: proc(attr: ^Attribute_Value) {
	switch v in attr {
	case String:
		delete(string(v))
	case Number:
		delete(string(v))
	case Binary:
		delete(string(v))
	case String_Set:
		for s in v {
			delete(s)
		}
		delete([]string(v))
	case Number_Set:
		for n in v {
			delete(n)
		}
		delete([]string(v))
	case Binary_Set:
		for b in v {
			delete(b)
		}
		delete([]string(v))
	case List:
		for item in v {
			item_copy := item
			attr_value_destroy(&item_copy)
		}
		delete([]Attribute_Value(v))
	case Map:
		for key, val in v {
			delete(key)
			val_copy := val
			attr_value_destroy(&val_copy)
		}
		delete(map[string]Attribute_Value(v))
	case Bool, Null:
		// Nothing to free
	}
}

// Free an item
item_destroy :: proc(item: ^Item) {
	for key, val in item {
		delete(key)
		val_copy := val
		attr_value_destroy(&val_copy)
	}
	delete(item^)
}


================================================================================
FILE: ./http.odin
================================================================================

package main

import "core:fmt"
import "core:mem"
import vmem "core:mem/virtual"
import "core:net"
import "core:strings"
import "core:strconv"

// HTTP Method enumeration
HTTP_Method :: enum {
	GET,
	POST,
	PUT,
	DELETE,
	OPTIONS,
	HEAD,
	PATCH,
}

method_from_string :: proc(s: string) -> HTTP_Method {
	switch s {
	case "GET":     return .GET
	case "POST":    return .POST
	case "PUT":     return .PUT
	case "DELETE":  return .DELETE
	case "OPTIONS": return .OPTIONS
	case "HEAD":    return .HEAD
	case "PATCH":   return .PATCH
	}
	return .GET
}

// HTTP Status codes
HTTP_Status :: enum u16 {
	OK                    = 200,
	Created               = 201,
	No_Content            = 204,
	Bad_Request           = 400,
	Unauthorized          = 401,
	Forbidden             = 403,
	Not_Found             = 404,
	Method_Not_Allowed    = 405,
	Conflict              = 409,
	Payload_Too_Large     = 413,
	Internal_Server_Error = 500,
	Service_Unavailable   = 503,
}

// HTTP Header
HTTP_Header :: struct {
	name:  string,
	value: string,
}

// HTTP Request
HTTP_Request :: struct {
	method:  HTTP_Method,
	path:    string,
	headers: []HTTP_Header,
	body:    []byte,
}

// Get header value by name (case-insensitive)
request_get_header :: proc(req: ^HTTP_Request, name: string) -> Maybe(string) {
	for header in req.headers {
		if strings.equal_fold(header.name, name) {
			return header.value
		}
	}
	return nil
}

// HTTP Response
HTTP_Response :: struct {
	status:  HTTP_Status,
	headers: [dynamic]HTTP_Header,
	body:    [dynamic]byte,
}

response_init :: proc(allocator: mem.Allocator) -> HTTP_Response {
	return HTTP_Response{
		status  = .OK,
		headers = make([dynamic]HTTP_Header, allocator),
		body    = make([dynamic]byte, allocator),
	}
}

response_set_status :: proc(resp: ^HTTP_Response, status: HTTP_Status) {
	resp.status = status
}

response_add_header :: proc(resp: ^HTTP_Response, name: string, value: string) {
	append(&resp.headers, HTTP_Header{name = name, value = value})
}

response_set_body :: proc(resp: ^HTTP_Response, data: []byte) {
	clear(&resp.body)
	append(&resp.body, ..data)
}

// Request handler function type
// Takes context pointer, request, and request-scoped allocator
Request_Handler :: #type proc(ctx: rawptr, request: ^HTTP_Request, request_alloc: mem.Allocator) -> HTTP_Response

// Server configuration
Server_Config :: struct {
	max_body_size:              int,  // default 100MB
	max_headers:                int,  // default 100
	read_buffer_size:           int,  // default 8KB
	enable_keep_alive:          bool, // default true
	max_requests_per_connection: int, // default 1000
}

default_server_config :: proc() -> Server_Config {
	return Server_Config{
		max_body_size              = 100 * 1024 * 1024,
		max_headers                = 100,
		read_buffer_size           = 8 * 1024,
		enable_keep_alive          = true,
		max_requests_per_connection = 1000,
	}
}

// Server
Server :: struct {
	allocator:   mem.Allocator,
	endpoint:    net.Endpoint,
	handler:     Request_Handler,
	handler_ctx: rawptr,
	config:      Server_Config,
	running:     bool,
	socket:      Maybe(net.TCP_Socket),
}

server_init :: proc(
	allocator: mem.Allocator,
	host: string,
	port: int,
	handler: Request_Handler,
	handler_ctx: rawptr,
	config: Server_Config,
) -> (Server, bool) {
	endpoint, endpoint_ok := net.parse_endpoint(fmt.tprintf("%s:%d", host, port))
	if !endpoint_ok {
		return {}, false
	}

	return Server{
		allocator   = allocator,
		endpoint    = endpoint,
		handler     = handler,
		handler_ctx = handler_ctx,
		config      = config,
		running     = false,
		socket      = nil,
	}, true
}

server_start :: proc(server: ^Server) -> bool {
	// Create listening socket
	socket, socket_err := net.listen_tcp(server.endpoint)
	if socket_err != nil {
		fmt.eprintfln("Failed to create listening socket: %v", socket_err)
		return false
	}

	server.socket = socket
	server.running = true

	fmt.printfln("HTTP server listening on %v", server.endpoint)

	// Accept loop
	for server.running {
		conn, source, accept_err := net.accept_tcp(socket)
		if accept_err != nil {
			if server.running {
				fmt.eprintfln("Accept error: %v", accept_err)
			}
			continue
		}

		// Handle connection in separate goroutine would go here
		// For now, handle synchronously (should spawn thread)
		handle_connection(server, conn, source)
	}

	return true
}

server_stop :: proc(server: ^Server) {
	server.running = false
	if sock, ok := server.socket.?; ok {
		net.close(sock)
		server.socket = nil
	}
}

// Handle a single connection
handle_connection :: proc(server: ^Server, conn: net.TCP_Socket, source: net.Endpoint) {
	defer net.close(conn)

	request_count := 0
	for request_count < server.config.max_requests_per_connection {
		request_count += 1

		// Growing arena for this request
		arena: vmem.Arena
		arena_err := vmem.arena_init_growing(&arena)
		if arena_err != .None {
			break
		}
		defer vmem.arena_destroy(&arena)

		request_alloc := vmem.arena_allocator(&arena)

		// TODO: Double check if we want *all* downstream allocations to use the request arena?
		old := context.allocator
		context.allocator = request_alloc
		defer context.allocator = old

		request, parse_ok := parse_request(conn, request_alloc, server.config)
		if !parse_ok {
			break
		}

		response := server.handler(server.handler_ctx, &request, request_alloc)

		send_ok := send_response(conn, &response, request_alloc)
		if !send_ok {
			break
		}

		// Check keep-alive
		keep_alive := request_get_header(&request, "Connection")
		if ka, ok := keep_alive.?; ok {
			if !strings.equal_fold(ka, "keep-alive") {
				break
			}
		} else if !server.config.enable_keep_alive {
			break
		}

		// Arena is automatically freed here
	}
}

// Parse HTTP request
parse_request :: proc(
	conn: net.TCP_Socket,
	allocator: mem.Allocator,
	config: Server_Config,
) -> (HTTP_Request, bool) {
	// Read request line and headers
	buffer := make([]byte, config.read_buffer_size, allocator)

	bytes_read, read_err := net.recv_tcp(conn, buffer)
	if read_err != nil || bytes_read == 0 {
		return {}, false
	}

	request_data := buffer[:bytes_read]

	// Find end of headers (\r\n\r\n)
	header_end_idx := strings.index(string(request_data), "\r\n\r\n")
	if header_end_idx < 0 {
		return {}, false
	}

	header_section := string(request_data[:header_end_idx])
	body_start := header_end_idx + 4

	// Parse request line
	lines := strings.split_lines(header_section, allocator)
	if len(lines) == 0 {
		return {}, false
	}

	request_line := lines[0]
	parts := strings.split(request_line, " ", allocator)
	if len(parts) < 3 {
		return {}, false
	}

	method := method_from_string(parts[0])
	path := strings.clone(parts[1], allocator)

	// Parse headers
	headers := make([dynamic]HTTP_Header, allocator)
	for i := 1; i < len(lines); i += 1 {
		line := lines[i]
		if len(line) == 0 {
			continue
		}

		colon_idx := strings.index(line, ":")
		if colon_idx < 0 {
			continue
		}

		name := strings.trim_space(line[:colon_idx])
		value := strings.trim_space(line[colon_idx+1:])

		append(&headers, HTTP_Header{
			name  = strings.clone(name, allocator),
			value = strings.clone(value, allocator),
		})
	}

	// Read body if Content-Length present
	body: []byte

	content_length_header := request_get_header_from_slice(headers[:], "Content-Length")
	if cl, ok := content_length_header.?; ok {
		content_length := strconv.parse_int(cl) or_else 0

		if content_length > 0 && content_length <= config.max_body_size {
			// Check if we already have the body in buffer
			existing_body := request_data[body_start:]

			if len(existing_body) >= content_length {
				// Body already in buffer
				body = make([]byte, content_length, allocator)
				copy(body, existing_body[:content_length])
			} else {
				// Need to read more
				body = make([]byte, content_length, allocator)
				copy(body, existing_body)

				remaining := content_length - len(existing_body)
				body_written := len(existing_body)

				for remaining > 0 {
					chunk_size := min(remaining, config.read_buffer_size)
					chunk := make([]byte, chunk_size, allocator)

					n, err := net.recv_tcp(conn, chunk)
					if err != nil || n == 0 {
						return {}, false
					}

					copy(body[body_written:], chunk[:n])
					body_written += n
					remaining -= n
				}
			}
		}
	}

	return HTTP_Request{
		method  = method,
		path    = path,
		headers = headers[:],
		body    = body,
	}, true
}

// Helper to get header from slice
request_get_header_from_slice :: proc(headers: []HTTP_Header, name: string) -> Maybe(string) {
	for header in headers {
		if strings.equal_fold(header.name, name) {
			return header.value
		}
	}
	return nil
}

// Send HTTP response
send_response :: proc(conn: net.TCP_Socket, resp: ^HTTP_Response, allocator: mem.Allocator) -> bool {
	// Build response string
	builder := strings.builder_make(allocator)
	defer strings.builder_destroy(&builder)

	// Status line
	strings.write_string(&builder, "HTTP/1.1 ")
	strings.write_int(&builder, int(resp.status))
	strings.write_string(&builder, " ")
	strings.write_string(&builder, status_text(resp.status))
	strings.write_string(&builder, "\r\n")

	// Headers
	response_add_header(resp, "Content-Length", fmt.tprintf("%d", len(resp.body)))

	for header in resp.headers {
		strings.write_string(&builder, header.name)
		strings.write_string(&builder, ": ")
		strings.write_string(&builder, header.value)
		strings.write_string(&builder, "\r\n")
	}

	// End of headers
	strings.write_string(&builder, "\r\n")

	// Send headers
	header_bytes := transmute([]byte)strings.to_string(builder)
	_, send_err := net.send_tcp(conn, header_bytes)
	if send_err != nil {
		return false
	}

	// Send body
	if len(resp.body) > 0 {
		_, send_err = net.send_tcp(conn, resp.body[:])
		if send_err != nil {
			return false
		}
	}

	return true
}

// Get status text for status code
status_text :: proc(status: HTTP_Status) -> string {
	switch status {
	case .OK:                    return "OK"
	case .Created:               return "Created"
	case .No_Content:            return "No Content"
	case .Bad_Request:           return "Bad Request"
	case .Unauthorized:          return "Unauthorized"
	case .Forbidden:             return "Forbidden"
	case .Not_Found:             return "Not Found"
	case .Method_Not_Allowed:    return "Method Not Allowed"
	case .Conflict:              return "Conflict"
	case .Payload_Too_Large:     return "Payload Too Large"
	case .Internal_Server_Error: return "Internal Server Error"
	case .Service_Unavailable:   return "Service Unavailable"
	}
	return "Unknown"
}


================================================================================
FILE: ./key_codec/key_codec.odin
================================================================================

package key_codec

import "core:bytes"
import "core:encoding/varint"
import "core:mem"

// Entity type prefix bytes for namespacing
Entity_Type :: enum u8 {
	Meta = 0x01,  // Table metadata
	Data = 0x02,  // Item data
	GSI  = 0x03,  // Global secondary index
	LSI  = 0x04,  // Local secondary index
}

// Encode a varint length prefix
encode_varint :: proc(buf: ^bytes.Buffer, value: int) {
	temp: [10]byte
	n := varint.encode_u64(temp[:], u64(value))
	bytes.buffer_write(buf, temp[:n])
}

// Decode a varint length prefix
decode_varint :: proc(data: []byte, offset: ^int) -> (value: int, ok: bool) {
	if offset^ >= len(data) {
		return 0, false
	}
	
	val, n := varint.decode_u64(data[offset^:])
	if n <= 0 {
		return 0, false
	}
	
	offset^ += n
	return int(val), true
}

// Build metadata key: [meta][table_name]
build_meta_key :: proc(table_name: string) -> []byte {
	buf: bytes.Buffer
	bytes.buffer_init_allocator(&buf, 0, 256, context.allocator)
	
	// Write entity type
	bytes.buffer_write_byte(&buf, u8(Entity_Type.Meta))
	
	// Write table name with length prefix
	encode_varint(&buf, len(table_name))
	bytes.buffer_write_string(&buf, table_name)
	
	return bytes.buffer_to_bytes(&buf)
}

// Build data key: [data][table_name][pk_value][sk_value?]
build_data_key :: proc(table_name: string, pk_value: []byte, sk_value: Maybe([]byte)) -> []byte {
	buf: bytes.Buffer
	bytes.buffer_init_allocator(&buf, 0, 512, context.allocator)
	
	// Write entity type
	bytes.buffer_write_byte(&buf, u8(Entity_Type.Data))
	
	// Write table name
	encode_varint(&buf, len(table_name))
	bytes.buffer_write_string(&buf, table_name)
	
	// Write partition key
	encode_varint(&buf, len(pk_value))
	bytes.buffer_write(&buf, pk_value)
	
	// Write sort key if present
	if sk, ok := sk_value.?; ok {
		encode_varint(&buf, len(sk))
		bytes.buffer_write(&buf, sk)
	}
	
	return bytes.buffer_to_bytes(&buf)
}

// Build table prefix for scanning: [data][table_name]
build_table_prefix :: proc(table_name: string) -> []byte {
	buf: bytes.Buffer
	bytes.buffer_init_allocator(&buf, 0, 256, context.allocator)
	
	// Write entity type
	bytes.buffer_write_byte(&buf, u8(Entity_Type.Data))
	
	// Write table name
	encode_varint(&buf, len(table_name))
	bytes.buffer_write_string(&buf, table_name)
	
	return bytes.buffer_to_bytes(&buf)
}

// Build partition prefix for querying: [data][table_name][pk_value]
build_partition_prefix :: proc(table_name: string, pk_value: []byte) -> []byte {
	buf: bytes.Buffer
	bytes.buffer_init_allocator(&buf, 0, 512, context.allocator)
	
	// Write entity type
	bytes.buffer_write_byte(&buf, u8(Entity_Type.Data))
	
	// Write table name
	encode_varint(&buf, len(table_name))
	bytes.buffer_write_string(&buf, table_name)
	
	// Write partition key
	encode_varint(&buf, len(pk_value))
	bytes.buffer_write(&buf, pk_value)
	
	return bytes.buffer_to_bytes(&buf)
}

// Build GSI key: [gsi][table_name][index_name][gsi_pk][gsi_sk?]
build_gsi_key :: proc(table_name: string, index_name: string, gsi_pk: []byte, gsi_sk: Maybe([]byte)) -> []byte {
	buf: bytes.Buffer
	bytes.buffer_init_allocator(&buf, 0, 512, context.allocator)
	
	// Write entity type
	bytes.buffer_write_byte(&buf, u8(Entity_Type.GSI))
	
	// Write table name
	encode_varint(&buf, len(table_name))
	bytes.buffer_write_string(&buf, table_name)
	
	// Write index name
	encode_varint(&buf, len(index_name))
	bytes.buffer_write_string(&buf, index_name)
	
	// Write GSI partition key
	encode_varint(&buf, len(gsi_pk))
	bytes.buffer_write(&buf, gsi_pk)
	
	// Write GSI sort key if present
	if sk, ok := gsi_sk.?; ok {
		encode_varint(&buf, len(sk))
		bytes.buffer_write(&buf, sk)
	}
	
	return bytes.buffer_to_bytes(&buf)
}

// Build LSI key: [lsi][table_name][index_name][pk][lsi_sk]
build_lsi_key :: proc(table_name: string, index_name: string, pk: []byte, lsi_sk: []byte) -> []byte {
	buf: bytes.Buffer
	bytes.buffer_init_allocator(&buf, 0, 512, context.allocator)
	
	// Write entity type
	bytes.buffer_write_byte(&buf, u8(Entity_Type.LSI))
	
	// Write table name
	encode_varint(&buf, len(table_name))
	bytes.buffer_write_string(&buf, table_name)
	
	// Write index name
	encode_varint(&buf, len(index_name))
	bytes.buffer_write_string(&buf, index_name)
	
	// Write partition key
	encode_varint(&buf, len(pk))
	bytes.buffer_write(&buf, pk)
	
	// Write LSI sort key
	encode_varint(&buf, len(lsi_sk))
	bytes.buffer_write(&buf, lsi_sk)
	
	return bytes.buffer_to_bytes(&buf)
}

// Key decoder for reading binary keys
Key_Decoder :: struct {
	data: []byte,
	pos:  int,
}

decoder_init :: proc(data: []byte) -> Key_Decoder {
	return Key_Decoder{data = data, pos = 0}
}

decoder_read_entity_type :: proc(decoder: ^Key_Decoder) -> (Entity_Type, bool) {
	if decoder.pos >= len(decoder.data) {
		return .Meta, false
	}
	
	entity_type := Entity_Type(decoder.data[decoder.pos])
	decoder.pos += 1
	return entity_type, true
}

decoder_read_segment :: proc(decoder: ^Key_Decoder) -> (segment: []byte, ok: bool) {
	// Read length
	length := decode_varint(decoder.data, &decoder.pos) or_return
	
	// Read data
	if decoder.pos + length > len(decoder.data) {
		return nil, false
	}
	
	// Return slice (owned by caller via context.allocator)
	segment = make([]byte, length, context.allocator)
	copy(segment, decoder.data[decoder.pos:decoder.pos + length])
	decoder.pos += length
	
	return segment, true
}

decoder_read_segment_borrowed :: proc(decoder: ^Key_Decoder) -> (segment: []byte, ok: bool) {
	// Read length
	length := decode_varint(decoder.data, &decoder.pos) or_return
	
	// Return borrowed slice
	if decoder.pos + length > len(decoder.data) {
		return nil, false
	}
	
	segment = decoder.data[decoder.pos:decoder.pos + length]
	decoder.pos += length
	
	return segment, true
}

decoder_has_more :: proc(decoder: ^Key_Decoder) -> bool {
	return decoder.pos < len(decoder.data)
}

// Decode a data key back into components
Decoded_Data_Key :: struct {
	table_name: string,
	pk_value:   []byte,
	sk_value:   Maybe([]byte),
}

decode_data_key :: proc(key: []byte) -> (result: Decoded_Data_Key, ok: bool) {
	decoder := decoder_init(key)
	
	// Read and verify entity type
	entity_type := decoder_read_entity_type(&decoder) or_return
	if entity_type != .Data {
		return {}, false
	}
	
	// Read table name
	table_name_bytes := decoder_read_segment(&decoder) or_return
	result.table_name = string(table_name_bytes)
	
	// Read partition key
	result.pk_value = decoder_read_segment(&decoder) or_return
	
	// Read sort key if present
	if decoder_has_more(&decoder) {
		sk := decoder_read_segment(&decoder) or_return
		result.sk_value = sk
	}
	
	return result, true
}


================================================================================
FILE: ./main.odin
================================================================================

package main

import "core:fmt"
import "core:mem"
import "core:os"
import "core:strconv"
//import "core:strings" // I know we'll use in future but because we're not right now, compiler is complaining
import "rocksdb"

Config :: struct {
	host:     string,
	port:     int,
	data_dir: string,
	verbose:  bool,
}

main :: proc() {
	// Parse configuration
	config := parse_config()

	// Print banner
	print_banner(config)

	// Create data directory
	os.make_directory(config.data_dir)

	// Initialize storage engine
	db, err := rocksdb.db_open(config.data_dir, true)
	if err != .None {
		fmt.eprintln("Failed to initialize storage:", err)
		os.exit(1)
	}
	defer rocksdb.db_close(&db)

	fmt.printfln("Storage engine initialized at %s", config.data_dir)
	fmt.printfln("Starting DynamoDB-compatible server on %s:%d", config.host, config.port)

	// Create HTTP server
	server_config := default_server_config()

	// For now, use a simple echo handler until we implement the full DynamoDB handler
	server, server_ok := server_init(
		context.allocator,
		config.host,
		config.port,
		handle_http_request,
		&db,
		server_config,
	)

	if !server_ok {
		fmt.eprintln("Failed to initialize HTTP server")
		os.exit(1)
	}
	defer server_stop(&server)

	fmt.println("Ready to accept connections!")

	// Start server (blocks)
	if !server_start(&server) {
		fmt.eprintln("Server failed to start")
		os.exit(1)
	}
}

// Temporary HTTP request handler
// TODO: Replace with full DynamoDB handler once dynamodb/handler.odin is implemented
handle_http_request :: proc(ctx: rawptr, request: ^HTTP_Request, request_alloc: mem.Allocator) -> HTTP_Response {
	//db := cast(^rocksdb.DB)ctx // I know we'll use in future but because we're not right now, compiler is complaining

	response := response_init(request_alloc)
	response_add_header(&response, "Content-Type", "application/x-amz-json-1.0")
	response_add_header(&response, "x-amzn-RequestId", "local-request-id")

	// Get X-Amz-Target header
	target := request_get_header(request, "X-Amz-Target")

	if t, ok := target.?; ok {
		// Echo back the operation for now
		body := fmt.aprintf("{{\"operation\":\"%s\",\"status\":\"not_implemented\"}}", t)
		response_set_body(&response, transmute([]byte)body)
	} else {
		response_set_status(&response, .Bad_Request)
		response_set_body(&response, transmute([]byte)string("{\"error\":\"Missing X-Amz-Target header\"}"))
	}

	return response
}

parse_config :: proc() -> Config {
	config := Config{
		host = "0.0.0.0",
		port = 8002,
		data_dir = "./data",
		verbose = false,
	}

	// Environment variables
	if port_str, env_ok := os.lookup_env("JORMUN_PORT"); env_ok {
		if port, parse_ok := strconv.parse_int(port_str); parse_ok {
			config.port = port
		}
	}

	if host, ok := os.lookup_env("JORMUN_HOST"); ok {
		config.host = host
	}

	if data_dir, ok := os.lookup_env("JORMUN_DATA_DIR"); ok {
		config.data_dir = data_dir
	}

	if verbose, ok := os.lookup_env("JORMUN_VERBOSE"); ok {
		config.verbose = verbose == "1"
	}

	// TODO: Parse command line arguments

	return config
}

print_banner :: proc(config: Config) {
	banner := `
  ╔═══════════════════════════════════════════════╗
  ║                                               ║
  ║     ╦╔═╗╦═╗╔╦╗╦ ╦╔╗╔╔╦╗╔╗                      ║
  ║     ║║ ║╠╦╝║║║║ ║║║║ ║║╠╩╗                     ║
  ║    ╚╝╚═╝╩╚═╩ ╩╚═╝╝╚╝═╩╝╚═╝                     ║
  ║                                               ║
  ║   DynamoDB-Compatible Database                ║
  ║   Powered by RocksDB + Odin                   ║
  ║                                               ║
  ╚═══════════════════════════════════════════════╝
`
	fmt.println(banner)
	fmt.printfln("  Port: %d | Data Dir: %s\n", config.port, config.data_dir)
}


================================================================================
FILE: ./ols.json
================================================================================

{
    "$schema": "https://raw.githubusercontent.com/DanielGavin/ols/master/misc/ols.schema.json",
    "enable_document_symbols": true,
    "enable_hover": true,
    "enable_snippets": true
}

================================================================================
FILE: ./QUICKSTART.md
================================================================================

# JormunDB Quick Start Guide

Get JormunDB running in 5 minutes.

## Prerequisites

### 1. Install Odin

**macOS:**
```bash
# Using Homebrew
brew install odin

# Or download from https://odin-lang.org/docs/install/
```

**Ubuntu/Debian:**
```bash
# Download latest release
wget https://github.com/odin-lang/Odin/releases/latest/download/odin-ubuntu-amd64.tar.gz
tar -xzf odin-ubuntu-amd64.tar.gz
sudo mv odin /usr/local/bin/

# Verify
odin version
```

**From Source:**
```bash
git clone https://github.com/odin-lang/Odin
cd Odin
make
sudo cp odin /usr/local/bin/
```

### 2. Install RocksDB

**macOS:**
```bash
brew install rocksdb
```

**Ubuntu/Debian:**
```bash
sudo apt update
sudo apt install -y librocksdb-dev libsnappy-dev liblz4-dev libzstd-dev libbz2-dev
```

**Arch Linux:**
```bash
sudo pacman -S rocksdb
```

### 3. Verify Installation

```bash
# Check Odin
odin version

# Check RocksDB
pkg-config --libs rocksdb
# Should output: -lrocksdb -lstdc++ ...
```

## Building JormunDB

### Clone and Build

```bash
# Clone the repository
git clone https://github.com/yourusername/jormundb.git
cd jormundb

# Build debug version
make build

# Or build optimized release
make release
```

### Troubleshooting Build Issues

**"cannot find rocksdb"**
```bash
# Check RocksDB installation
pkg-config --cflags --libs rocksdb

# If not found, install RocksDB (see prerequisites)
```

**"odin: command not found"**
```bash
# Add Odin to PATH
export PATH=$PATH:/path/to/odin

# Or install system-wide (see prerequisites)
```

## Running the Server

### Basic Usage

```bash
# Run with defaults (localhost:8002, ./data directory)
make run
```

You should see:
```
  ╔═══════════════════════════════════════════════╗
  ║                                               ║
  ║     ╦╔═╗╦═╗╔╦╗╦ ╦╔╗╔╔╦╗╔╗                      ║
  ║     ║║ ║╠╦╝║║║║ ║║║║ ║║╠╩╗                     ║
  ║    ╚╝╚═╝╩╚═╩ ╩╚═╝╝╚╝═╩╝╚═╝                     ║
  ║                                               ║
  ║   DynamoDB-Compatible Database                ║
  ║   Powered by RocksDB + Odin                   ║
  ║                                               ║
  ╚═══════════════════════════════════════════════╝

  Port: 8002 | Data Dir: ./data

Storage engine initialized at ./data
Starting DynamoDB-compatible server on 0.0.0.0:8002
Ready to accept connections!
```

### Custom Configuration

```bash
# Custom port
make run PORT=9000

# Custom data directory
make run DATA_DIR=/tmp/jormun

# Enable verbose logging
make run VERBOSE=1

# Combine options
make run PORT=9000 DATA_DIR=/var/jormun VERBOSE=1
```

### Environment Variables

```bash
# Set via environment
export JORMUN_PORT=9000
export JORMUN_HOST=127.0.0.1
export JORMUN_DATA_DIR=/var/jormun
make run
```

## Testing with AWS CLI

### Install AWS CLI

**macOS:**
```bash
brew install awscli
```

**Ubuntu/Debian:**
```bash
sudo apt install awscli
```

**Verify:**
```bash
aws --version
```

### Configure AWS CLI (for local use)

```bash
# Set dummy credentials (required but not checked by JormunDB)
aws configure
# AWS Access Key ID: dummy
# AWS Secret Access Key: dummy
# Default region name: us-east-1
# Default output format: json
```

### Basic Operations

**Create a Table:**
```bash
aws dynamodb create-table \
    --endpoint-url http://localhost:8002 \
    --table-name Users \
    --key-schema \
        AttributeName=id,KeyType=HASH \
    --attribute-definitions \
        AttributeName=id,AttributeType=S \
    --billing-mode PAY_PER_REQUEST
```

**List Tables:**
```bash
aws dynamodb list-tables --endpoint-url http://localhost:8002
```

**Put an Item:**
```bash
aws dynamodb put-item \
    --endpoint-url http://localhost:8002 \
    --table-name Users \
    --item '{
        "id": {"S": "user123"},
        "name": {"S": "Alice"},
        "age": {"N": "30"},
        "email": {"S": "alice@example.com"}
    }'
```

**Get an Item:**
```bash
aws dynamodb get-item \
    --endpoint-url http://localhost:8002 \
    --table-name Users \
    --key '{"id": {"S": "user123"}}'
```

**Query Items:**
```bash
aws dynamodb query \
    --endpoint-url http://localhost:8002 \
    --table-name Users \
    --key-condition-expression "id = :id" \
    --expression-attribute-values '{
        ":id": {"S": "user123"}
    }'
```

**Scan Table:**
```bash
aws dynamodb scan \
    --endpoint-url http://localhost:8002 \
    --table-name Users
```

**Delete an Item:**
```bash
aws dynamodb delete-item \
    --endpoint-url http://localhost:8002 \
    --table-name Users \
    --key '{"id": {"S": "user123"}}'
```

**Delete a Table:**
```bash
aws dynamodb delete-table \
    --endpoint-url http://localhost:8002 \
    --table-name Users
```

## Testing with AWS SDK

### Node.js Example

```javascript
const { DynamoDBClient, PutItemCommand, GetItemCommand } = require("@aws-sdk/client-dynamodb");

const client = new DynamoDBClient({
    endpoint: "http://localhost:8002",
    region: "us-east-1",
    credentials: {
        accessKeyId: "dummy",
        secretAccessKey: "dummy"
    }
});

async function test() {
    // Put an item
    await client.send(new PutItemCommand({
        TableName: "Users",
        Item: {
            id: { S: "user123" },
            name: { S: "Alice" }
        }
    }));

    // Get the item
    const result = await client.send(new GetItemCommand({
        TableName: "Users",
        Key: { id: { S: "user123" } }
    }));

    console.log(result.Item);
}

test();
```

### Python Example

```python
import boto3

dynamodb = boto3.client(
    'dynamodb',
    endpoint_url='http://localhost:8002',
    region_name='us-east-1',
    aws_access_key_id='dummy',
    aws_secret_access_key='dummy'
)

# Put an item
dynamodb.put_item(
    TableName='Users',
    Item={
        'id': {'S': 'user123'},
        'name': {'S': 'Alice'}
    }
)

# Get the item
response = dynamodb.get_item(
    TableName='Users',
    Key={'id': {'S': 'user123'}}
)

print(response['Item'])
```

## Development Workflow

### Quick Rebuild

```bash
# Fast rebuild and run
make quick
```

### Clean Start

```bash
# Remove all build artifacts and data
make clean

# Build and run fresh
make dev
```

### Running Tests

```bash
# Run unit tests
make test

# Run AWS CLI integration tests
make aws-test
```

### Code Formatting

```bash
# Format all Odin files
make fmt
```

## Common Issues

### Port Already in Use

```bash
# Check what's using port 8002
lsof -i :8002

# Use a different port
make run PORT=9000
```

### Cannot Create Data Directory

```bash
# Create with proper permissions
mkdir -p ./data
chmod 755 ./data

# Or use a different directory
make run DATA_DIR=/tmp/jormun
```

### RocksDB Not Found

```bash
# Check installation
pkg-config --libs rocksdb

# Install if missing (see Prerequisites)
```

### Odin Compiler Errors

```bash
# Check Odin version
odin version

# Update Odin if needed
brew upgrade odin  # macOS
# or download latest from odin-lang.org
```

## Next Steps

- Read [ARCHITECTURE.md](ARCHITECTURE.md) for internals
- Check [TODO.md](TODO.md) for implementation status
- Browse source code in `dynamodb/`, `rocksdb/`, etc.
- Contribute! See [CONTRIBUTING.md](CONTRIBUTING.md)

## Getting Help

- **Issues**: https://github.com/yourusername/jormundb/issues
- **Discussions**: https://github.com/yourusername/jormundb/discussions
- **Odin Discord**: https://discord.gg/sVBPHEv

## Benchmarking

```bash
# Run benchmarks
make bench

# Profile memory usage
make profile

# Load test
ab -n 10000 -c 100 -p item.json -T application/json \
   http://localhost:8002/
```

## Production Deployment

JormunDB is designed for **local development only**. For production, use:

- AWS DynamoDB (managed service)
- DynamoDB Accelerator (DAX)
- ScyllaDB (DynamoDB-compatible)

## Uninstalling

```bash
# Remove build artifacts
make clean

# Remove installed binary (if installed)
make uninstall

# Remove data directory
rm -rf ./data
```

---

**Happy coding! 🚀**

For questions or issues, please open a GitHub issue or join our Discord.


================================================================================
FILE: ./README.md
================================================================================

# JormunDB

A high-performance, DynamoDB-compatible database server written in Odin, backed by RocksDB.

```
     ╦╔═╗╦═╗╔╦╗╦ ╦╔╗╔╔╦╗╔╗
     ║║ ║╠╦╝║║║║ ║║║║ ║║╠╩╗
    ╚╝╚═╝╩╚═╩ ╩╚═╝╝╚╝═╩╝╚═╝
    DynamoDB-Compatible Database
    Powered by RocksDB + Odin
```

## What is JormunDB?

JormunDB (formerly ZynamoDB) is a local DynamoDB replacement that speaks the DynamoDB wire protocol. Point your AWS SDK or CLI at it and use it as a drop-in development database.

**Why Odin?** The original Zig implementation suffered from explicit allocator threading—every function taking an `allocator` parameter, every allocation needing `errdefer` cleanup. Odin's implicit context allocator system eliminates this ceremony: one `context.allocator = arena_allocator` at the request handler entry and everything downstream just works.

## Features

- ✅ **DynamoDB Wire Protocol**: Works with AWS SDKs and CLI out of the box
- ✅ **Binary Storage**: Efficient TLV encoding for items, varint-prefixed keys
- ✅ **Arena-per-Request**: Zero explicit memory management in business logic
- ✅ **Table Operations**: CreateTable, DeleteTable, DescribeTable, ListTables
- ✅ **Item Operations**: PutItem, GetItem, DeleteItem
- ✅ **Query & Scan**: With pagination support (Limit, ExclusiveStartKey)
- ✅ **Expression Parsing**: KeyConditionExpression for Query operations
- ✅ **Persistent Storage**: RocksDB-backed with full ACID guarantees
- ✅ **Concurrency**: Table-level RW locks for safe concurrent access

## Quick Start

### Prerequisites

- Odin compiler (latest)
- RocksDB development libraries
- Standard compression libraries (snappy, lz4, zstd, etc.)

#### macOS (Homebrew)

```bash
brew install rocksdb odin
```

#### Ubuntu/Debian

```bash
sudo apt install librocksdb-dev libsnappy-dev liblz4-dev libzstd-dev libbz2-dev
# Install Odin from https://odin-lang.org/docs/install/
```

### Build & Run

```bash
# Build the server
make build

# Run with default settings (localhost:8002, ./data directory)
make run

# Run with custom port
make run PORT=9000

# Run with custom data directory
make run DATA_DIR=/tmp/jormundb
```

### Test with AWS CLI

```bash
# Create a table
aws dynamodb create-table \
    --endpoint-url http://localhost:8002 \
    --table-name Users \
    --key-schema AttributeName=id,KeyType=HASH \
    --attribute-definitions AttributeName=id,AttributeType=S \
    --billing-mode PAY_PER_REQUEST

# Put an item
aws dynamodb put-item \
    --endpoint-url http://localhost:8002 \
    --table-name Users \
    --item '{"id":{"S":"user123"},"name":{"S":"Alice"},"age":{"N":"30"}}'

# Get an item
aws dynamodb get-item \
    --endpoint-url http://localhost:8002 \
    --table-name Users \
    --key '{"id":{"S":"user123"}}'

# Query items
aws dynamodb query \
    --endpoint-url http://localhost:8002 \
    --table-name Users \
    --key-condition-expression "id = :id" \
    --expression-attribute-values '{":id":{"S":"user123"}}'

# Scan table
aws dynamodb scan \
    --endpoint-url http://localhost:8002 \
    --table-name Users
```

## Architecture

```
HTTP Request (POST /)
  ↓
X-Amz-Target header → Operation routing
  ↓
JSON body → DynamoDB types
  ↓
Storage engine → RocksDB operations
  ↓
Binary encoding → Disk
  ↓
JSON response → Client
```

### Module Structure

```
jormundb/
├── rocksdb/         - C FFI bindings to librocksdb
├── dynamodb/        - Core types and operations
│   ├── types.odin   - AttributeValue, Item, Key, etc.
│   ├── json.odin    - DynamoDB JSON serialization
│   ├── storage.odin - Storage engine with RocksDB
│   └── handler.odin - HTTP request handlers
├── key_codec/       - Binary key encoding (varint-prefixed)
├── item_codec/      - Binary TLV item encoding
└── main.odin        - HTTP server and entry point
```

### Storage Format

**Keys** (varint-length-prefixed segments):
```
Meta:  [0x01][len][table_name]
Data:  [0x02][len][table_name][len][pk_value][len][sk_value]?
GSI:   [0x03][len][table_name][len][index_name][len][gsi_pk][len][gsi_sk]?
LSI:   [0x04][len][table_name][len][index_name][len][pk][len][lsi_sk]
```

**Values** (TLV binary encoding):
```
[attr_count:varint]
  [name_len:varint][name:bytes][type_tag:u8][value_encoded:bytes]...

Type tags:
  String=0x01, Number=0x02, Binary=0x03, Bool=0x04, Null=0x05
  SS=0x10, NS=0x11, BS=0x12
  List=0x20, Map=0x21
```

## Memory Management

JormunDB uses Odin's context allocator system for elegant memory management:

```odin
// Request handler entry point
handle_request :: proc(conn: net.TCP_Socket) {
    arena: mem.Arena
    mem.arena_init(&arena, make([]byte, mem.Megabyte * 4))
    defer mem.arena_destroy(&arena)

    context.allocator = mem.arena_allocator(&arena)

    // Everything below uses the arena automatically
    // No manual frees, no errdefer cleanup needed
    request := parse_request()  // Uses context.allocator
    response := process(request) // Uses context.allocator
    send_response(response)      // Uses context.allocator

    // Arena is freed here automatically
}
```

Long-lived data (table metadata, locks) uses the default allocator. Request-scoped data uses the arena.

## Development

```bash
# Build debug version
make build

# Build optimized release
make release

# Run tests
make test

# Format code
make fmt

# Clean build artifacts
make clean

# Run with custom settings
make run PORT=9000 DATA_DIR=/tmp/db VERBOSE=1
```

## Performance

From benchmarks on the original Zig version (Odin expected to be similar or better):

```
Sequential Writes     |      10000 ops |   245.32 ms |    40765 ops/sec
Random Reads          |      10000 ops |   312.45 ms |    32006 ops/sec
Batch Writes          |      10000 ops |    89.23 ms |   112071 ops/sec
PutItem               |       5000 ops |   892.34 ms |     5604 ops/sec
GetItem               |       5000 ops |   678.91 ms |     7365 ops/sec
Scan (full table)     |       5000 ops |   234.56 ms |    21320 ops/sec
```

## API Compatibility

### Supported Operations

- ✅ CreateTable
- ✅ DeleteTable
- ✅ DescribeTable
- ✅ ListTables
- ✅ PutItem
- ✅ GetItem
- ✅ DeleteItem
- ✅ Query (with KeyConditionExpression)
- ✅ Scan (with pagination)

### Coming Soon

- ⏳ UpdateItem (with UpdateExpression)
- ⏳ BatchWriteItem
- ⏳ BatchGetItem
- ⏳ Global Secondary Indexes
- ⏳ Local Secondary Indexes
- ⏳ ConditionExpression
- ⏳ FilterExpression
- ⏳ ProjectionExpression

## Configuration

### Environment Variables

```bash
JORMUN_PORT=8002           # Server port
JORMUN_HOST=0.0.0.0        # Bind address
JORMUN_DATA_DIR=./data     # RocksDB data directory
JORMUN_VERBOSE=1           # Enable verbose logging
```

### Command Line Arguments

```bash
./jormundb --port 9000 --host 127.0.0.1 --data-dir /var/db --verbose
```

## Troubleshooting

### "Cannot open RocksDB"

Ensure RocksDB libraries are installed and the data directory is writable:

```bash
# Check RocksDB installation
pkg-config --libs rocksdb

# Check permissions
mkdir -p ./data
chmod 755 ./data
```

### "Connection refused"

Check if the port is already in use:

```bash
lsof -i :8002
```

### "Invalid JSON" errors

Ensure you're using the correct DynamoDB JSON format:

```json
{
  "TableName": "Users",
  "Item": {
    "id": {"S": "user123"},
    "age": {"N": "30"}
  }
}
```

## License

MIT License - see LICENSE file for details.

## Credits

- Inspired by DynamoDB Local
- Built with [Odin](https://odin-lang.org/)
- Powered by [RocksDB](https://rocksdb.org/)
- Originally implemented as ZynamoDB in Zig

## Contributing

Contributions welcome! Please:

1. Format code with `make fmt`
2. Run tests with `make test`
3. Update documentation as needed
4. Follow Odin idioms (context allocators, explicit returns, etc.)

---

**Why "Jormun"?** Jörmungandr, the World Serpent from Norse mythology—a fitting name for something that wraps around your data. Also, it sounds cool.


================================================================================
FILE: ./rocksdb/rocksdb.odin
================================================================================

package rocksdb

import "core:c"
import "core:fmt"

foreign import rocksdb "system:rocksdb"

// In order to use RocksDB's WAL replication helpers, we need to import the C++ library so we use this shim
//foreign import rocksdb_shim "system:jormun_rocksdb_shim"  // I know we'll use in future but because we're not right now, compiler is complaining

// RocksDB C API types
RocksDB_T :: distinct rawptr
RocksDB_Options :: distinct rawptr
RocksDB_WriteOptions :: distinct rawptr
RocksDB_ReadOptions :: distinct rawptr
RocksDB_WriteBatch :: distinct rawptr
RocksDB_Iterator :: distinct rawptr
RocksDB_FlushOptions :: distinct rawptr

// Error type
Error :: enum {
	None,
	OpenFailed,
	WriteFailed,
	ReadFailed,
	DeleteFailed,
	InvalidArgument,
	Corruption,
	NotFound,
	IOError,
	Unknown,
}

// Database handle with options
DB :: struct {
	handle:        RocksDB_T,
	options:       RocksDB_Options,
	write_options: RocksDB_WriteOptions,
	read_options:  RocksDB_ReadOptions,
}

// Foreign C functions
@(default_calling_convention = "c")
foreign rocksdb {
	// Database operations
	rocksdb_open :: proc(options: RocksDB_Options, path: cstring, errptr: ^cstring) -> RocksDB_T ---
	rocksdb_close :: proc(db: RocksDB_T) ---

	// Options
	rocksdb_options_create :: proc() -> RocksDB_Options ---
	rocksdb_options_destroy :: proc(options: RocksDB_Options) ---
	rocksdb_options_set_create_if_missing :: proc(options: RocksDB_Options, val: c.uchar) ---
	rocksdb_options_increase_parallelism :: proc(options: RocksDB_Options, total_threads: c.int) ---
	rocksdb_options_optimize_level_style_compaction :: proc(options: RocksDB_Options, memtable_memory_budget: c.uint64_t) ---
	rocksdb_options_set_compression :: proc(options: RocksDB_Options, compression: c.int) ---

	// Write options
	rocksdb_writeoptions_create :: proc() -> RocksDB_WriteOptions ---
	rocksdb_writeoptions_destroy :: proc(options: RocksDB_WriteOptions) ---

	// Read options
	rocksdb_readoptions_create :: proc() -> RocksDB_ReadOptions ---
	rocksdb_readoptions_destroy :: proc(options: RocksDB_ReadOptions) ---

	// Put/Get/Delete
	rocksdb_put :: proc(db: RocksDB_T, options: RocksDB_WriteOptions, key: [^]byte, keylen: c.size_t, val: [^]byte, vallen: c.size_t, errptr: ^cstring) ---
	rocksdb_get :: proc(db: RocksDB_T, options: RocksDB_ReadOptions, key: [^]byte, keylen: c.size_t, vallen: ^c.size_t, errptr: ^cstring) -> [^]byte ---
	rocksdb_delete :: proc(db: RocksDB_T, options: RocksDB_WriteOptions, key: [^]byte, keylen: c.size_t, errptr: ^cstring) ---

	// Flush
	rocksdb_flushoptions_create :: proc() -> RocksDB_FlushOptions ---
	rocksdb_flushoptions_destroy :: proc(options: RocksDB_FlushOptions) ---
	rocksdb_flush :: proc(db: RocksDB_T, options: RocksDB_FlushOptions, errptr: ^cstring) ---

	// Write batch
	rocksdb_writebatch_create :: proc() -> RocksDB_WriteBatch ---
	rocksdb_writebatch_destroy :: proc(batch: RocksDB_WriteBatch) ---
	rocksdb_writebatch_put :: proc(batch: RocksDB_WriteBatch, key: [^]byte, keylen: c.size_t, val: [^]byte, vallen: c.size_t) ---
	rocksdb_writebatch_delete :: proc(batch: RocksDB_WriteBatch, key: [^]byte, keylen: c.size_t) ---
	rocksdb_writebatch_clear :: proc(batch: RocksDB_WriteBatch) ---
	rocksdb_write :: proc(db: RocksDB_T, options: RocksDB_WriteOptions, batch: RocksDB_WriteBatch, errptr: ^cstring) ---

	// Iterator
	rocksdb_create_iterator :: proc(db: RocksDB_T, options: RocksDB_ReadOptions) -> RocksDB_Iterator ---
	rocksdb_iter_destroy :: proc(iter: RocksDB_Iterator) ---
	rocksdb_iter_seek_to_first :: proc(iter: RocksDB_Iterator) ---
	rocksdb_iter_seek_to_last :: proc(iter: RocksDB_Iterator) ---
	rocksdb_iter_seek :: proc(iter: RocksDB_Iterator, key: [^]byte, keylen: c.size_t) ---
	rocksdb_iter_seek_for_prev :: proc(iter: RocksDB_Iterator, key: [^]byte, keylen: c.size_t) ---
	rocksdb_iter_valid :: proc(iter: RocksDB_Iterator) -> c.uchar ---
	rocksdb_iter_next :: proc(iter: RocksDB_Iterator) ---
	rocksdb_iter_prev :: proc(iter: RocksDB_Iterator) ---
	rocksdb_iter_key :: proc(iter: RocksDB_Iterator, klen: ^c.size_t) -> [^]byte ---
	rocksdb_iter_value :: proc(iter: RocksDB_Iterator, vlen: ^c.size_t) -> [^]byte ---

	// Memory management
	rocksdb_free :: proc(ptr: rawptr) ---
}

// Compression types
ROCKSDB_NO_COMPRESSION :: 0
ROCKSDB_SNAPPY_COMPRESSION :: 1
ROCKSDB_ZLIB_COMPRESSION :: 2
ROCKSDB_BZIP2_COMPRESSION :: 3
ROCKSDB_LZ4_COMPRESSION :: 4
ROCKSDB_LZ4HC_COMPRESSION :: 5
ROCKSDB_ZSTD_COMPRESSION :: 7

// Open a database
db_open :: proc(path: string, create_if_missing := true) -> (DB, Error) {
	options := rocksdb_options_create()
	if options == nil {
		return {}, .Unknown
	}

	// Set create if missing
	rocksdb_options_set_create_if_missing(options, create_if_missing ? 1 : 0)

	// Performance optimizations
	rocksdb_options_increase_parallelism(options, 4)
	rocksdb_options_optimize_level_style_compaction(options, 512 * 1024 * 1024)
	rocksdb_options_set_compression(options, ROCKSDB_LZ4_COMPRESSION)

	// Create write and read options
	write_options := rocksdb_writeoptions_create()
	if write_options == nil {
		rocksdb_options_destroy(options)
		return {}, .Unknown
	}

	read_options := rocksdb_readoptions_create()
	if read_options == nil {
		rocksdb_writeoptions_destroy(write_options)
		rocksdb_options_destroy(options)
		return {}, .Unknown
	}

	// Open database
	err: cstring
	path_cstr := fmt.ctprintf("%s", path)
	handle := rocksdb_open(options, path_cstr, &err)
	if err != nil {
		defer rocksdb_free(rawptr(err)) // Cast it here and now so we don't deal with issues from FFI down the line
		rocksdb_readoptions_destroy(read_options)
		rocksdb_writeoptions_destroy(write_options)
		rocksdb_options_destroy(options)
		return {}, .OpenFailed
	}

	return DB{
		handle = handle,
		options = options,
		write_options = write_options,
		read_options = read_options,
	}, .None
}

// Close database
db_close :: proc(db: ^DB) {
	rocksdb_readoptions_destroy(db.read_options)
	rocksdb_writeoptions_destroy(db.write_options)
	rocksdb_close(db.handle)
	rocksdb_options_destroy(db.options)
}

// Put key-value pair
db_put :: proc(db: ^DB, key: []byte, value: []byte) -> Error {
	err: cstring
	rocksdb_put(
		db.handle,
		db.write_options,
		raw_data(key),
		c.size_t(len(key)),
		raw_data(value),
		c.size_t(len(value)),
		&err,
	)
	if err != nil {
		defer rocksdb_free(rawptr(err)) // Cast it here and now so we don't deal with issues from FFI down the line
		return .WriteFailed
	}
	return .None
}

// Get value by key (returns owned slice - caller must free)
db_get :: proc(db: ^DB, key: []byte) -> (value: []byte, err: Error) {
	errptr: cstring
	value_len: c.size_t

	value_ptr := rocksdb_get(
		db.handle,
		db.read_options,
		raw_data(key),
		c.size_t(len(key)),
		&value_len,
		&errptr,
	)

	if errptr != nil {
		defer rocksdb_free(rawptr(errptr)) // Cast it here and now so we don't deal with issues from FFI down the line
		return nil, .ReadFailed
	}

	if value_ptr == nil {
		return nil, .NotFound
	}

	// Copy the data and free RocksDB's buffer
	result := make([]byte, value_len, context.allocator)
	copy(result, value_ptr[:value_len])
	rocksdb_free(rawptr(value_ptr)) // Cast it here and now so we don't deal with issues from FFI down the line

	return result, .None
}

// Delete key
db_delete :: proc(db: ^DB, key: []byte) -> Error {
	err: cstring
	rocksdb_delete(
		db.handle,
		db.write_options,
		raw_data(key),
		c.size_t(len(key)),
		&err,
	)
	if err != nil {
		defer rocksdb_free(rawptr(err)) // Cast it here and now so we don't deal with issues from FFI down the line
		return .DeleteFailed
	}
	return .None
}

// Flush database
db_flush :: proc(db: ^DB) -> Error {
	flush_opts := rocksdb_flushoptions_create()
	if flush_opts == nil {
		return .Unknown
	}
	defer rocksdb_flushoptions_destroy(flush_opts)

	err: cstring
	rocksdb_flush(db.handle, flush_opts, &err)
	if err != nil {
		defer rocksdb_free(rawptr(err)) // Cast it here and now so we don't deal with issues from FFI down the line
		return .IOError
	}
	return .None
}

// Write batch
WriteBatch :: struct {
	handle: RocksDB_WriteBatch,
}

// Create write batch
batch_create :: proc() -> (WriteBatch, Error) {
	handle := rocksdb_writebatch_create()
	if handle == nil {
		return {}, .Unknown
	}
	return WriteBatch{handle = handle}, .None
}

// Destroy write batch
batch_destroy :: proc(batch: ^WriteBatch) {
	rocksdb_writebatch_destroy(batch.handle)
}

// Add put operation to batch
batch_put :: proc(batch: ^WriteBatch, key: []byte, value: []byte) {
	rocksdb_writebatch_put(
		batch.handle,
		raw_data(key),
		c.size_t(len(key)),
		raw_data(value),
		c.size_t(len(value)),
	)
}

// Add delete operation to batch
batch_delete :: proc(batch: ^WriteBatch, key: []byte) {
	rocksdb_writebatch_delete(
		batch.handle,
		raw_data(key),
		c.size_t(len(key)),
	)
}

// Clear batch
batch_clear :: proc(batch: ^WriteBatch) {
	rocksdb_writebatch_clear(batch.handle)
}

// Write batch to database
batch_write :: proc(db: ^DB, batch: ^WriteBatch) -> Error {
	err: cstring
	rocksdb_write(db.handle, db.write_options, batch.handle, &err)
	if err != nil {
		defer rocksdb_free(rawptr(err)) // Cast it here and now so we don't deal with issues from FFI down the line
		return .WriteFailed
	}
	return .None
}

// Iterator
Iterator :: struct {
	handle: RocksDB_Iterator,
}

// Create iterator
iter_create :: proc(db: ^DB) -> (Iterator, Error) {
	handle := rocksdb_create_iterator(db.handle, db.read_options)
	if handle == nil {
		return {}, .Unknown
	}
	return Iterator{handle = handle}, .None
}

// Destroy iterator
iter_destroy :: proc(iter: ^Iterator) {
	rocksdb_iter_destroy(iter.handle)
}

// Seek to first
iter_seek_to_first :: proc(iter: ^Iterator) {
	rocksdb_iter_seek_to_first(iter.handle)
}

// Seek to last
iter_seek_to_last :: proc(iter: ^Iterator) {
	rocksdb_iter_seek_to_last(iter.handle)
}

// Seek to key
iter_seek :: proc(iter: ^Iterator, target: []byte) {
	rocksdb_iter_seek(iter.handle, raw_data(target), c.size_t(len(target)))
}

// Check if iterator is valid
iter_valid :: proc(iter: ^Iterator) -> bool {
	return rocksdb_iter_valid(iter.handle) != 0
}

// Move to next
iter_next :: proc(iter: ^Iterator) {
	rocksdb_iter_next(iter.handle)
}

// Move to previous
iter_prev :: proc(iter: ^Iterator) {
	rocksdb_iter_prev(iter.handle)
}

// Get current key (returns borrowed slice)
iter_key :: proc(iter: ^Iterator) -> []byte {
	klen: c.size_t
	key_ptr := rocksdb_iter_key(iter.handle, &klen)
	if key_ptr == nil {
		return nil
	}
	return key_ptr[:klen]
}

// Get current value (returns borrowed slice)
iter_value :: proc(iter: ^Iterator) -> []byte {
	vlen: c.size_t
	value_ptr := rocksdb_iter_value(iter.handle, &vlen)
	if value_ptr == nil {
		return nil
	}
	return value_ptr[:vlen]
}


================================================================================
FILE: ./rocksdb_shim/rocksdb_shim.cc
================================================================================

// TODO: In order to use RocksDB's WAL replication helpers, we need to import the C++ library so we use this shim
/**
	C++ shim implementation notes (the important bits)

	In this rocksdb_shim.cc we'll need to use:

	rocksdb::DB::Open(...)

	db->GetLatestSequenceNumber()

	db->GetUpdatesSince(seq, &iter)

	from each TransactionLogIterator entry:

	get WriteBatch and serialize via WriteBatch::Data()

	apply via rocksdb::WriteBatch wb(data); db->Write(write_options, &wb);

	Also we must configure WAL retention so the followers don’t fall off the end. RocksDB warns the iterator can become invalid if WAL is cleared aggressively; typical controls are WAL TTL / size limit.

	https://github.com/facebook/rocksdb/issues/1565
 */

================================================================================
FILE: ./rocksdb_shim/rocksdb_shim.h
================================================================================

// In order to use RocksDB's WAL replication helpers, we need to import the C++ library so we use this shim
#pragma once
#include <stdint.h>
#include <stddef.h>

#ifdef __cplusplus
extern "C"
{
#endif

	typedef struct jormun_db jormun_db;
	typedef struct jormun_wal_iter jormun_wal_iter;

	// Open/close (so Odin never touches rocksdb_t directly)
	jormun_db *jormun_db_open(const char *path, int create_if_missing, char **err);
	void jormun_db_close(jormun_db *db);

	// Basic ops (you can mirror what you already have)
	void jormun_db_put(jormun_db *db,
					   const void *key, size_t keylen,
					   const void *val, size_t vallen,
					   char **err);

	unsigned char *jormun_db_get(jormun_db *db,
								 const void *key, size_t keylen,
								 size_t *vallen,
								 char **err);

	// caller frees with this:
	void jormun_free(void *p);

	// Replication primitives
	uint64_t jormun_latest_sequence(jormun_db *db);

	// Iterator: start at seq (inclusive-ish; RocksDB positions to batch containing seq or first after)
	jormun_wal_iter *jormun_wal_iter_create(jormun_db *db, uint64_t seq, char **err);
	void jormun_wal_iter_destroy(jormun_wal_iter *it);

	// Next batch -> returns 1 if produced a batch, 0 if no more / not available
	// You get a serialized “write batch” blob (rocksdb::WriteBatch::Data()) plus the batch start seq.
	int jormun_wal_iter_next(jormun_wal_iter *it,
							 uint64_t *batch_start_seq,
							 unsigned char **out_data,
							 size_t *out_len,
							 char **err);

	// Apply serialized writebatch blob on follower
	void jormun_apply_writebatch(jormun_db *db,
								 const unsigned char *data, size_t len,
								 char **err);

#ifdef __cplusplus
}
#endif


================================================================================
FILE: ./TODO.md
================================================================================

# JormunDB Implementation TODO

This tracks the rewrite from Zig to Odin and remaining features.

## ✅ Completed

- [x] Project structure
- [x] Makefile with build/run/test targets
- [x] README with usage instructions
- [x] ARCHITECTURE documentation
- [x] RocksDB FFI bindings (rocksdb/rocksdb.odin)
- [x] Core types (dynamodb/types.odin)
- [x] Key codec with varint encoding (key_codec/key_codec.odin)
- [x] Main entry point with arena pattern demo
- [x] .gitignore
- [x] HTTP Server Scaffolding

## 🚧 In Progress (Need to Complete)

### Core Modules

- [ ] **dynamodb/json.odin** - DynamoDB JSON parsing and serialization
  - Parse `{"S": "value"}` format
  - Serialize AttributeValue to DynamoDB JSON
  - Parse request bodies (PutItem, GetItem, etc.)

- [ ] **item_codec/item_codec.odin** - Binary TLV encoding for items
  - Encode Item to binary TLV format
  - Decode binary TLV back to Item
  - Type tag handling for all DynamoDB types

- [ ] **dynamodb/storage.odin** - Storage engine with RocksDB
  - Table metadata management
  - create_table, delete_table, describe_table, list_tables
  - put_item, get_item, delete_item
  - scan, query with pagination
  - Table-level RW locks

- [ ] **dynamodb/handler.odin** - HTTP request handlers
  - Route X-Amz-Target to handler functions
  - handle_create_table, handle_put_item, etc.
  - Build responses with proper error handling
  - Arena allocator integration

### HTTP Server

- [ ] **HTTP server implementation**
  - Accept TCP connections
  - Parse HTTP POST requests
  - Read JSON bodies
  - Send HTTP responses with headers
  - Keep-alive support
  - Options (Why we haven't checked this off yet, we need to make sure we chose the right option as the project grows, might make more sense to impliment different option):
    - Use `core:net` directly
    - Use C FFI with libmicrohttpd
    - Use Odin's vendor:microui (if suitable)

### Expression Parsers (Priority 3)

- [ ] **KeyConditionExpression parser**
  - Tokenizer for expressions
  - Parse `pk = :pk AND sk > :sk`
  - Support begins_with, BETWEEN
  - ExpressionAttributeNames/Values

- [ ] **UpdateExpression parser** (later)
  - SET operations
  - REMOVE operations
  - ADD operations
  - DELETE operations

### Replication Support (Priority 4)

  - [ ] **Build C++ Shim in order to use RocksDB's WAL replication helpers**
  - [ ] **Add configurator to set instance as a master or slave node and point to proper Target and Destination IPs**
  - [ ] **Leverage C++ helpers from shim**

### Subscribe To Changes Feature (Priority LAST [But keep in mind because semantics we decide now will make this easier later])

  - [ ] **Best-effort notifications (Postgres-ish LISTEN/NOTIFY [in-memory pub/sub fanout. If you’re not connected, you miss it.])**
    - Add an in-process “event bus” channels: table-wide, partition-key, item-key, “all”.
    - When putItem/deleteItem/updateItem/createTable/... commits successfully publish {op, table, key, timestamp, item?}

  - [ ] **Durable change streams (Mongo-ish [append every mutation to a persistent log and let consumers read it with resume tokens.])**
    - Create a “changelog” keyspace
    - Generate a monotonically increasing sequence by using a stable Per-partition sequence cursor
    - Expose via an API (I prefer publishing to MQTT or SSE)

## 📋 Testing

- [ ] Unit tests for key_codec
- [ ] Unit tests for item_codec
- [ ] Unit tests for JSON parsing
- [ ] Integration tests with AWS CLI
- [ ] Benchmark suite

## 🔧 Build & Tooling

- [ ] Verify Makefile works on macOS
- [ ] Verify Makefile works on Linux
- [ ] Add Docker support (optional)
- [ ] Add install script

## 📚 Documentation

- [ ] Code comments for public APIs
- [ ] Usage examples in README
- [ ] API compatibility matrix
- [ ] Performance tuning guide

## 🎯 Priority Order

1. **HTTP Server** - Need this to accept requests
2. **JSON Parsing** - Need this to understand DynamoDB format
3. **Storage Engine** - Core CRUD operations
4. **Handlers** - Wire everything together
5. **Item Codec** - Efficient binary storage
6. **Expression Parsers** - Query functionality

## 📝 Notes

### Zig → Odin Translation Patterns

**Memory Management:**
```zig
// Zig
const item = try allocator.create(Item);
defer allocator.destroy(item);
```
```odin
// Odin
item := new(Item)
// No defer needed if using arena
```

**Error Handling:**
```zig
// Zig
fn foo() !Result {
    return error.Failed;
}
const x = try foo();
```
```odin
// Odin
foo :: proc() -> (Result, bool) {
    return {}, false
}
x := foo() or_return
```

**Slices:**
```zig
// Zig
const slice: []const u8 = data;
```
```odin
// Odin
slice: []byte = data
```

**Maps:**
```zig
// Zig
var map = std.StringHashMap(Value).init(allocator);
defer map.deinit();
```
```odin
// Odin
map := make(map[string]Value)
defer delete(map)
```

### Key Decisions

1. **Use `Maybe(T)` instead of `?T`** - Odin's optional type
2. **Use `or_return` instead of `try`** - Odin's error propagation
3. **Use `context.allocator`** - Implicit allocator from context
4. **Use `#partial switch`** - For union type checking
5. **Use `transmute`** - For zero-cost type conversions

### Reference Zig Files

When implementing, reference these Zig files:
- `src/dynamodb/json.zig` - 400 lines, DynamoDB JSON format
- `src/dynamodb/storage.zig` - 460 lines, storage engine
- `src/dynamodb/handler.zig` - 500+ lines, request handlers
- `src/item_codec.zig` - 350 lines, TLV encoding
- `src/http.zig` - 250 lines, HTTP server

### Quick Test Commands

```bash
# Build and test
make build
make test

# Run server
make run

# Test with AWS CLI
aws dynamodb list-tables --endpoint-url http://localhost:8002
```