first commit
This commit is contained in:
426
ARCHITECTURE.md
Normal file
426
ARCHITECTURE.md
Normal file
@@ -0,0 +1,426 @@
|
||||
## JormunDB Architecture
|
||||
|
||||
This document explains the internal architecture of JormunDB, including design decisions, storage formats, and the arena-per-request memory management pattern.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Overview](#overview)
|
||||
- [Why Odin?](#why-odin)
|
||||
- [Memory Management](#memory-management)
|
||||
- [Storage Format](#storage-format)
|
||||
- [Module Structure](#module-structure)
|
||||
- [Request Flow](#request-flow)
|
||||
- [Concurrency Model](#concurrency-model)
|
||||
|
||||
## Overview
|
||||
|
||||
JormunDB is a DynamoDB-compatible database server that speaks the DynamoDB wire protocol. It uses RocksDB for persistent storage and is written in Odin for elegant memory management.
|
||||
|
||||
### Key Design Goals
|
||||
|
||||
1. **Zero allocation ceremony** - No explicit `defer free()` or error handling for every allocation
|
||||
2. **Binary storage** - Efficient TLV encoding instead of JSON
|
||||
3. **API compatibility** - Drop-in replacement for DynamoDB Local
|
||||
4. **Performance** - RocksDB-backed with efficient key encoding
|
||||
|
||||
## Why Odin?
|
||||
|
||||
The original implementation in Zig suffered from explicit allocator threading:
|
||||
|
||||
```zig
|
||||
// Zig version - explicit allocator everywhere
|
||||
fn handleRequest(allocator: std.mem.Allocator, request: []const u8) !Response {
|
||||
const parsed = try parseJson(allocator, request);
|
||||
defer parsed.deinit(allocator);
|
||||
|
||||
const item = try storage.getItem(allocator, parsed.table_name, parsed.key);
|
||||
defer if (item) |i| freeItem(allocator, i);
|
||||
|
||||
const response = try serializeResponse(allocator, item);
|
||||
defer allocator.free(response);
|
||||
|
||||
return response; // Wait, we deferred the free!
|
||||
}
|
||||
```
|
||||
|
||||
Odin's context allocator system eliminates this:
|
||||
|
||||
```odin
|
||||
// Odin version - implicit context allocator
|
||||
handle_request :: proc(request: []byte) -> Response {
|
||||
// All allocations use context.allocator automatically
|
||||
parsed := parse_json(request)
|
||||
item := storage_get_item(parsed.table_name, parsed.key)
|
||||
response := serialize_response(item)
|
||||
|
||||
return response
|
||||
// Everything freed when arena is destroyed
|
||||
}
|
||||
```
|
||||
|
||||
## Memory Management
|
||||
|
||||
JormunDB uses a two-allocator strategy:
|
||||
|
||||
### 1. Arena Allocator (Request-Scoped)
|
||||
|
||||
Every HTTP request gets its own arena:
|
||||
|
||||
```odin
|
||||
handle_connection :: proc(conn: net.TCP_Socket) {
|
||||
// Create arena for this request (4MB)
|
||||
arena: mem.Arena
|
||||
mem.arena_init(&arena, make([]byte, mem.Megabyte * 4))
|
||||
defer mem.arena_destroy(&arena)
|
||||
|
||||
// Set context allocator
|
||||
context.allocator = mem.arena_allocator(&arena)
|
||||
|
||||
// All downstream code uses context.allocator
|
||||
request := parse_http_request(conn) // uses arena
|
||||
response := handle_request(request) // uses arena
|
||||
send_response(conn, response) // uses arena
|
||||
|
||||
// Arena is freed here - everything cleaned up automatically
|
||||
}
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- No individual `free()` calls needed
|
||||
- No `errdefer` cleanup
|
||||
- No use-after-free bugs
|
||||
- No memory leaks from forgotten frees
|
||||
- Predictable performance (no GC pauses)
|
||||
|
||||
### 2. Default Allocator (Long-Lived Data)
|
||||
|
||||
The default allocator (typically `context.allocator` at program start) is used for:
|
||||
|
||||
- Table metadata
|
||||
- Table locks (sync.RW_Mutex)
|
||||
- Engine state
|
||||
- Items returned from storage layer (copied to request arena when needed)
|
||||
|
||||
## Storage Format
|
||||
|
||||
### Binary Keys (Varint-Prefixed Segments)
|
||||
|
||||
All keys use varint length prefixes for space efficiency:
|
||||
|
||||
```
|
||||
Meta key: [0x01][len][table_name]
|
||||
Data key: [0x02][len][table_name][len][pk_value][len][sk_value]?
|
||||
GSI key: [0x03][len][table_name][len][index_name][len][gsi_pk][len][gsi_sk]?
|
||||
LSI key: [0x04][len][table_name][len][index_name][len][pk][len][lsi_sk]
|
||||
```
|
||||
|
||||
**Example Data Key:**
|
||||
```
|
||||
Table: "Users"
|
||||
PK: "user:123"
|
||||
SK: "profile"
|
||||
|
||||
Encoded:
|
||||
[0x02] // Entity type (Data)
|
||||
[0x05] // Table name length (5)
|
||||
Users // Table name bytes
|
||||
[0x08] // PK length (8)
|
||||
user:123 // PK bytes
|
||||
[0x07] // SK length (7)
|
||||
profile // SK bytes
|
||||
```
|
||||
|
||||
### Item Encoding (TLV Format)
|
||||
|
||||
Items use Tag-Length-Value encoding for space efficiency:
|
||||
|
||||
```
|
||||
Format:
|
||||
[attr_count:varint]
|
||||
[name_len:varint][name:bytes][type_tag:u8][value_len:varint][value:bytes]...
|
||||
|
||||
Type Tags:
|
||||
String = 0x01 Number = 0x02 Binary = 0x03
|
||||
Bool = 0x04 Null = 0x05
|
||||
SS = 0x10 NS = 0x11 BS = 0x12
|
||||
List = 0x20 Map = 0x21
|
||||
```
|
||||
|
||||
**Example Item:**
|
||||
```json
|
||||
{
|
||||
"id": {"S": "user123"},
|
||||
"age": {"N": "30"}
|
||||
}
|
||||
```
|
||||
|
||||
Encoded as:
|
||||
```
|
||||
[0x02] // 2 attributes
|
||||
[0x02] // name length (2)
|
||||
id // name bytes
|
||||
[0x01] // type tag (String)
|
||||
[0x07] // value length (7)
|
||||
user123 // value bytes
|
||||
|
||||
[0x03] // name length (3)
|
||||
age // name bytes
|
||||
[0x02] // type tag (Number)
|
||||
[0x02] // value length (2)
|
||||
30 // value bytes (stored as string)
|
||||
```
|
||||
|
||||
## Module Structure
|
||||
|
||||
```
|
||||
jormundb/
|
||||
├── main.odin # Entry point, HTTP server
|
||||
├── rocksdb/ # RocksDB C FFI bindings
|
||||
│ └── rocksdb.odin # db_open, db_put, db_get, etc.
|
||||
├── dynamodb/ # DynamoDB protocol implementation
|
||||
│ ├── types.odin # Core types (Attribute_Value, Item, Key, etc.)
|
||||
│ ├── json.odin # DynamoDB JSON parsing/serialization
|
||||
│ ├── storage.odin # Storage engine (CRUD, scan, query)
|
||||
│ └── handler.odin # HTTP request handlers
|
||||
├── key_codec/ # Binary key encoding
|
||||
│ └── key_codec.odin # build_data_key, decode_data_key, etc.
|
||||
└── item_codec/ # Binary TLV item encoding
|
||||
└── item_codec.odin # encode, decode
|
||||
```
|
||||
|
||||
## Request Flow
|
||||
|
||||
```
|
||||
1. HTTP POST / arrives
|
||||
↓
|
||||
2. Create arena allocator (4MB)
|
||||
Set context.allocator = arena_allocator
|
||||
↓
|
||||
3. Parse HTTP headers
|
||||
Extract X-Amz-Target → Operation
|
||||
↓
|
||||
4. Parse JSON body
|
||||
Convert DynamoDB JSON → internal types
|
||||
↓
|
||||
5. Route to handler (e.g., handle_put_item)
|
||||
↓
|
||||
6. Storage engine operation
|
||||
- Build binary key
|
||||
- Encode item to TLV
|
||||
- RocksDB put/get/delete
|
||||
↓
|
||||
7. Build response
|
||||
- Serialize item to DynamoDB JSON
|
||||
- Format HTTP response
|
||||
↓
|
||||
8. Send response
|
||||
↓
|
||||
9. Destroy arena
|
||||
All request memory freed automatically
|
||||
```
|
||||
|
||||
## Concurrency Model
|
||||
|
||||
### Table-Level RW Locks
|
||||
|
||||
Each table has a reader-writer lock:
|
||||
|
||||
```odin
|
||||
Storage_Engine :: struct {
|
||||
db: rocksdb.DB,
|
||||
table_locks: map[string]^sync.RW_Mutex,
|
||||
table_locks_mutex: sync.Mutex,
|
||||
}
|
||||
```
|
||||
|
||||
**Read Operations** (GetItem, Query, Scan):
|
||||
- Acquire shared lock
|
||||
- Multiple readers can run concurrently
|
||||
- Writers are blocked
|
||||
|
||||
**Write Operations** (PutItem, DeleteItem, UpdateItem):
|
||||
- Acquire exclusive lock
|
||||
- Only one writer at a time
|
||||
- All readers are blocked
|
||||
|
||||
### Thread Safety
|
||||
|
||||
- RocksDB handles are thread-safe (column family-based)
|
||||
- Table metadata is protected by locks
|
||||
- Request arenas are thread-local (no sharing)
|
||||
|
||||
## Error Handling
|
||||
|
||||
Odin uses explicit error returns via `or_return`:
|
||||
|
||||
```odin
|
||||
// Odin error handling
|
||||
parse_json :: proc(data: []byte) -> (Item, bool) {
|
||||
parsed := json.parse(data) or_return
|
||||
item := json_to_item(parsed) or_return
|
||||
return item, true
|
||||
}
|
||||
|
||||
// Usage
|
||||
item := parse_json(request.body) or_else {
|
||||
return error_response(.ValidationException, "Invalid JSON")
|
||||
}
|
||||
```
|
||||
|
||||
No exceptions, no panic-recover patterns. Every error path is explicit.
|
||||
|
||||
## DynamoDB Wire Protocol
|
||||
|
||||
### Request Format
|
||||
|
||||
```
|
||||
POST / HTTP/1.1
|
||||
X-Amz-Target: DynamoDB_20120810.PutItem
|
||||
Content-Type: application/x-amz-json-1.0
|
||||
|
||||
{
|
||||
"TableName": "Users",
|
||||
"Item": {
|
||||
"id": {"S": "user123"},
|
||||
"name": {"S": "Alice"}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Response Format
|
||||
|
||||
```
|
||||
HTTP/1.1 200 OK
|
||||
Content-Type: application/x-amz-json-1.0
|
||||
x-amzn-RequestId: local-request-id
|
||||
|
||||
{}
|
||||
```
|
||||
|
||||
### Error Format
|
||||
|
||||
```json
|
||||
{
|
||||
"__type": "com.amazonaws.dynamodb.v20120810#ResourceNotFoundException",
|
||||
"message": "Table not found"
|
||||
}
|
||||
```
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
### Time Complexity
|
||||
|
||||
| Operation | Complexity | Notes |
|
||||
|-----------|-----------|-------|
|
||||
| PutItem | O(log n) | RocksDB LSM tree insert |
|
||||
| GetItem | O(log n) | RocksDB point lookup |
|
||||
| DeleteItem | O(log n) | RocksDB deletion |
|
||||
| Query | O(log n + m) | n = items in table, m = result set |
|
||||
| Scan | O(n) | Full table scan |
|
||||
|
||||
### Space Complexity
|
||||
|
||||
- Binary keys: ~20-100 bytes (vs 50-200 bytes JSON)
|
||||
- Binary items: ~30% smaller than JSON
|
||||
- Varint encoding saves space on small integers
|
||||
|
||||
### Benchmarks (Expected)
|
||||
|
||||
Based on Zig version performance:
|
||||
|
||||
```
|
||||
Operation Throughput Latency (p50)
|
||||
PutItem ~5,000/sec ~0.2ms
|
||||
GetItem ~7,000/sec ~0.14ms
|
||||
Query (1 item) ~8,000/sec ~0.12ms
|
||||
Scan (1000 items) ~20/sec ~50ms
|
||||
```
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
### Planned Features
|
||||
|
||||
1. **UpdateExpression** - SET/REMOVE/ADD/DELETE operations
|
||||
2. **FilterExpression** - Post-query filtering
|
||||
3. **ProjectionExpression** - Return subset of attributes
|
||||
4. **Global Secondary Indexes** - Query by non-key attributes
|
||||
5. **Local Secondary Indexes** - Alternate sort keys
|
||||
6. **BatchWriteItem** - Batch mutations
|
||||
7. **BatchGetItem** - Batch reads
|
||||
8. **Transactions** - ACID multi-item operations
|
||||
|
||||
### Optimization Opportunities
|
||||
|
||||
1. **Connection pooling** - Reuse HTTP connections
|
||||
2. **Bloom filters** - Faster negative lookups
|
||||
3. **Compression** - LZ4/Zstd on large items
|
||||
4. **Caching layer** - Hot item cache
|
||||
5. **Parallel scan** - Segment-based scanning
|
||||
|
||||
## Debugging
|
||||
|
||||
### Enable Verbose Logging
|
||||
|
||||
```bash
|
||||
make run VERBOSE=1
|
||||
```
|
||||
|
||||
### Inspect RocksDB
|
||||
|
||||
```bash
|
||||
# Use ldb tool to inspect database
|
||||
ldb --db=./data scan
|
||||
ldb --db=./data get <key_hex>
|
||||
```
|
||||
|
||||
### Memory Profiling
|
||||
|
||||
Odin's tracking allocator can detect leaks:
|
||||
|
||||
```odin
|
||||
when ODIN_DEBUG {
|
||||
track: mem.Tracking_Allocator
|
||||
mem.tracking_allocator_init(&track, context.allocator)
|
||||
context.allocator = mem.tracking_allocator(&track)
|
||||
|
||||
defer {
|
||||
for _, leak in track.allocation_map {
|
||||
fmt.printfln("Leaked %d bytes at %p", leak.size, leak.location)
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Migration from Zig Version
|
||||
|
||||
The Zig version (ZynamoDB) used the same binary storage format, so existing RocksDB databases can be read by JormunDB without migration.
|
||||
|
||||
### Compatibility
|
||||
|
||||
- ✅ Binary key format (byte-compatible)
|
||||
- ✅ Binary item format (byte-compatible)
|
||||
- ✅ Table metadata (JSON, compatible)
|
||||
- ✅ HTTP wire protocol (identical)
|
||||
|
||||
### Breaking Changes
|
||||
|
||||
None - JormunDB can open ZynamoDB databases directly.
|
||||
|
||||
---
|
||||
|
||||
## Contributing
|
||||
|
||||
When contributing to JormunDB:
|
||||
|
||||
1. **Use the context allocator** - All request-scoped allocations should use `context.allocator`
|
||||
2. **Avoid manual frees** - Let the arena handle it
|
||||
3. **Long-lived data** - Use the default allocator explicitly
|
||||
4. **Test thoroughly** - Run `make test` before committing
|
||||
5. **Format code** - Run `make fmt` before committing
|
||||
|
||||
## References
|
||||
|
||||
- [Odin Language](https://odin-lang.org/)
|
||||
- [RocksDB Wiki](https://github.com/facebook/rocksdb/wiki)
|
||||
- [DynamoDB API Reference](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/)
|
||||
- [Varint Encoding](https://developers.google.com/protocol-buffers/docs/encoding#varints)
|
||||
198
Makefile
Normal file
198
Makefile
Normal file
@@ -0,0 +1,198 @@
|
||||
.PHONY: all build release run test clean fmt help install
|
||||
|
||||
# Project configuration
|
||||
PROJECT_NAME := jormundb
|
||||
ODIN := odin
|
||||
BUILD_DIR := build
|
||||
SRC_DIR := .
|
||||
|
||||
# RocksDB and compression libraries
|
||||
ROCKSDB_LIBS := -lrocksdb -lstdc++ -lsnappy -llz4 -lzstd -lz -lbz2
|
||||
|
||||
# Platform-specific library paths
|
||||
UNAME_S := $(shell uname -s)
|
||||
ifeq ($(UNAME_S),Darwin)
|
||||
# macOS (Homebrew)
|
||||
LIB_PATH := -L/usr/local/lib -L/opt/homebrew/lib
|
||||
INCLUDE_PATH := -I/usr/local/include -I/opt/homebrew/include
|
||||
else ifeq ($(UNAME_S),Linux)
|
||||
# Linux
|
||||
LIB_PATH := -L/usr/local/lib -L/usr/lib
|
||||
INCLUDE_PATH := -I/usr/local/include
|
||||
endif
|
||||
|
||||
# Build flags
|
||||
DEBUG_FLAGS := -debug -o:none
|
||||
RELEASE_FLAGS := -o:speed -disable-assert -no-bounds-check
|
||||
COMMON_FLAGS := -vet -strict-style
|
||||
|
||||
# Linker flags
|
||||
EXTRA_LINKER_FLAGS := $(LIB_PATH) $(ROCKSDB_LIBS)
|
||||
|
||||
# Runtime configuration
|
||||
PORT ?= 8000
|
||||
HOST ?= 0.0.0.0
|
||||
DATA_DIR ?= ./data
|
||||
VERBOSE ?= 0
|
||||
|
||||
# Colors for output
|
||||
BLUE := \033[0;34m
|
||||
GREEN := \033[0;32m
|
||||
YELLOW := \033[0;33m
|
||||
RED := \033[0;31m
|
||||
NC := \033[0m # No Color
|
||||
|
||||
# Default target
|
||||
all: build
|
||||
|
||||
# Build debug version
|
||||
build:
|
||||
@echo "$(BLUE)Building $(PROJECT_NAME) (debug)...$(NC)"
|
||||
@mkdir -p $(BUILD_DIR)
|
||||
$(ODIN) build $(SRC_DIR) \
|
||||
$(COMMON_FLAGS) \
|
||||
$(DEBUG_FLAGS) \
|
||||
-out:$(BUILD_DIR)/$(PROJECT_NAME) \
|
||||
-extra-linker-flags:"$(EXTRA_LINKER_FLAGS)"
|
||||
@echo "$(GREEN)✓ Build complete: $(BUILD_DIR)/$(PROJECT_NAME)$(NC)"
|
||||
|
||||
# Build optimized release version
|
||||
release:
|
||||
@echo "$(BLUE)Building $(PROJECT_NAME) (release)...$(NC)"
|
||||
@mkdir -p $(BUILD_DIR)
|
||||
$(ODIN) build $(SRC_DIR) \
|
||||
$(COMMON_FLAGS) \
|
||||
$(RELEASE_FLAGS) \
|
||||
-out:$(BUILD_DIR)/$(PROJECT_NAME) \
|
||||
-extra-linker-flags:"$(EXTRA_LINKER_FLAGS)"
|
||||
@echo "$(GREEN)✓ Release build complete: $(BUILD_DIR)/$(PROJECT_NAME)$(NC)"
|
||||
|
||||
# Run the server
|
||||
run: build
|
||||
@echo "$(BLUE)Starting $(PROJECT_NAME)...$(NC)"
|
||||
@mkdir -p $(DATA_DIR)
|
||||
@JORMUN_PORT=$(PORT) \
|
||||
JORMUN_HOST=$(HOST) \
|
||||
JORMUN_DATA_DIR=$(DATA_DIR) \
|
||||
JORMUN_VERBOSE=$(VERBOSE) \
|
||||
$(BUILD_DIR)/$(PROJECT_NAME)
|
||||
|
||||
# Run with custom port
|
||||
run-port: build
|
||||
@echo "$(BLUE)Starting $(PROJECT_NAME) on port $(PORT)...$(NC)"
|
||||
@mkdir -p $(DATA_DIR)
|
||||
@JORMUN_PORT=$(PORT) $(BUILD_DIR)/$(PROJECT_NAME)
|
||||
|
||||
# Run tests
|
||||
test:
|
||||
@echo "$(BLUE)Running tests...$(NC)"
|
||||
$(ODIN) test $(SRC_DIR) \
|
||||
$(COMMON_FLAGS) \
|
||||
$(DEBUG_FLAGS) \
|
||||
-extra-linker-flags:"$(EXTRA_LINKER_FLAGS)"
|
||||
@echo "$(GREEN)✓ Tests passed$(NC)"
|
||||
|
||||
# Format code
|
||||
fmt:
|
||||
@echo "$(BLUE)Formatting code...$(NC)"
|
||||
@find $(SRC_DIR) -name "*.odin" -exec odin-format -w {} \;
|
||||
@echo "$(GREEN)✓ Code formatted$(NC)"
|
||||
|
||||
# Clean build artifacts
|
||||
clean:
|
||||
@echo "$(YELLOW)Cleaning build artifacts...$(NC)"
|
||||
@rm -rf $(BUILD_DIR)
|
||||
@rm -rf $(DATA_DIR)
|
||||
@echo "$(GREEN)✓ Clean complete$(NC)"
|
||||
|
||||
# Install to /usr/local/bin (requires sudo)
|
||||
install: release
|
||||
@echo "$(BLUE)Installing $(PROJECT_NAME)...$(NC)"
|
||||
@sudo cp $(BUILD_DIR)/$(PROJECT_NAME) /usr/local/bin/
|
||||
@sudo chmod +x /usr/local/bin/$(PROJECT_NAME)
|
||||
@echo "$(GREEN)✓ Installed to /usr/local/bin/$(PROJECT_NAME)$(NC)"
|
||||
|
||||
# Uninstall from /usr/local/bin
|
||||
uninstall:
|
||||
@echo "$(YELLOW)Uninstalling $(PROJECT_NAME)...$(NC)"
|
||||
@sudo rm -f /usr/local/bin/$(PROJECT_NAME)
|
||||
@echo "$(GREEN)✓ Uninstalled$(NC)"
|
||||
|
||||
# Check dependencies
|
||||
check-deps:
|
||||
@echo "$(BLUE)Checking dependencies...$(NC)"
|
||||
@which $(ODIN) > /dev/null || (echo "$(RED)✗ Odin compiler not found$(NC)" && exit 1)
|
||||
@pkg-config --exists rocksdb || (echo "$(RED)✗ RocksDB not found$(NC)" && exit 1)
|
||||
@echo "$(GREEN)✓ All dependencies found$(NC)"
|
||||
|
||||
# AWS CLI test commands
|
||||
aws-test: run &
|
||||
@sleep 2
|
||||
@echo "$(BLUE)Testing with AWS CLI...$(NC)"
|
||||
@echo "\n$(YELLOW)Creating table...$(NC)"
|
||||
@aws dynamodb create-table \
|
||||
--endpoint-url http://localhost:$(PORT) \
|
||||
--table-name TestTable \
|
||||
--key-schema AttributeName=pk,KeyType=HASH \
|
||||
--attribute-definitions AttributeName=pk,AttributeType=S \
|
||||
--billing-mode PAY_PER_REQUEST || true
|
||||
@echo "\n$(YELLOW)Listing tables...$(NC)"
|
||||
@aws dynamodb list-tables --endpoint-url http://localhost:$(PORT)
|
||||
@echo "\n$(YELLOW)Putting item...$(NC)"
|
||||
@aws dynamodb put-item \
|
||||
--endpoint-url http://localhost:$(PORT) \
|
||||
--table-name TestTable \
|
||||
--item '{"pk":{"S":"test1"},"data":{"S":"hello world"}}'
|
||||
@echo "\n$(YELLOW)Getting item...$(NC)"
|
||||
@aws dynamodb get-item \
|
||||
--endpoint-url http://localhost:$(PORT) \
|
||||
--table-name TestTable \
|
||||
--key '{"pk":{"S":"test1"}}'
|
||||
@echo "\n$(YELLOW)Scanning table...$(NC)"
|
||||
@aws dynamodb scan \
|
||||
--endpoint-url http://localhost:$(PORT) \
|
||||
--table-name TestTable
|
||||
@echo "\n$(GREEN)✓ AWS CLI test complete$(NC)"
|
||||
|
||||
# Development workflow
|
||||
dev: clean build run
|
||||
|
||||
# Quick rebuild and run
|
||||
quick:
|
||||
@$(MAKE) build run
|
||||
|
||||
# Show help
|
||||
help:
|
||||
@echo "$(BLUE)JormunDB - DynamoDB-compatible database$(NC)"
|
||||
@echo ""
|
||||
@echo "$(GREEN)Build Commands:$(NC)"
|
||||
@echo " make build - Build debug version"
|
||||
@echo " make release - Build optimized release version"
|
||||
@echo " make clean - Remove build artifacts"
|
||||
@echo ""
|
||||
@echo "$(GREEN)Run Commands:$(NC)"
|
||||
@echo " make run - Build and run server (default: localhost:8000)"
|
||||
@echo " make run PORT=9000 - Run on custom port"
|
||||
@echo " make dev - Clean, build, and run"
|
||||
@echo " make quick - Fast rebuild and run"
|
||||
@echo ""
|
||||
@echo "$(GREEN)Test Commands:$(NC)"
|
||||
@echo " make test - Run unit tests"
|
||||
@echo " make aws-test - Test with AWS CLI commands"
|
||||
@echo ""
|
||||
@echo "$(GREEN)Utility Commands:$(NC)"
|
||||
@echo " make fmt - Format source code"
|
||||
@echo " make check-deps - Check for required dependencies"
|
||||
@echo " make install - Install to /usr/local/bin (requires sudo)"
|
||||
@echo " make uninstall - Remove from /usr/local/bin"
|
||||
@echo ""
|
||||
@echo "$(GREEN)Configuration:$(NC)"
|
||||
@echo " PORT=$(PORT) - Server port"
|
||||
@echo " HOST=$(HOST) - Bind address"
|
||||
@echo " DATA_DIR=$(DATA_DIR) - RocksDB data directory"
|
||||
@echo " VERBOSE=$(VERBOSE) - Enable verbose logging (0/1)"
|
||||
@echo ""
|
||||
@echo "$(GREEN)Examples:$(NC)"
|
||||
@echo " make run PORT=9000"
|
||||
@echo " make run DATA_DIR=/tmp/jormun VERBOSE=1"
|
||||
@echo " make dev"
|
||||
457
QUICKSTART.md
Normal file
457
QUICKSTART.md
Normal file
@@ -0,0 +1,457 @@
|
||||
# JormunDB Quick Start Guide
|
||||
|
||||
Get JormunDB running in 5 minutes.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### 1. Install Odin
|
||||
|
||||
**macOS:**
|
||||
```bash
|
||||
# Using Homebrew
|
||||
brew install odin
|
||||
|
||||
# Or download from https://odin-lang.org/docs/install/
|
||||
```
|
||||
|
||||
**Ubuntu/Debian:**
|
||||
```bash
|
||||
# Download latest release
|
||||
wget https://github.com/odin-lang/Odin/releases/latest/download/odin-ubuntu-amd64.tar.gz
|
||||
tar -xzf odin-ubuntu-amd64.tar.gz
|
||||
sudo mv odin /usr/local/bin/
|
||||
|
||||
# Verify
|
||||
odin version
|
||||
```
|
||||
|
||||
**From Source:**
|
||||
```bash
|
||||
git clone https://github.com/odin-lang/Odin
|
||||
cd Odin
|
||||
make
|
||||
sudo cp odin /usr/local/bin/
|
||||
```
|
||||
|
||||
### 2. Install RocksDB
|
||||
|
||||
**macOS:**
|
||||
```bash
|
||||
brew install rocksdb
|
||||
```
|
||||
|
||||
**Ubuntu/Debian:**
|
||||
```bash
|
||||
sudo apt update
|
||||
sudo apt install -y librocksdb-dev libsnappy-dev liblz4-dev libzstd-dev libbz2-dev
|
||||
```
|
||||
|
||||
**Arch Linux:**
|
||||
```bash
|
||||
sudo pacman -S rocksdb
|
||||
```
|
||||
|
||||
### 3. Verify Installation
|
||||
|
||||
```bash
|
||||
# Check Odin
|
||||
odin version
|
||||
|
||||
# Check RocksDB
|
||||
pkg-config --libs rocksdb
|
||||
# Should output: -lrocksdb -lstdc++ ...
|
||||
```
|
||||
|
||||
## Building JormunDB
|
||||
|
||||
### Clone and Build
|
||||
|
||||
```bash
|
||||
# Clone the repository
|
||||
git clone https://github.com/yourusername/jormundb.git
|
||||
cd jormundb
|
||||
|
||||
# Build debug version
|
||||
make build
|
||||
|
||||
# Or build optimized release
|
||||
make release
|
||||
```
|
||||
|
||||
### Troubleshooting Build Issues
|
||||
|
||||
**"cannot find rocksdb"**
|
||||
```bash
|
||||
# Check RocksDB installation
|
||||
pkg-config --cflags --libs rocksdb
|
||||
|
||||
# If not found, install RocksDB (see prerequisites)
|
||||
```
|
||||
|
||||
**"odin: command not found"**
|
||||
```bash
|
||||
# Add Odin to PATH
|
||||
export PATH=$PATH:/path/to/odin
|
||||
|
||||
# Or install system-wide (see prerequisites)
|
||||
```
|
||||
|
||||
## Running the Server
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```bash
|
||||
# Run with defaults (localhost:8000, ./data directory)
|
||||
make run
|
||||
```
|
||||
|
||||
You should see:
|
||||
```
|
||||
╔═══════════════════════════════════════════════╗
|
||||
║ ║
|
||||
║ ╦╔═╗╦═╗╔╦╗╦ ╦╔╗╔╔╦╗╔╗ ║
|
||||
║ ║║ ║╠╦╝║║║║ ║║║║ ║║╠╩╗ ║
|
||||
║ ╚╝╚═╝╩╚═╩ ╩╚═╝╝╚╝═╩╝╚═╝ ║
|
||||
║ ║
|
||||
║ DynamoDB-Compatible Database ║
|
||||
║ Powered by RocksDB + Odin ║
|
||||
║ ║
|
||||
╚═══════════════════════════════════════════════╝
|
||||
|
||||
Port: 8000 | Data Dir: ./data
|
||||
|
||||
Storage engine initialized at ./data
|
||||
Starting DynamoDB-compatible server on 0.0.0.0:8000
|
||||
Ready to accept connections!
|
||||
```
|
||||
|
||||
### Custom Configuration
|
||||
|
||||
```bash
|
||||
# Custom port
|
||||
make run PORT=9000
|
||||
|
||||
# Custom data directory
|
||||
make run DATA_DIR=/tmp/jormun
|
||||
|
||||
# Enable verbose logging
|
||||
make run VERBOSE=1
|
||||
|
||||
# Combine options
|
||||
make run PORT=9000 DATA_DIR=/var/jormun VERBOSE=1
|
||||
```
|
||||
|
||||
### Environment Variables
|
||||
|
||||
```bash
|
||||
# Set via environment
|
||||
export JORMUN_PORT=9000
|
||||
export JORMUN_HOST=127.0.0.1
|
||||
export JORMUN_DATA_DIR=/var/jormun
|
||||
make run
|
||||
```
|
||||
|
||||
## Testing with AWS CLI
|
||||
|
||||
### Install AWS CLI
|
||||
|
||||
**macOS:**
|
||||
```bash
|
||||
brew install awscli
|
||||
```
|
||||
|
||||
**Ubuntu/Debian:**
|
||||
```bash
|
||||
sudo apt install awscli
|
||||
```
|
||||
|
||||
**Verify:**
|
||||
```bash
|
||||
aws --version
|
||||
```
|
||||
|
||||
### Configure AWS CLI (for local use)
|
||||
|
||||
```bash
|
||||
# Set dummy credentials (required but not checked by JormunDB)
|
||||
aws configure
|
||||
# AWS Access Key ID: dummy
|
||||
# AWS Secret Access Key: dummy
|
||||
# Default region name: us-east-1
|
||||
# Default output format: json
|
||||
```
|
||||
|
||||
### Basic Operations
|
||||
|
||||
**Create a Table:**
|
||||
```bash
|
||||
aws dynamodb create-table \
|
||||
--endpoint-url http://localhost:8000 \
|
||||
--table-name Users \
|
||||
--key-schema \
|
||||
AttributeName=id,KeyType=HASH \
|
||||
--attribute-definitions \
|
||||
AttributeName=id,AttributeType=S \
|
||||
--billing-mode PAY_PER_REQUEST
|
||||
```
|
||||
|
||||
**List Tables:**
|
||||
```bash
|
||||
aws dynamodb list-tables --endpoint-url http://localhost:8000
|
||||
```
|
||||
|
||||
**Put an Item:**
|
||||
```bash
|
||||
aws dynamodb put-item \
|
||||
--endpoint-url http://localhost:8000 \
|
||||
--table-name Users \
|
||||
--item '{
|
||||
"id": {"S": "user123"},
|
||||
"name": {"S": "Alice"},
|
||||
"age": {"N": "30"},
|
||||
"email": {"S": "alice@example.com"}
|
||||
}'
|
||||
```
|
||||
|
||||
**Get an Item:**
|
||||
```bash
|
||||
aws dynamodb get-item \
|
||||
--endpoint-url http://localhost:8000 \
|
||||
--table-name Users \
|
||||
--key '{"id": {"S": "user123"}}'
|
||||
```
|
||||
|
||||
**Query Items:**
|
||||
```bash
|
||||
aws dynamodb query \
|
||||
--endpoint-url http://localhost:8000 \
|
||||
--table-name Users \
|
||||
--key-condition-expression "id = :id" \
|
||||
--expression-attribute-values '{
|
||||
":id": {"S": "user123"}
|
||||
}'
|
||||
```
|
||||
|
||||
**Scan Table:**
|
||||
```bash
|
||||
aws dynamodb scan \
|
||||
--endpoint-url http://localhost:8000 \
|
||||
--table-name Users
|
||||
```
|
||||
|
||||
**Delete an Item:**
|
||||
```bash
|
||||
aws dynamodb delete-item \
|
||||
--endpoint-url http://localhost:8000 \
|
||||
--table-name Users \
|
||||
--key '{"id": {"S": "user123"}}'
|
||||
```
|
||||
|
||||
**Delete a Table:**
|
||||
```bash
|
||||
aws dynamodb delete-table \
|
||||
--endpoint-url http://localhost:8000 \
|
||||
--table-name Users
|
||||
```
|
||||
|
||||
## Testing with AWS SDK
|
||||
|
||||
### Node.js Example
|
||||
|
||||
```javascript
|
||||
const { DynamoDBClient, PutItemCommand, GetItemCommand } = require("@aws-sdk/client-dynamodb");
|
||||
|
||||
const client = new DynamoDBClient({
|
||||
endpoint: "http://localhost:8000",
|
||||
region: "us-east-1",
|
||||
credentials: {
|
||||
accessKeyId: "dummy",
|
||||
secretAccessKey: "dummy"
|
||||
}
|
||||
});
|
||||
|
||||
async function test() {
|
||||
// Put an item
|
||||
await client.send(new PutItemCommand({
|
||||
TableName: "Users",
|
||||
Item: {
|
||||
id: { S: "user123" },
|
||||
name: { S: "Alice" }
|
||||
}
|
||||
}));
|
||||
|
||||
// Get the item
|
||||
const result = await client.send(new GetItemCommand({
|
||||
TableName: "Users",
|
||||
Key: { id: { S: "user123" } }
|
||||
}));
|
||||
|
||||
console.log(result.Item);
|
||||
}
|
||||
|
||||
test();
|
||||
```
|
||||
|
||||
### Python Example
|
||||
|
||||
```python
|
||||
import boto3
|
||||
|
||||
dynamodb = boto3.client(
|
||||
'dynamodb',
|
||||
endpoint_url='http://localhost:8000',
|
||||
region_name='us-east-1',
|
||||
aws_access_key_id='dummy',
|
||||
aws_secret_access_key='dummy'
|
||||
)
|
||||
|
||||
# Put an item
|
||||
dynamodb.put_item(
|
||||
TableName='Users',
|
||||
Item={
|
||||
'id': {'S': 'user123'},
|
||||
'name': {'S': 'Alice'}
|
||||
}
|
||||
)
|
||||
|
||||
# Get the item
|
||||
response = dynamodb.get_item(
|
||||
TableName='Users',
|
||||
Key={'id': {'S': 'user123'}}
|
||||
)
|
||||
|
||||
print(response['Item'])
|
||||
```
|
||||
|
||||
## Development Workflow
|
||||
|
||||
### Quick Rebuild
|
||||
|
||||
```bash
|
||||
# Fast rebuild and run
|
||||
make quick
|
||||
```
|
||||
|
||||
### Clean Start
|
||||
|
||||
```bash
|
||||
# Remove all build artifacts and data
|
||||
make clean
|
||||
|
||||
# Build and run fresh
|
||||
make dev
|
||||
```
|
||||
|
||||
### Running Tests
|
||||
|
||||
```bash
|
||||
# Run unit tests
|
||||
make test
|
||||
|
||||
# Run AWS CLI integration tests
|
||||
make aws-test
|
||||
```
|
||||
|
||||
### Code Formatting
|
||||
|
||||
```bash
|
||||
# Format all Odin files
|
||||
make fmt
|
||||
```
|
||||
|
||||
## Common Issues
|
||||
|
||||
### Port Already in Use
|
||||
|
||||
```bash
|
||||
# Check what's using port 8000
|
||||
lsof -i :8000
|
||||
|
||||
# Use a different port
|
||||
make run PORT=9000
|
||||
```
|
||||
|
||||
### Cannot Create Data Directory
|
||||
|
||||
```bash
|
||||
# Create with proper permissions
|
||||
mkdir -p ./data
|
||||
chmod 755 ./data
|
||||
|
||||
# Or use a different directory
|
||||
make run DATA_DIR=/tmp/jormun
|
||||
```
|
||||
|
||||
### RocksDB Not Found
|
||||
|
||||
```bash
|
||||
# Check installation
|
||||
pkg-config --libs rocksdb
|
||||
|
||||
# Install if missing (see Prerequisites)
|
||||
```
|
||||
|
||||
### Odin Compiler Errors
|
||||
|
||||
```bash
|
||||
# Check Odin version
|
||||
odin version
|
||||
|
||||
# Update Odin if needed
|
||||
brew upgrade odin # macOS
|
||||
# or download latest from odin-lang.org
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
- Read [ARCHITECTURE.md](ARCHITECTURE.md) for internals
|
||||
- Check [TODO.md](TODO.md) for implementation status
|
||||
- Browse source code in `dynamodb/`, `rocksdb/`, etc.
|
||||
- Contribute! See [CONTRIBUTING.md](CONTRIBUTING.md)
|
||||
|
||||
## Getting Help
|
||||
|
||||
- **Issues**: https://github.com/yourusername/jormundb/issues
|
||||
- **Discussions**: https://github.com/yourusername/jormundb/discussions
|
||||
- **Odin Discord**: https://discord.gg/sVBPHEv
|
||||
|
||||
## Benchmarking
|
||||
|
||||
```bash
|
||||
# Run benchmarks
|
||||
make bench
|
||||
|
||||
# Profile memory usage
|
||||
make profile
|
||||
|
||||
# Load test
|
||||
ab -n 10000 -c 100 -p item.json -T application/json \
|
||||
http://localhost:8000/
|
||||
```
|
||||
|
||||
## Production Deployment
|
||||
|
||||
JormunDB is designed for **local development only**. For production, use:
|
||||
|
||||
- AWS DynamoDB (managed service)
|
||||
- DynamoDB Accelerator (DAX)
|
||||
- ScyllaDB (DynamoDB-compatible)
|
||||
|
||||
## Uninstalling
|
||||
|
||||
```bash
|
||||
# Remove build artifacts
|
||||
make clean
|
||||
|
||||
# Remove installed binary (if installed)
|
||||
make uninstall
|
||||
|
||||
# Remove data directory
|
||||
rm -rf ./data
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Happy coding! 🚀**
|
||||
|
||||
For questions or issues, please open a GitHub issue or join our Discord.
|
||||
317
README.md
Normal file
317
README.md
Normal file
@@ -0,0 +1,317 @@
|
||||
# JormunDB
|
||||
|
||||
A high-performance, DynamoDB-compatible database server written in Odin, backed by RocksDB.
|
||||
|
||||
```
|
||||
╦╔═╗╦═╗╔╦╗╦ ╦╔╗╔╔╦╗╔╗
|
||||
║║ ║╠╦╝║║║║ ║║║║ ║║╠╩╗
|
||||
╚╝╚═╝╩╚═╩ ╩╚═╝╝╚╝═╩╝╚═╝
|
||||
DynamoDB-Compatible Database
|
||||
Powered by RocksDB + Odin
|
||||
```
|
||||
|
||||
## What is JormunDB?
|
||||
|
||||
JormunDB (formerly ZynamoDB) is a local DynamoDB replacement that speaks the DynamoDB wire protocol. Point your AWS SDK or CLI at it and use it as a drop-in development database.
|
||||
|
||||
**Why Odin?** The original Zig implementation suffered from explicit allocator threading—every function taking an `allocator` parameter, every allocation needing `errdefer` cleanup. Odin's implicit context allocator system eliminates this ceremony: one `context.allocator = arena_allocator` at the request handler entry and everything downstream just works.
|
||||
|
||||
## Features
|
||||
|
||||
- ✅ **DynamoDB Wire Protocol**: Works with AWS SDKs and CLI out of the box
|
||||
- ✅ **Binary Storage**: Efficient TLV encoding for items, varint-prefixed keys
|
||||
- ✅ **Arena-per-Request**: Zero explicit memory management in business logic
|
||||
- ✅ **Table Operations**: CreateTable, DeleteTable, DescribeTable, ListTables
|
||||
- ✅ **Item Operations**: PutItem, GetItem, DeleteItem
|
||||
- ✅ **Query & Scan**: With pagination support (Limit, ExclusiveStartKey)
|
||||
- ✅ **Expression Parsing**: KeyConditionExpression for Query operations
|
||||
- ✅ **Persistent Storage**: RocksDB-backed with full ACID guarantees
|
||||
- ✅ **Concurrency**: Table-level RW locks for safe concurrent access
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Odin compiler (latest)
|
||||
- RocksDB development libraries
|
||||
- Standard compression libraries (snappy, lz4, zstd, etc.)
|
||||
|
||||
#### macOS (Homebrew)
|
||||
|
||||
```bash
|
||||
brew install rocksdb odin
|
||||
```
|
||||
|
||||
#### Ubuntu/Debian
|
||||
|
||||
```bash
|
||||
sudo apt install librocksdb-dev libsnappy-dev liblz4-dev libzstd-dev libbz2-dev
|
||||
# Install Odin from https://odin-lang.org/docs/install/
|
||||
```
|
||||
|
||||
### Build & Run
|
||||
|
||||
```bash
|
||||
# Build the server
|
||||
make build
|
||||
|
||||
# Run with default settings (localhost:8000, ./data directory)
|
||||
make run
|
||||
|
||||
# Run with custom port
|
||||
make run PORT=9000
|
||||
|
||||
# Run with custom data directory
|
||||
make run DATA_DIR=/tmp/jormundb
|
||||
```
|
||||
|
||||
### Test with AWS CLI
|
||||
|
||||
```bash
|
||||
# Create a table
|
||||
aws dynamodb create-table \
|
||||
--endpoint-url http://localhost:8000 \
|
||||
--table-name Users \
|
||||
--key-schema AttributeName=id,KeyType=HASH \
|
||||
--attribute-definitions AttributeName=id,AttributeType=S \
|
||||
--billing-mode PAY_PER_REQUEST
|
||||
|
||||
# Put an item
|
||||
aws dynamodb put-item \
|
||||
--endpoint-url http://localhost:8000 \
|
||||
--table-name Users \
|
||||
--item '{"id":{"S":"user123"},"name":{"S":"Alice"},"age":{"N":"30"}}'
|
||||
|
||||
# Get an item
|
||||
aws dynamodb get-item \
|
||||
--endpoint-url http://localhost:8000 \
|
||||
--table-name Users \
|
||||
--key '{"id":{"S":"user123"}}'
|
||||
|
||||
# Query items
|
||||
aws dynamodb query \
|
||||
--endpoint-url http://localhost:8000 \
|
||||
--table-name Users \
|
||||
--key-condition-expression "id = :id" \
|
||||
--expression-attribute-values '{":id":{"S":"user123"}}'
|
||||
|
||||
# Scan table
|
||||
aws dynamodb scan \
|
||||
--endpoint-url http://localhost:8000 \
|
||||
--table-name Users
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
HTTP Request (POST /)
|
||||
↓
|
||||
X-Amz-Target header → Operation routing
|
||||
↓
|
||||
JSON body → DynamoDB types
|
||||
↓
|
||||
Storage engine → RocksDB operations
|
||||
↓
|
||||
Binary encoding → Disk
|
||||
↓
|
||||
JSON response → Client
|
||||
```
|
||||
|
||||
### Module Structure
|
||||
|
||||
```
|
||||
jormundb/
|
||||
├── rocksdb/ - C FFI bindings to librocksdb
|
||||
├── dynamodb/ - Core types and operations
|
||||
│ ├── types.odin - AttributeValue, Item, Key, etc.
|
||||
│ ├── json.odin - DynamoDB JSON serialization
|
||||
│ ├── storage.odin - Storage engine with RocksDB
|
||||
│ └── handler.odin - HTTP request handlers
|
||||
├── key_codec/ - Binary key encoding (varint-prefixed)
|
||||
├── item_codec/ - Binary TLV item encoding
|
||||
└── main.odin - HTTP server and entry point
|
||||
```
|
||||
|
||||
### Storage Format
|
||||
|
||||
**Keys** (varint-length-prefixed segments):
|
||||
```
|
||||
Meta: [0x01][len][table_name]
|
||||
Data: [0x02][len][table_name][len][pk_value][len][sk_value]?
|
||||
GSI: [0x03][len][table_name][len][index_name][len][gsi_pk][len][gsi_sk]?
|
||||
LSI: [0x04][len][table_name][len][index_name][len][pk][len][lsi_sk]
|
||||
```
|
||||
|
||||
**Values** (TLV binary encoding):
|
||||
```
|
||||
[attr_count:varint]
|
||||
[name_len:varint][name:bytes][type_tag:u8][value_encoded:bytes]...
|
||||
|
||||
Type tags:
|
||||
String=0x01, Number=0x02, Binary=0x03, Bool=0x04, Null=0x05
|
||||
SS=0x10, NS=0x11, BS=0x12
|
||||
List=0x20, Map=0x21
|
||||
```
|
||||
|
||||
## Memory Management
|
||||
|
||||
JormunDB uses Odin's context allocator system for elegant memory management:
|
||||
|
||||
```odin
|
||||
// Request handler entry point
|
||||
handle_request :: proc(conn: net.TCP_Socket) {
|
||||
arena: mem.Arena
|
||||
mem.arena_init(&arena, make([]byte, mem.Megabyte * 4))
|
||||
defer mem.arena_destroy(&arena)
|
||||
|
||||
context.allocator = mem.arena_allocator(&arena)
|
||||
|
||||
// Everything below uses the arena automatically
|
||||
// No manual frees, no errdefer cleanup needed
|
||||
request := parse_request() // Uses context.allocator
|
||||
response := process(request) // Uses context.allocator
|
||||
send_response(response) // Uses context.allocator
|
||||
|
||||
// Arena is freed here automatically
|
||||
}
|
||||
```
|
||||
|
||||
Long-lived data (table metadata, locks) uses the default allocator. Request-scoped data uses the arena.
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
# Build debug version
|
||||
make build
|
||||
|
||||
# Build optimized release
|
||||
make release
|
||||
|
||||
# Run tests
|
||||
make test
|
||||
|
||||
# Format code
|
||||
make fmt
|
||||
|
||||
# Clean build artifacts
|
||||
make clean
|
||||
|
||||
# Run with custom settings
|
||||
make run PORT=9000 DATA_DIR=/tmp/db VERBOSE=1
|
||||
```
|
||||
|
||||
## Performance
|
||||
|
||||
From benchmarks on the original Zig version (Odin expected to be similar or better):
|
||||
|
||||
```
|
||||
Sequential Writes | 10000 ops | 245.32 ms | 40765 ops/sec
|
||||
Random Reads | 10000 ops | 312.45 ms | 32006 ops/sec
|
||||
Batch Writes | 10000 ops | 89.23 ms | 112071 ops/sec
|
||||
PutItem | 5000 ops | 892.34 ms | 5604 ops/sec
|
||||
GetItem | 5000 ops | 678.91 ms | 7365 ops/sec
|
||||
Scan (full table) | 5000 ops | 234.56 ms | 21320 ops/sec
|
||||
```
|
||||
|
||||
## API Compatibility
|
||||
|
||||
### Supported Operations
|
||||
|
||||
- ✅ CreateTable
|
||||
- ✅ DeleteTable
|
||||
- ✅ DescribeTable
|
||||
- ✅ ListTables
|
||||
- ✅ PutItem
|
||||
- ✅ GetItem
|
||||
- ✅ DeleteItem
|
||||
- ✅ Query (with KeyConditionExpression)
|
||||
- ✅ Scan (with pagination)
|
||||
|
||||
### Coming Soon
|
||||
|
||||
- ⏳ UpdateItem (with UpdateExpression)
|
||||
- ⏳ BatchWriteItem
|
||||
- ⏳ BatchGetItem
|
||||
- ⏳ Global Secondary Indexes
|
||||
- ⏳ Local Secondary Indexes
|
||||
- ⏳ ConditionExpression
|
||||
- ⏳ FilterExpression
|
||||
- ⏳ ProjectionExpression
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
```bash
|
||||
JORMUN_PORT=8000 # Server port
|
||||
JORMUN_HOST=0.0.0.0 # Bind address
|
||||
JORMUN_DATA_DIR=./data # RocksDB data directory
|
||||
JORMUN_VERBOSE=1 # Enable verbose logging
|
||||
```
|
||||
|
||||
### Command Line Arguments
|
||||
|
||||
```bash
|
||||
./jormundb --port 9000 --host 127.0.0.1 --data-dir /var/db --verbose
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Cannot open RocksDB"
|
||||
|
||||
Ensure RocksDB libraries are installed and the data directory is writable:
|
||||
|
||||
```bash
|
||||
# Check RocksDB installation
|
||||
pkg-config --libs rocksdb
|
||||
|
||||
# Check permissions
|
||||
mkdir -p ./data
|
||||
chmod 755 ./data
|
||||
```
|
||||
|
||||
### "Connection refused"
|
||||
|
||||
Check if the port is already in use:
|
||||
|
||||
```bash
|
||||
lsof -i :8000
|
||||
```
|
||||
|
||||
### "Invalid JSON" errors
|
||||
|
||||
Ensure you're using the correct DynamoDB JSON format:
|
||||
|
||||
```json
|
||||
{
|
||||
"TableName": "Users",
|
||||
"Item": {
|
||||
"id": {"S": "user123"},
|
||||
"age": {"N": "30"}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
MIT License - see LICENSE file for details.
|
||||
|
||||
## Credits
|
||||
|
||||
- Inspired by DynamoDB Local
|
||||
- Built with [Odin](https://odin-lang.org/)
|
||||
- Powered by [RocksDB](https://rocksdb.org/)
|
||||
- Originally implemented as ZynamoDB in Zig
|
||||
|
||||
## Contributing
|
||||
|
||||
Contributions welcome! Please:
|
||||
|
||||
1. Format code with `make fmt`
|
||||
2. Run tests with `make test`
|
||||
3. Update documentation as needed
|
||||
4. Follow Odin idioms (context allocators, explicit returns, etc.)
|
||||
|
||||
---
|
||||
|
||||
**Why "Jormun"?** Jörmungandr, the World Serpent from Norse mythology—a fitting name for something that wraps around your data. Also, it sounds cool.
|
||||
186
TODO.md
Normal file
186
TODO.md
Normal file
@@ -0,0 +1,186 @@
|
||||
# JormunDB Implementation TODO
|
||||
|
||||
This tracks the rewrite from Zig to Odin and remaining features.
|
||||
|
||||
## ✅ Completed
|
||||
|
||||
- [x] Project structure
|
||||
- [x] Makefile with build/run/test targets
|
||||
- [x] README with usage instructions
|
||||
- [x] ARCHITECTURE documentation
|
||||
- [x] RocksDB FFI bindings (rocksdb/rocksdb.odin)
|
||||
- [x] Core types (dynamodb/types.odin)
|
||||
- [x] Key codec with varint encoding (key_codec/key_codec.odin)
|
||||
- [x] Main entry point with arena pattern demo
|
||||
- [x] LICENSE file
|
||||
- [x] .gitignore
|
||||
|
||||
## 🚧 In Progress (Need to Complete)
|
||||
|
||||
### Core Modules
|
||||
|
||||
- [ ] **dynamodb/json.odin** - DynamoDB JSON parsing and serialization
|
||||
- Parse `{"S": "value"}` format
|
||||
- Serialize AttributeValue to DynamoDB JSON
|
||||
- Parse request bodies (PutItem, GetItem, etc.)
|
||||
|
||||
- [ ] **item_codec/item_codec.odin** - Binary TLV encoding for items
|
||||
- Encode Item to binary TLV format
|
||||
- Decode binary TLV back to Item
|
||||
- Type tag handling for all DynamoDB types
|
||||
|
||||
- [ ] **dynamodb/storage.odin** - Storage engine with RocksDB
|
||||
- Table metadata management
|
||||
- create_table, delete_table, describe_table, list_tables
|
||||
- put_item, get_item, delete_item
|
||||
- scan, query with pagination
|
||||
- Table-level RW locks
|
||||
|
||||
- [ ] **dynamodb/handler.odin** - HTTP request handlers
|
||||
- Route X-Amz-Target to handler functions
|
||||
- handle_create_table, handle_put_item, etc.
|
||||
- Build responses with proper error handling
|
||||
- Arena allocator integration
|
||||
|
||||
### HTTP Server
|
||||
|
||||
- [ ] **HTTP server implementation**
|
||||
- Accept TCP connections
|
||||
- Parse HTTP POST requests
|
||||
- Read JSON bodies
|
||||
- Send HTTP responses with headers
|
||||
- Keep-alive support
|
||||
- Options:
|
||||
- Use `core:net` directly
|
||||
- Use C FFI with libmicrohttpd
|
||||
- Use Odin's vendor:microui (if suitable)
|
||||
|
||||
### Expression Parsers (Priority 3)
|
||||
|
||||
- [ ] **KeyConditionExpression parser**
|
||||
- Tokenizer for expressions
|
||||
- Parse `pk = :pk AND sk > :sk`
|
||||
- Support begins_with, BETWEEN
|
||||
- ExpressionAttributeNames/Values
|
||||
|
||||
- [ ] **UpdateExpression parser** (later)
|
||||
- SET operations
|
||||
- REMOVE operations
|
||||
- ADD operations
|
||||
- DELETE operations
|
||||
|
||||
## 📋 Testing
|
||||
|
||||
- [ ] Unit tests for key_codec
|
||||
- [ ] Unit tests for item_codec
|
||||
- [ ] Unit tests for JSON parsing
|
||||
- [ ] Integration tests with AWS CLI
|
||||
- [ ] Benchmark suite
|
||||
|
||||
## 🔧 Build & Tooling
|
||||
|
||||
- [ ] Verify Makefile works on macOS
|
||||
- [ ] Verify Makefile works on Linux
|
||||
- [ ] Add Docker support (optional)
|
||||
- [ ] Add install script
|
||||
|
||||
## 📚 Documentation
|
||||
|
||||
- [ ] Code comments for public APIs
|
||||
- [ ] Usage examples in README
|
||||
- [ ] API compatibility matrix
|
||||
- [ ] Performance tuning guide
|
||||
|
||||
## 🎯 Priority Order
|
||||
|
||||
1. **HTTP Server** - Need this to accept requests
|
||||
2. **JSON Parsing** - Need this to understand DynamoDB format
|
||||
3. **Storage Engine** - Core CRUD operations
|
||||
4. **Handlers** - Wire everything together
|
||||
5. **Item Codec** - Efficient binary storage
|
||||
6. **Expression Parsers** - Query functionality
|
||||
|
||||
## 📝 Notes
|
||||
|
||||
### Zig → Odin Translation Patterns
|
||||
|
||||
**Memory Management:**
|
||||
```zig
|
||||
// Zig
|
||||
const item = try allocator.create(Item);
|
||||
defer allocator.destroy(item);
|
||||
```
|
||||
```odin
|
||||
// Odin
|
||||
item := new(Item)
|
||||
// No defer needed if using arena
|
||||
```
|
||||
|
||||
**Error Handling:**
|
||||
```zig
|
||||
// Zig
|
||||
fn foo() !Result {
|
||||
return error.Failed;
|
||||
}
|
||||
const x = try foo();
|
||||
```
|
||||
```odin
|
||||
// Odin
|
||||
foo :: proc() -> (Result, bool) {
|
||||
return {}, false
|
||||
}
|
||||
x := foo() or_return
|
||||
```
|
||||
|
||||
**Slices:**
|
||||
```zig
|
||||
// Zig
|
||||
const slice: []const u8 = data;
|
||||
```
|
||||
```odin
|
||||
// Odin
|
||||
slice: []byte = data
|
||||
```
|
||||
|
||||
**Maps:**
|
||||
```zig
|
||||
// Zig
|
||||
var map = std.StringHashMap(Value).init(allocator);
|
||||
defer map.deinit();
|
||||
```
|
||||
```odin
|
||||
// Odin
|
||||
map := make(map[string]Value)
|
||||
defer delete(map)
|
||||
```
|
||||
|
||||
### Key Decisions
|
||||
|
||||
1. **Use `Maybe(T)` instead of `?T`** - Odin's optional type
|
||||
2. **Use `or_return` instead of `try`** - Odin's error propagation
|
||||
3. **Use `context.allocator`** - Implicit allocator from context
|
||||
4. **Use `#partial switch`** - For union type checking
|
||||
5. **Use `transmute`** - For zero-cost type conversions
|
||||
|
||||
### Reference Zig Files
|
||||
|
||||
When implementing, reference these Zig files:
|
||||
- `src/dynamodb/json.zig` - 400 lines, DynamoDB JSON format
|
||||
- `src/dynamodb/storage.zig` - 460 lines, storage engine
|
||||
- `src/dynamodb/handler.zig` - 500+ lines, request handlers
|
||||
- `src/item_codec.zig` - 350 lines, TLV encoding
|
||||
- `src/http.zig` - 250 lines, HTTP server
|
||||
|
||||
### Quick Test Commands
|
||||
|
||||
```bash
|
||||
# Build and test
|
||||
make build
|
||||
make test
|
||||
|
||||
# Run server
|
||||
make run
|
||||
|
||||
# Test with AWS CLI
|
||||
aws dynamodb list-tables --endpoint-url http://localhost:8000
|
||||
```
|
||||
BIN
build/jormundb
Executable file
BIN
build/jormundb
Executable file
Binary file not shown.
90
concat_project.sh
Executable file
90
concat_project.sh
Executable file
@@ -0,0 +1,90 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Output file
|
||||
OUTPUT_FILE="project_context.txt"
|
||||
|
||||
# Directories to exclude
|
||||
EXCLUDE_DIRS=("odin-out" "data" ".git" "node_modules" ".odin-cache" "tests")
|
||||
|
||||
# File extensions to include (add more as needed)
|
||||
INCLUDE_EXTENSIONS=("odin" "Makefile" "md")
|
||||
|
||||
# Special files to include (without extension)
|
||||
INCLUDE_FILES=("ols.json" "Makefile" "build.odin.zon")
|
||||
|
||||
# Clear the output file
|
||||
> "$OUTPUT_FILE"
|
||||
|
||||
# Function to check if directory should be excluded
|
||||
should_exclude_dir() {
|
||||
local dir="$1"
|
||||
for exclude in "${EXCLUDE_DIRS[@]}"; do
|
||||
if [[ "$dir" == *"/$exclude"* ]] || [[ "$dir" == "$exclude"* ]]; then
|
||||
return 0
|
||||
fi
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
# Function to check if file should be included
|
||||
should_include_file() {
|
||||
local file="$1"
|
||||
local basename=$(basename "$file")
|
||||
|
||||
# Check if it's in the special files list
|
||||
for special in "${INCLUDE_FILES[@]}"; do
|
||||
if [[ "$basename" == "$special" ]]; then
|
||||
return 0
|
||||
fi
|
||||
done
|
||||
|
||||
# Check extension
|
||||
local ext="${file##*.}"
|
||||
for include_ext in "${INCLUDE_EXTENSIONS[@]}"; do
|
||||
if [[ "$ext" == "$include_ext" ]]; then
|
||||
return 0
|
||||
fi
|
||||
done
|
||||
|
||||
return 1
|
||||
}
|
||||
|
||||
# Add header
|
||||
echo "# Project: jormun-db" >> "$OUTPUT_FILE"
|
||||
echo "# Generated: $(date)" >> "$OUTPUT_FILE"
|
||||
echo "" >> "$OUTPUT_FILE"
|
||||
echo "================================================================================" >> "$OUTPUT_FILE"
|
||||
echo "" >> "$OUTPUT_FILE"
|
||||
|
||||
# Find and concatenate files
|
||||
while IFS= read -r -d '' file; do
|
||||
# Get directory path
|
||||
dir=$(dirname "$file")
|
||||
|
||||
# Skip excluded directories
|
||||
if should_exclude_dir "$dir"; then
|
||||
continue
|
||||
fi
|
||||
|
||||
# Check if file should be included
|
||||
if should_include_file "$file"; then
|
||||
echo "Adding: $file"
|
||||
|
||||
# Add file delimiter
|
||||
echo "================================================================================" >> "$OUTPUT_FILE"
|
||||
echo "FILE: $file" >> "$OUTPUT_FILE"
|
||||
echo "================================================================================" >> "$OUTPUT_FILE"
|
||||
echo "" >> "$OUTPUT_FILE"
|
||||
|
||||
# Add file contents
|
||||
cat "$file" >> "$OUTPUT_FILE"
|
||||
|
||||
# Add spacing
|
||||
echo "" >> "$OUTPUT_FILE"
|
||||
echo "" >> "$OUTPUT_FILE"
|
||||
fi
|
||||
done < <(find . -type f -print0 | sort -z)
|
||||
|
||||
echo ""
|
||||
echo "Done! Output written to: $OUTPUT_FILE"
|
||||
echo "File size: $(du -h "$OUTPUT_FILE" | cut -f1)"
|
||||
0
data/000004.log
Normal file
0
data/000004.log
Normal file
1
data/CURRENT
Normal file
1
data/CURRENT
Normal file
@@ -0,0 +1 @@
|
||||
MANIFEST-000005
|
||||
1
data/IDENTITY
Normal file
1
data/IDENTITY
Normal file
@@ -0,0 +1 @@
|
||||
8dff41c8-9c17-41a7-bcc0-29dc39228555
|
||||
437
data/LOG
Normal file
437
data/LOG
Normal file
@@ -0,0 +1,437 @@
|
||||
2026/02/15-08:54:06.511816 414231 RocksDB version: 10.9.1
|
||||
2026/02/15-08:54:06.511863 414231 Git sha 5fbc1cd5bcf63782675168b98e114151490de6d9
|
||||
2026/02/15-08:54:06.511865 414231 Compile date 2026-01-06 12:13:12
|
||||
2026/02/15-08:54:06.511866 414231 DB SUMMARY
|
||||
2026/02/15-08:54:06.511867 414231 Host name (Env): arch
|
||||
2026/02/15-08:54:06.511868 414231 DB Session ID: Q7XLE4A3DKSF5H391J6S
|
||||
2026/02/15-08:54:06.511889 414231 SST files in ./data dir, Total Num: 0, files:
|
||||
2026/02/15-08:54:06.511893 414231 Write Ahead Log file in ./data:
|
||||
2026/02/15-08:54:06.511894 414231 Options.error_if_exists: 0
|
||||
2026/02/15-08:54:06.511895 414231 Options.create_if_missing: 1
|
||||
2026/02/15-08:54:06.511896 414231 Options.paranoid_checks: 1
|
||||
2026/02/15-08:54:06.511896 414231 Options.flush_verify_memtable_count: 1
|
||||
2026/02/15-08:54:06.511897 414231 Options.compaction_verify_record_count: 1
|
||||
2026/02/15-08:54:06.511898 414231 Options.track_and_verify_wals_in_manifest: 0
|
||||
2026/02/15-08:54:06.511898 414231 Options.track_and_verify_wals: 0
|
||||
2026/02/15-08:54:06.511899 414231 Options.verify_sst_unique_id_in_manifest: 1
|
||||
2026/02/15-08:54:06.511900 414231 Options.env: 0x3a7859c0
|
||||
2026/02/15-08:54:06.511900 414231 Options.fs: PosixFileSystem
|
||||
2026/02/15-08:54:06.511901 414231 Options.info_log: 0x3a78ff90
|
||||
2026/02/15-08:54:06.511902 414231 Options.max_file_opening_threads: 16
|
||||
2026/02/15-08:54:06.511902 414231 Options.statistics: (nil)
|
||||
2026/02/15-08:54:06.511904 414231 Options.use_fsync: 0
|
||||
2026/02/15-08:54:06.511904 414231 Options.max_log_file_size: 0
|
||||
2026/02/15-08:54:06.511905 414231 Options.log_file_time_to_roll: 0
|
||||
2026/02/15-08:54:06.511905 414231 Options.keep_log_file_num: 1000
|
||||
2026/02/15-08:54:06.511906 414231 Options.recycle_log_file_num: 0
|
||||
2026/02/15-08:54:06.511907 414231 Options.allow_fallocate: 1
|
||||
2026/02/15-08:54:06.511907 414231 Options.allow_mmap_reads: 0
|
||||
2026/02/15-08:54:06.511908 414231 Options.allow_mmap_writes: 0
|
||||
2026/02/15-08:54:06.511909 414231 Options.use_direct_reads: 0
|
||||
2026/02/15-08:54:06.511909 414231 Options.use_direct_io_for_flush_and_compaction: 0
|
||||
2026/02/15-08:54:06.511910 414231 Options.create_missing_column_families: 0
|
||||
2026/02/15-08:54:06.511910 414231 Options.db_log_dir:
|
||||
2026/02/15-08:54:06.511911 414231 Options.wal_dir:
|
||||
2026/02/15-08:54:06.511912 414231 Options.table_cache_numshardbits: 6
|
||||
2026/02/15-08:54:06.511912 414231 Options.WAL_ttl_seconds: 0
|
||||
2026/02/15-08:54:06.511913 414231 Options.WAL_size_limit_MB: 0
|
||||
2026/02/15-08:54:06.511914 414231 Options.max_write_batch_group_size_bytes: 1048576
|
||||
2026/02/15-08:54:06.511914 414231 Options.is_fd_close_on_exec: 1
|
||||
2026/02/15-08:54:06.511915 414231 Options.advise_random_on_open: 1
|
||||
2026/02/15-08:54:06.511915 414231 Options.db_write_buffer_size: 0
|
||||
2026/02/15-08:54:06.511916 414231 Options.write_buffer_manager: 0x3a790180
|
||||
2026/02/15-08:54:06.511917 414231 Options.use_adaptive_mutex: 0
|
||||
2026/02/15-08:54:06.511917 414231 Options.rate_limiter: (nil)
|
||||
2026/02/15-08:54:06.511918 414231 Options.sst_file_manager.rate_bytes_per_sec: 0
|
||||
2026/02/15-08:54:06.511919 414231 Options.wal_recovery_mode: 2
|
||||
2026/02/15-08:54:06.511919 414231 Options.enable_thread_tracking: 0
|
||||
2026/02/15-08:54:06.511920 414231 Options.enable_pipelined_write: 0
|
||||
2026/02/15-08:54:06.511921 414231 Options.unordered_write: 0
|
||||
2026/02/15-08:54:06.511921 414231 Options.allow_concurrent_memtable_write: 1
|
||||
2026/02/15-08:54:06.511923 414231 Options.enable_write_thread_adaptive_yield: 1
|
||||
2026/02/15-08:54:06.511924 414231 Options.write_thread_max_yield_usec: 100
|
||||
2026/02/15-08:54:06.511924 414231 Options.write_thread_slow_yield_usec: 3
|
||||
2026/02/15-08:54:06.511925 414231 Options.row_cache: None
|
||||
2026/02/15-08:54:06.511926 414231 Options.wal_filter: None
|
||||
2026/02/15-08:54:06.511926 414231 Options.avoid_flush_during_recovery: 0
|
||||
2026/02/15-08:54:06.511927 414231 Options.allow_ingest_behind: 0
|
||||
2026/02/15-08:54:06.511928 414231 Options.two_write_queues: 0
|
||||
2026/02/15-08:54:06.511928 414231 Options.manual_wal_flush: 0
|
||||
2026/02/15-08:54:06.511929 414231 Options.wal_compression: 0
|
||||
2026/02/15-08:54:06.511929 414231 Options.background_close_inactive_wals: 0
|
||||
2026/02/15-08:54:06.511930 414231 Options.atomic_flush: 0
|
||||
2026/02/15-08:54:06.511931 414231 Options.avoid_unnecessary_blocking_io: 0
|
||||
2026/02/15-08:54:06.511931 414231 Options.prefix_seek_opt_in_only: 0
|
||||
2026/02/15-08:54:06.511932 414231 Options.persist_stats_to_disk: 0
|
||||
2026/02/15-08:54:06.511932 414231 Options.write_dbid_to_manifest: 1
|
||||
2026/02/15-08:54:06.511933 414231 Options.write_identity_file: 1
|
||||
2026/02/15-08:54:06.511934 414231 Options.log_readahead_size: 0
|
||||
2026/02/15-08:54:06.511934 414231 Options.file_checksum_gen_factory: Unknown
|
||||
2026/02/15-08:54:06.511935 414231 Options.best_efforts_recovery: 0
|
||||
2026/02/15-08:54:06.511935 414231 Options.max_bgerror_resume_count: 2147483647
|
||||
2026/02/15-08:54:06.511936 414231 Options.bgerror_resume_retry_interval: 1000000
|
||||
2026/02/15-08:54:06.511937 414231 Options.allow_data_in_errors: 0
|
||||
2026/02/15-08:54:06.511937 414231 Options.db_host_id: __hostname__
|
||||
2026/02/15-08:54:06.511938 414231 Options.enforce_single_del_contracts: true
|
||||
2026/02/15-08:54:06.511939 414231 Options.metadata_write_temperature: kUnknown
|
||||
2026/02/15-08:54:06.511940 414231 Options.wal_write_temperature: kUnknown
|
||||
2026/02/15-08:54:06.511940 414231 Options.max_background_jobs: 4
|
||||
2026/02/15-08:54:06.511941 414231 Options.max_background_compactions: -1
|
||||
2026/02/15-08:54:06.511942 414231 Options.max_subcompactions: 1
|
||||
2026/02/15-08:54:06.511942 414231 Options.avoid_flush_during_shutdown: 0
|
||||
2026/02/15-08:54:06.511943 414231 Options.writable_file_max_buffer_size: 1048576
|
||||
2026/02/15-08:54:06.511944 414231 Options.delayed_write_rate : 16777216
|
||||
2026/02/15-08:54:06.511944 414231 Options.max_total_wal_size: 0
|
||||
2026/02/15-08:54:06.511945 414231 Options.delete_obsolete_files_period_micros: 21600000000
|
||||
2026/02/15-08:54:06.511946 414231 Options.stats_dump_period_sec: 600
|
||||
2026/02/15-08:54:06.511946 414231 Options.stats_persist_period_sec: 600
|
||||
2026/02/15-08:54:06.511947 414231 Options.stats_history_buffer_size: 1048576
|
||||
2026/02/15-08:54:06.511947 414231 Options.max_open_files: -1
|
||||
2026/02/15-08:54:06.511948 414231 Options.bytes_per_sync: 0
|
||||
2026/02/15-08:54:06.511949 414231 Options.wal_bytes_per_sync: 0
|
||||
2026/02/15-08:54:06.511949 414231 Options.strict_bytes_per_sync: 0
|
||||
2026/02/15-08:54:06.511950 414231 Options.compaction_readahead_size: 2097152
|
||||
2026/02/15-08:54:06.511951 414231 Options.max_background_flushes: -1
|
||||
2026/02/15-08:54:06.511951 414231 Options.max_manifest_file_size: 1073741824
|
||||
2026/02/15-08:54:06.511952 414231 Options.max_manifest_space_amp_pct: 500
|
||||
2026/02/15-08:54:06.511952 414231 Options.manifest_preallocation_size: 4194304
|
||||
2026/02/15-08:54:06.511953 414231 Options.daily_offpeak_time_utc:
|
||||
2026/02/15-08:54:06.511954 414231 Compression algorithms supported:
|
||||
2026/02/15-08:54:06.511955 414231 kCustomCompressionFE supported: 0
|
||||
2026/02/15-08:54:06.511956 414231 kCustomCompressionFC supported: 0
|
||||
2026/02/15-08:54:06.511957 414231 kCustomCompressionF8 supported: 0
|
||||
2026/02/15-08:54:06.511958 414231 kCustomCompressionF7 supported: 0
|
||||
2026/02/15-08:54:06.511958 414231 kCustomCompressionB2 supported: 0
|
||||
2026/02/15-08:54:06.511959 414231 kLZ4Compression supported: 1
|
||||
2026/02/15-08:54:06.511960 414231 kCustomCompression88 supported: 0
|
||||
2026/02/15-08:54:06.511960 414231 kCustomCompressionD8 supported: 0
|
||||
2026/02/15-08:54:06.511961 414231 kCustomCompression9F supported: 0
|
||||
2026/02/15-08:54:06.511961 414231 kCustomCompressionD6 supported: 0
|
||||
2026/02/15-08:54:06.511962 414231 kCustomCompressionA9 supported: 0
|
||||
2026/02/15-08:54:06.511963 414231 kCustomCompressionEC supported: 0
|
||||
2026/02/15-08:54:06.511964 414231 kCustomCompressionA3 supported: 0
|
||||
2026/02/15-08:54:06.511964 414231 kCustomCompressionCB supported: 0
|
||||
2026/02/15-08:54:06.511965 414231 kCustomCompression90 supported: 0
|
||||
2026/02/15-08:54:06.511966 414231 kCustomCompressionA0 supported: 0
|
||||
2026/02/15-08:54:06.511966 414231 kCustomCompressionC6 supported: 0
|
||||
2026/02/15-08:54:06.511967 414231 kCustomCompression9D supported: 0
|
||||
2026/02/15-08:54:06.511967 414231 kCustomCompression8B supported: 0
|
||||
2026/02/15-08:54:06.511968 414231 kCustomCompressionA8 supported: 0
|
||||
2026/02/15-08:54:06.511969 414231 kCustomCompression8D supported: 0
|
||||
2026/02/15-08:54:06.511969 414231 kCustomCompression97 supported: 0
|
||||
2026/02/15-08:54:06.511970 414231 kCustomCompression98 supported: 0
|
||||
2026/02/15-08:54:06.511971 414231 kCustomCompressionAC supported: 0
|
||||
2026/02/15-08:54:06.511971 414231 kCustomCompressionE9 supported: 0
|
||||
2026/02/15-08:54:06.511972 414231 kCustomCompression96 supported: 0
|
||||
2026/02/15-08:54:06.511973 414231 kCustomCompressionB1 supported: 0
|
||||
2026/02/15-08:54:06.511973 414231 kCustomCompression95 supported: 0
|
||||
2026/02/15-08:54:06.511974 414231 kCustomCompression84 supported: 0
|
||||
2026/02/15-08:54:06.511975 414231 kCustomCompression91 supported: 0
|
||||
2026/02/15-08:54:06.511975 414231 kCustomCompressionAB supported: 0
|
||||
2026/02/15-08:54:06.511976 414231 kCustomCompressionB3 supported: 0
|
||||
2026/02/15-08:54:06.511976 414231 kCustomCompression81 supported: 0
|
||||
2026/02/15-08:54:06.511977 414231 kCustomCompressionDC supported: 0
|
||||
2026/02/15-08:54:06.511978 414231 kBZip2Compression supported: 1
|
||||
2026/02/15-08:54:06.511978 414231 kCustomCompressionBB supported: 0
|
||||
2026/02/15-08:54:06.511979 414231 kCustomCompression9C supported: 0
|
||||
2026/02/15-08:54:06.511980 414231 kCustomCompressionC9 supported: 0
|
||||
2026/02/15-08:54:06.511980 414231 kCustomCompressionCC supported: 0
|
||||
2026/02/15-08:54:06.511981 414231 kCustomCompression92 supported: 0
|
||||
2026/02/15-08:54:06.511981 414231 kCustomCompressionB9 supported: 0
|
||||
2026/02/15-08:54:06.511982 414231 kCustomCompression8F supported: 0
|
||||
2026/02/15-08:54:06.511983 414231 kCustomCompression8A supported: 0
|
||||
2026/02/15-08:54:06.511983 414231 kCustomCompression9B supported: 0
|
||||
2026/02/15-08:54:06.511984 414231 kZSTD supported: 1
|
||||
2026/02/15-08:54:06.511985 414231 kCustomCompressionAA supported: 0
|
||||
2026/02/15-08:54:06.511985 414231 kCustomCompressionA2 supported: 0
|
||||
2026/02/15-08:54:06.511986 414231 kZlibCompression supported: 1
|
||||
2026/02/15-08:54:06.511986 414231 kXpressCompression supported: 0
|
||||
2026/02/15-08:54:06.511987 414231 kCustomCompressionFD supported: 0
|
||||
2026/02/15-08:54:06.511988 414231 kCustomCompressionE2 supported: 0
|
||||
2026/02/15-08:54:06.511988 414231 kLZ4HCCompression supported: 1
|
||||
2026/02/15-08:54:06.511989 414231 kCustomCompressionA6 supported: 0
|
||||
2026/02/15-08:54:06.511990 414231 kCustomCompression85 supported: 0
|
||||
2026/02/15-08:54:06.511990 414231 kCustomCompressionA4 supported: 0
|
||||
2026/02/15-08:54:06.511991 414231 kCustomCompression86 supported: 0
|
||||
2026/02/15-08:54:06.511992 414231 kCustomCompression83 supported: 0
|
||||
2026/02/15-08:54:06.511992 414231 kCustomCompression87 supported: 0
|
||||
2026/02/15-08:54:06.511993 414231 kCustomCompression89 supported: 0
|
||||
2026/02/15-08:54:06.511994 414231 kCustomCompression8C supported: 0
|
||||
2026/02/15-08:54:06.511995 414231 kCustomCompressionDB supported: 0
|
||||
2026/02/15-08:54:06.512022 414231 kCustomCompressionF3 supported: 0
|
||||
2026/02/15-08:54:06.512024 414231 kCustomCompressionE6 supported: 0
|
||||
2026/02/15-08:54:06.512024 414231 kCustomCompression8E supported: 0
|
||||
2026/02/15-08:54:06.512025 414231 kCustomCompressionDA supported: 0
|
||||
2026/02/15-08:54:06.512025 414231 kCustomCompression93 supported: 0
|
||||
2026/02/15-08:54:06.512026 414231 kCustomCompression94 supported: 0
|
||||
2026/02/15-08:54:06.512027 414231 kCustomCompression9E supported: 0
|
||||
2026/02/15-08:54:06.512027 414231 kCustomCompressionB4 supported: 0
|
||||
2026/02/15-08:54:06.512028 414231 kCustomCompressionFB supported: 0
|
||||
2026/02/15-08:54:06.512029 414231 kCustomCompressionB5 supported: 0
|
||||
2026/02/15-08:54:06.512030 414231 kCustomCompressionD5 supported: 0
|
||||
2026/02/15-08:54:06.512030 414231 kCustomCompressionB8 supported: 0
|
||||
2026/02/15-08:54:06.512031 414231 kCustomCompressionD1 supported: 0
|
||||
2026/02/15-08:54:06.512031 414231 kCustomCompressionBA supported: 0
|
||||
2026/02/15-08:54:06.512032 414231 kCustomCompressionBC supported: 0
|
||||
2026/02/15-08:54:06.512033 414231 kCustomCompressionCE supported: 0
|
||||
2026/02/15-08:54:06.512033 414231 kCustomCompressionBD supported: 0
|
||||
2026/02/15-08:54:06.512034 414231 kCustomCompressionC4 supported: 0
|
||||
2026/02/15-08:54:06.512035 414231 kCustomCompression9A supported: 0
|
||||
2026/02/15-08:54:06.512035 414231 kCustomCompression99 supported: 0
|
||||
2026/02/15-08:54:06.512036 414231 kCustomCompressionBE supported: 0
|
||||
2026/02/15-08:54:06.512053 414231 kCustomCompressionE5 supported: 0
|
||||
2026/02/15-08:54:06.512054 414231 kCustomCompressionD9 supported: 0
|
||||
2026/02/15-08:54:06.512055 414231 kCustomCompressionC1 supported: 0
|
||||
2026/02/15-08:54:06.512055 414231 kCustomCompressionC5 supported: 0
|
||||
2026/02/15-08:54:06.512056 414231 kCustomCompressionC2 supported: 0
|
||||
2026/02/15-08:54:06.512057 414231 kCustomCompressionA5 supported: 0
|
||||
2026/02/15-08:54:06.512057 414231 kCustomCompressionC7 supported: 0
|
||||
2026/02/15-08:54:06.512058 414231 kCustomCompressionBF supported: 0
|
||||
2026/02/15-08:54:06.512058 414231 kCustomCompressionE8 supported: 0
|
||||
2026/02/15-08:54:06.512059 414231 kCustomCompressionC8 supported: 0
|
||||
2026/02/15-08:54:06.512060 414231 kCustomCompressionAF supported: 0
|
||||
2026/02/15-08:54:06.512060 414231 kCustomCompressionCA supported: 0
|
||||
2026/02/15-08:54:06.512061 414231 kCustomCompressionCD supported: 0
|
||||
2026/02/15-08:54:06.512061 414231 kCustomCompressionC0 supported: 0
|
||||
2026/02/15-08:54:06.512062 414231 kCustomCompressionCF supported: 0
|
||||
2026/02/15-08:54:06.512063 414231 kCustomCompressionF9 supported: 0
|
||||
2026/02/15-08:54:06.512063 414231 kCustomCompressionD0 supported: 0
|
||||
2026/02/15-08:54:06.512064 414231 kCustomCompressionD2 supported: 0
|
||||
2026/02/15-08:54:06.512064 414231 kCustomCompressionAD supported: 0
|
||||
2026/02/15-08:54:06.512065 414231 kCustomCompressionD3 supported: 0
|
||||
2026/02/15-08:54:06.512066 414231 kCustomCompressionD4 supported: 0
|
||||
2026/02/15-08:54:06.512066 414231 kCustomCompressionD7 supported: 0
|
||||
2026/02/15-08:54:06.512067 414231 kCustomCompression82 supported: 0
|
||||
2026/02/15-08:54:06.512068 414231 kCustomCompressionDD supported: 0
|
||||
2026/02/15-08:54:06.512068 414231 kCustomCompressionC3 supported: 0
|
||||
2026/02/15-08:54:06.512069 414231 kCustomCompressionEE supported: 0
|
||||
2026/02/15-08:54:06.512070 414231 kCustomCompressionDE supported: 0
|
||||
2026/02/15-08:54:06.512070 414231 kCustomCompressionDF supported: 0
|
||||
2026/02/15-08:54:06.512071 414231 kCustomCompressionA7 supported: 0
|
||||
2026/02/15-08:54:06.512071 414231 kCustomCompressionE0 supported: 0
|
||||
2026/02/15-08:54:06.512072 414231 kCustomCompressionF1 supported: 0
|
||||
2026/02/15-08:54:06.512073 414231 kCustomCompressionE1 supported: 0
|
||||
2026/02/15-08:54:06.512073 414231 kCustomCompressionF5 supported: 0
|
||||
2026/02/15-08:54:06.512074 414231 kCustomCompression80 supported: 0
|
||||
2026/02/15-08:54:06.512075 414231 kCustomCompressionE3 supported: 0
|
||||
2026/02/15-08:54:06.512075 414231 kCustomCompressionE4 supported: 0
|
||||
2026/02/15-08:54:06.512077 414231 kCustomCompressionB0 supported: 0
|
||||
2026/02/15-08:54:06.512077 414231 kCustomCompressionEA supported: 0
|
||||
2026/02/15-08:54:06.512078 414231 kCustomCompressionFA supported: 0
|
||||
2026/02/15-08:54:06.512079 414231 kCustomCompressionE7 supported: 0
|
||||
2026/02/15-08:54:06.512079 414231 kCustomCompressionAE supported: 0
|
||||
2026/02/15-08:54:06.512080 414231 kCustomCompressionEB supported: 0
|
||||
2026/02/15-08:54:06.512081 414231 kCustomCompressionED supported: 0
|
||||
2026/02/15-08:54:06.512081 414231 kCustomCompressionB6 supported: 0
|
||||
2026/02/15-08:54:06.512082 414231 kCustomCompressionEF supported: 0
|
||||
2026/02/15-08:54:06.512082 414231 kCustomCompressionF0 supported: 0
|
||||
2026/02/15-08:54:06.512083 414231 kCustomCompressionB7 supported: 0
|
||||
2026/02/15-08:54:06.512084 414231 kCustomCompressionF2 supported: 0
|
||||
2026/02/15-08:54:06.512084 414231 kCustomCompressionA1 supported: 0
|
||||
2026/02/15-08:54:06.512085 414231 kCustomCompressionF4 supported: 0
|
||||
2026/02/15-08:54:06.512086 414231 kSnappyCompression supported: 1
|
||||
2026/02/15-08:54:06.512086 414231 kCustomCompressionF6 supported: 0
|
||||
2026/02/15-08:54:06.512087 414231 Fast CRC32 supported: Not supported on x86
|
||||
2026/02/15-08:54:06.512088 414231 DMutex implementation: pthread_mutex_t
|
||||
2026/02/15-08:54:06.512088 414231 Jemalloc supported: 0
|
||||
2026/02/15-08:54:06.518228 414231 [db/db_impl/db_impl_open.cc:312] Creating manifest 1
|
||||
2026/02/15-08:54:06.526884 414231 [db/version_set.cc:6460] Recovering from manifest file: ./data/MANIFEST-000001
|
||||
2026/02/15-08:54:06.527736 414231 [db/column_family.cc:691] --------------- Options for column family [default]:
|
||||
2026/02/15-08:54:06.527984 414231 Options.comparator: leveldb.BytewiseComparator
|
||||
2026/02/15-08:54:06.527985 414231 Options.merge_operator: None
|
||||
2026/02/15-08:54:06.527986 414231 Options.compaction_filter: None
|
||||
2026/02/15-08:54:06.527986 414231 Options.compaction_filter_factory: None
|
||||
2026/02/15-08:54:06.527987 414231 Options.sst_partitioner_factory: None
|
||||
2026/02/15-08:54:06.527987 414231 Options.memtable_factory: SkipListFactory
|
||||
2026/02/15-08:54:06.527988 414231 Options.table_factory: BlockBasedTable
|
||||
2026/02/15-08:54:06.528029 414231 table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x3a78dde0)
|
||||
cache_index_and_filter_blocks: 0
|
||||
cache_index_and_filter_blocks_with_high_priority: 1
|
||||
pin_l0_filter_and_index_blocks_in_cache: 0
|
||||
pin_top_level_index_and_filter: 1
|
||||
index_type: 0
|
||||
data_block_index_type: 0
|
||||
index_shortening: 1
|
||||
data_block_hash_table_util_ratio: 0.750000
|
||||
checksum: 4
|
||||
no_block_cache: 0
|
||||
block_cache: 0x3a7878e0
|
||||
block_cache_name: AutoHyperClockCache
|
||||
block_cache_options:
|
||||
capacity : 33554432
|
||||
num_shard_bits : 0
|
||||
strict_capacity_limit : 0
|
||||
memory_allocator : None
|
||||
persistent_cache: (nil)
|
||||
block_size: 4096
|
||||
block_size_deviation: 10
|
||||
block_restart_interval: 16
|
||||
index_block_restart_interval: 1
|
||||
metadata_block_size: 4096
|
||||
partition_filters: 0
|
||||
use_delta_encoding: 1
|
||||
filter_policy: nullptr
|
||||
user_defined_index_factory: nullptr
|
||||
fail_if_no_udi_on_open: 0
|
||||
whole_key_filtering: 1
|
||||
verify_compression: 0
|
||||
read_amp_bytes_per_bit: 0
|
||||
format_version: 6
|
||||
enable_index_compression: 1
|
||||
block_align: 0
|
||||
super_block_alignment_size: 0
|
||||
super_block_alignment_space_overhead_ratio: 128
|
||||
max_auto_readahead_size: 262144
|
||||
prepopulate_block_cache: 0
|
||||
initial_auto_readahead_size: 8192
|
||||
num_file_reads_for_auto_readahead: 2
|
||||
2026/02/15-08:54:06.528036 414231 Options.write_buffer_size: 134217728
|
||||
2026/02/15-08:54:06.528037 414231 Options.max_write_buffer_number: 6
|
||||
2026/02/15-08:54:06.528057 414231 Options.compression[0]: NoCompression
|
||||
2026/02/15-08:54:06.528058 414231 Options.compression[1]: NoCompression
|
||||
2026/02/15-08:54:06.528059 414231 Options.compression[2]: LZ4
|
||||
2026/02/15-08:54:06.528059 414231 Options.compression[3]: LZ4
|
||||
2026/02/15-08:54:06.528060 414231 Options.compression[4]: LZ4
|
||||
2026/02/15-08:54:06.528060 414231 Options.compression[5]: LZ4
|
||||
2026/02/15-08:54:06.528061 414231 Options.compression[6]: LZ4
|
||||
2026/02/15-08:54:06.528063 414231 Options.bottommost_compression: Disabled
|
||||
2026/02/15-08:54:06.528064 414231 Options.prefix_extractor: nullptr
|
||||
2026/02/15-08:54:06.528064 414231 Options.memtable_insert_with_hint_prefix_extractor: nullptr
|
||||
2026/02/15-08:54:06.528065 414231 Options.num_levels: 7
|
||||
2026/02/15-08:54:06.528065 414231 Options.min_write_buffer_number_to_merge: 2
|
||||
2026/02/15-08:54:06.528066 414231 Options.max_write_buffer_size_to_maintain: 0
|
||||
2026/02/15-08:54:06.528067 414231 Options.bottommost_compression_opts.window_bits: -14
|
||||
2026/02/15-08:54:06.528067 414231 Options.bottommost_compression_opts.level: 32767
|
||||
2026/02/15-08:54:06.528068 414231 Options.bottommost_compression_opts.strategy: 0
|
||||
2026/02/15-08:54:06.528069 414231 Options.bottommost_compression_opts.max_dict_bytes: 0
|
||||
2026/02/15-08:54:06.528069 414231 Options.bottommost_compression_opts.zstd_max_train_bytes: 0
|
||||
2026/02/15-08:54:06.528070 414231 Options.bottommost_compression_opts.parallel_threads: 1
|
||||
2026/02/15-08:54:06.528071 414231 Options.bottommost_compression_opts.enabled: false
|
||||
2026/02/15-08:54:06.528071 414231 Options.bottommost_compression_opts.max_dict_buffer_bytes: 0
|
||||
2026/02/15-08:54:06.528072 414231 Options.bottommost_compression_opts.use_zstd_dict_trainer: true
|
||||
2026/02/15-08:54:06.528073 414231 Options.compression_opts.window_bits: -14
|
||||
2026/02/15-08:54:06.528073 414231 Options.compression_opts.level: 32767
|
||||
2026/02/15-08:54:06.528074 414231 Options.compression_opts.strategy: 0
|
||||
2026/02/15-08:54:06.528075 414231 Options.compression_opts.max_dict_bytes: 0
|
||||
2026/02/15-08:54:06.528075 414231 Options.compression_opts.zstd_max_train_bytes: 0
|
||||
2026/02/15-08:54:06.528076 414231 Options.compression_opts.use_zstd_dict_trainer: true
|
||||
2026/02/15-08:54:06.528077 414231 Options.compression_opts.parallel_threads: 1
|
||||
2026/02/15-08:54:06.528077 414231 Options.compression_opts.enabled: false
|
||||
2026/02/15-08:54:06.528078 414231 Options.compression_opts.max_dict_buffer_bytes: 0
|
||||
2026/02/15-08:54:06.528079 414231 Options.level0_file_num_compaction_trigger: 2
|
||||
2026/02/15-08:54:06.528079 414231 Options.level0_slowdown_writes_trigger: 20
|
||||
2026/02/15-08:54:06.528080 414231 Options.level0_stop_writes_trigger: 36
|
||||
2026/02/15-08:54:06.528081 414231 Options.target_file_size_base: 67108864
|
||||
2026/02/15-08:54:06.528081 414231 Options.target_file_size_multiplier: 1
|
||||
2026/02/15-08:54:06.528082 414231 Options.target_file_size_is_upper_bound: 0
|
||||
2026/02/15-08:54:06.528083 414231 Options.max_bytes_for_level_base: 536870912
|
||||
2026/02/15-08:54:06.528083 414231 Options.level_compaction_dynamic_level_bytes: 1
|
||||
2026/02/15-08:54:06.528084 414231 Options.max_bytes_for_level_multiplier: 10.000000
|
||||
2026/02/15-08:54:06.528085 414231 Options.max_bytes_for_level_multiplier_addtl[0]: 1
|
||||
2026/02/15-08:54:06.528086 414231 Options.max_bytes_for_level_multiplier_addtl[1]: 1
|
||||
2026/02/15-08:54:06.528087 414231 Options.max_bytes_for_level_multiplier_addtl[2]: 1
|
||||
2026/02/15-08:54:06.528088 414231 Options.max_bytes_for_level_multiplier_addtl[3]: 1
|
||||
2026/02/15-08:54:06.528088 414231 Options.max_bytes_for_level_multiplier_addtl[4]: 1
|
||||
2026/02/15-08:54:06.528089 414231 Options.max_bytes_for_level_multiplier_addtl[5]: 1
|
||||
2026/02/15-08:54:06.528089 414231 Options.max_bytes_for_level_multiplier_addtl[6]: 1
|
||||
2026/02/15-08:54:06.528090 414231 Options.max_sequential_skip_in_iterations: 8
|
||||
2026/02/15-08:54:06.528091 414231 Options.memtable_op_scan_flush_trigger: 0
|
||||
2026/02/15-08:54:06.528091 414231 Options.memtable_avg_op_scan_flush_trigger: 0
|
||||
2026/02/15-08:54:06.528092 414231 Options.max_compaction_bytes: 1677721600
|
||||
2026/02/15-08:54:06.528093 414231 Options.arena_block_size: 1048576
|
||||
2026/02/15-08:54:06.528093 414231 Options.soft_pending_compaction_bytes_limit: 68719476736
|
||||
2026/02/15-08:54:06.528095 414231 Options.hard_pending_compaction_bytes_limit: 274877906944
|
||||
2026/02/15-08:54:06.528095 414231 Options.disable_auto_compactions: 0
|
||||
2026/02/15-08:54:06.528097 414231 Options.compaction_style: kCompactionStyleLevel
|
||||
2026/02/15-08:54:06.528098 414231 Options.compaction_pri: kMinOverlappingRatio
|
||||
2026/02/15-08:54:06.528098 414231 Options.compaction_options_universal.size_ratio: 1
|
||||
2026/02/15-08:54:06.528099 414231 Options.compaction_options_universal.min_merge_width: 2
|
||||
2026/02/15-08:54:06.528100 414231 Options.compaction_options_universal.max_merge_width: 4294967295
|
||||
2026/02/15-08:54:06.528100 414231 Options.compaction_options_universal.max_size_amplification_percent: 200
|
||||
2026/02/15-08:54:06.528101 414231 Options.compaction_options_universal.compression_size_percent: -1
|
||||
2026/02/15-08:54:06.528102 414231 Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize
|
||||
2026/02/15-08:54:06.528102 414231 Options.compaction_options_universal.max_read_amp: -1
|
||||
2026/02/15-08:54:06.528103 414231 Options.compaction_options_universal.reduce_file_locking: 0
|
||||
2026/02/15-08:54:06.528104 414231 Options.compaction_options_fifo.max_table_files_size: 1073741824
|
||||
2026/02/15-08:54:06.528104 414231 Options.compaction_options_fifo.allow_compaction: 0
|
||||
2026/02/15-08:54:06.528109 414231 Options.table_properties_collectors:
|
||||
2026/02/15-08:54:06.528110 414231 Options.inplace_update_support: 0
|
||||
2026/02/15-08:54:06.528111 414231 Options.inplace_update_num_locks: 10000
|
||||
2026/02/15-08:54:06.528112 414231 Options.memtable_prefix_bloom_size_ratio: 0.000000
|
||||
2026/02/15-08:54:06.528112 414231 Options.memtable_whole_key_filtering: 0
|
||||
2026/02/15-08:54:06.528113 414231 Options.memtable_huge_page_size: 0
|
||||
2026/02/15-08:54:06.528114 414231 Options.bloom_locality: 0
|
||||
2026/02/15-08:54:06.528114 414231 Options.max_successive_merges: 0
|
||||
2026/02/15-08:54:06.528115 414231 Options.strict_max_successive_merges: 0
|
||||
2026/02/15-08:54:06.528116 414231 Options.optimize_filters_for_hits: 0
|
||||
2026/02/15-08:54:06.528116 414231 Options.paranoid_file_checks: 0
|
||||
2026/02/15-08:54:06.528117 414231 Options.force_consistency_checks: 1
|
||||
2026/02/15-08:54:06.528118 414231 Options.report_bg_io_stats: 0
|
||||
2026/02/15-08:54:06.528118 414231 Options.disallow_memtable_writes: 0
|
||||
2026/02/15-08:54:06.528119 414231 Options.ttl: 2592000
|
||||
2026/02/15-08:54:06.528119 414231 Options.periodic_compaction_seconds: 0
|
||||
2026/02/15-08:54:06.528120 414231 Options.default_temperature: kUnknown
|
||||
2026/02/15-08:54:06.528121 414231 Options.preclude_last_level_data_seconds: 0
|
||||
2026/02/15-08:54:06.528122 414231 Options.preserve_internal_time_seconds: 0
|
||||
2026/02/15-08:54:06.528122 414231 Options.enable_blob_files: false
|
||||
2026/02/15-08:54:06.528123 414231 Options.min_blob_size: 0
|
||||
2026/02/15-08:54:06.528123 414231 Options.blob_file_size: 268435456
|
||||
2026/02/15-08:54:06.528124 414231 Options.blob_compression_type: NoCompression
|
||||
2026/02/15-08:54:06.528125 414231 Options.enable_blob_garbage_collection: false
|
||||
2026/02/15-08:54:06.528125 414231 Options.blob_garbage_collection_age_cutoff: 0.250000
|
||||
2026/02/15-08:54:06.528126 414231 Options.blob_garbage_collection_force_threshold: 1.000000
|
||||
2026/02/15-08:54:06.528127 414231 Options.blob_compaction_readahead_size: 0
|
||||
2026/02/15-08:54:06.528128 414231 Options.blob_file_starting_level: 0
|
||||
2026/02/15-08:54:06.528128 414231 Options.experimental_mempurge_threshold: 0.000000
|
||||
2026/02/15-08:54:06.528129 414231 Options.memtable_max_range_deletions: 0
|
||||
2026/02/15-08:54:06.528130 414231 Options.cf_allow_ingest_behind: false
|
||||
2026/02/15-08:54:06.529411 414231 [db/version_set.cc:6510] Recovered from manifest file:./data/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0
|
||||
2026/02/15-08:54:06.529414 414231 [db/version_set.cc:6525] Column family [default] (ID 0), log number is 0
|
||||
2026/02/15-08:54:06.529415 414231 [db/db_impl/db_impl_open.cc:686] DB ID: 8dff41c8-9c17-41a7-bcc0-29dc39228555
|
||||
2026/02/15-08:54:06.537957 414231 [db/version_set.cc:6070] Created manifest 5, compacted+appended from 52 to 116
|
||||
2026/02/15-08:54:06.547313 414231 [db/db_impl/db_impl_open.cc:2626] SstFileManager instance 0x3a78f5b0
|
||||
2026/02/15-08:54:06.547637 414231 [DEBUG] [db/db_impl/db_impl_files.cc:389] [JOB 1] Delete ./data/MANIFEST-000001 type=3 #1 -- OK
|
||||
2026/02/15-08:54:06.547650 414231 DB pointer 0x3a7903c0
|
||||
2026/02/15-08:54:06.547993 414252 [DEBUG] [cache/clock_cache.cc:1568] Slot occupancy stats: Overall 1% (1/64), Min/Max/Window = 100%/0%/500, MaxRun{Pos/Neg} = 1/56
|
||||
2026/02/15-08:54:06.547995 414252 [DEBUG] [cache/clock_cache.cc:1570] Eviction effort exceeded: 0
|
||||
2026/02/15-08:54:06.548022 414252 [DEBUG] [cache/clock_cache.cc:3639] Head occupancy stats: Overall 1% (1/64), Min/Max/Window = 100%/0%/500, MaxRun{Pos/Neg} = 1/56
|
||||
2026/02/15-08:54:06.548023 414252 [DEBUG] [cache/clock_cache.cc:3641] Entries at home count: 1
|
||||
2026/02/15-08:54:06.548024 414252 [DEBUG] [cache/clock_cache.cc:3643] Yield count: 0
|
||||
2026/02/15-08:54:06.548796 414252 [db/db_impl/db_impl.cc:1116] ------- DUMPING STATS -------
|
||||
2026/02/15-08:54:06.548802 414252 [db/db_impl/db_impl.cc:1118]
|
||||
** DB Stats **
|
||||
Uptime(secs): 0.0 total, 0.0 interval
|
||||
Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s
|
||||
Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s
|
||||
Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
|
||||
Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s
|
||||
Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s
|
||||
Interval stall: 00:00:0.000 H:M:S, 0.0 percent
|
||||
Write Stall (count): write-buffer-manager-limit-stops: 0
|
||||
|
||||
** Compaction Stats [default] **
|
||||
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) WPreComp(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB)
|
||||
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
||||
Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0
|
||||
Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0
|
||||
|
||||
** Compaction Stats [default] **
|
||||
Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) WPreComp(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB)
|
||||
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0
|
||||
|
||||
Uptime(secs): 0.0 total, 0.0 interval
|
||||
Flush(GB): cumulative 0.000, interval 0.000
|
||||
AddFile(GB): cumulative 0.000, interval 0.000
|
||||
AddFile(Total Files): cumulative 0, interval 0
|
||||
AddFile(L0 Files): cumulative 0, interval 0
|
||||
AddFile(Keys): cumulative 0, interval 0
|
||||
Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
|
||||
Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
|
||||
Estimated pending compaction bytes: 0
|
||||
Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 0, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 0
|
||||
Block cache AutoHyperClockCache@0x3a7878e0#414231 capacity: 32.00 MB seed: 585643374 usage: 4.00 KB table_size: 64 occupancy: 1 collections: 1 last_copies: 0 last_secs: 1.3e-05 secs_since: 0
|
||||
Block cache entry stats(count,size,portion): Misc(1,0.00 KB,0%)
|
||||
|
||||
** File Read Latency Histogram By Level [default] **
|
||||
BIN
data/MANIFEST-000005
Normal file
BIN
data/MANIFEST-000005
Normal file
Binary file not shown.
226
data/OPTIONS-000007
Normal file
226
data/OPTIONS-000007
Normal file
@@ -0,0 +1,226 @@
|
||||
# This is a RocksDB option file.
|
||||
#
|
||||
# For detailed file format spec, please refer to the example file
|
||||
# in examples/rocksdb_option_file_example.ini
|
||||
#
|
||||
|
||||
[Version]
|
||||
rocksdb_version=10.9.1
|
||||
options_file_version=1.1
|
||||
|
||||
[DBOptions]
|
||||
max_manifest_space_amp_pct=500
|
||||
manifest_preallocation_size=4194304
|
||||
max_manifest_file_size=1073741824
|
||||
compaction_readahead_size=2097152
|
||||
strict_bytes_per_sync=false
|
||||
bytes_per_sync=0
|
||||
max_background_jobs=4
|
||||
avoid_flush_during_shutdown=false
|
||||
max_background_flushes=-1
|
||||
delayed_write_rate=16777216
|
||||
max_open_files=-1
|
||||
max_subcompactions=1
|
||||
writable_file_max_buffer_size=1048576
|
||||
wal_bytes_per_sync=0
|
||||
max_background_compactions=-1
|
||||
max_total_wal_size=0
|
||||
delete_obsolete_files_period_micros=21600000000
|
||||
stats_dump_period_sec=600
|
||||
stats_history_buffer_size=1048576
|
||||
stats_persist_period_sec=600
|
||||
follower_refresh_catchup_period_ms=10000
|
||||
enforce_single_del_contracts=true
|
||||
lowest_used_cache_tier=kNonVolatileBlockTier
|
||||
bgerror_resume_retry_interval=1000000
|
||||
metadata_write_temperature=kUnknown
|
||||
best_efforts_recovery=false
|
||||
log_readahead_size=0
|
||||
write_identity_file=true
|
||||
write_dbid_to_manifest=true
|
||||
prefix_seek_opt_in_only=false
|
||||
wal_compression=kNoCompression
|
||||
manual_wal_flush=false
|
||||
db_host_id=__hostname__
|
||||
two_write_queues=false
|
||||
skip_checking_sst_file_sizes_on_db_open=false
|
||||
flush_verify_memtable_count=true
|
||||
atomic_flush=false
|
||||
verify_sst_unique_id_in_manifest=true
|
||||
skip_stats_update_on_db_open=false
|
||||
track_and_verify_wals=false
|
||||
track_and_verify_wals_in_manifest=false
|
||||
compaction_verify_record_count=true
|
||||
paranoid_checks=true
|
||||
create_if_missing=true
|
||||
max_write_batch_group_size_bytes=1048576
|
||||
follower_catchup_retry_count=10
|
||||
avoid_flush_during_recovery=false
|
||||
file_checksum_gen_factory=nullptr
|
||||
enable_thread_tracking=false
|
||||
allow_fallocate=true
|
||||
allow_data_in_errors=false
|
||||
error_if_exists=false
|
||||
use_direct_io_for_flush_and_compaction=false
|
||||
background_close_inactive_wals=false
|
||||
create_missing_column_families=false
|
||||
WAL_size_limit_MB=0
|
||||
use_direct_reads=false
|
||||
persist_stats_to_disk=false
|
||||
allow_2pc=false
|
||||
max_log_file_size=0
|
||||
is_fd_close_on_exec=true
|
||||
avoid_unnecessary_blocking_io=false
|
||||
max_file_opening_threads=16
|
||||
wal_filter=nullptr
|
||||
wal_write_temperature=kUnknown
|
||||
follower_catchup_retry_wait_ms=100
|
||||
allow_mmap_reads=false
|
||||
allow_mmap_writes=false
|
||||
use_adaptive_mutex=false
|
||||
use_fsync=false
|
||||
table_cache_numshardbits=6
|
||||
dump_malloc_stats=false
|
||||
db_write_buffer_size=0
|
||||
allow_ingest_behind=false
|
||||
keep_log_file_num=1000
|
||||
max_bgerror_resume_count=2147483647
|
||||
allow_concurrent_memtable_write=true
|
||||
recycle_log_file_num=0
|
||||
log_file_time_to_roll=0
|
||||
WAL_ttl_seconds=0
|
||||
enable_pipelined_write=false
|
||||
write_thread_slow_yield_usec=3
|
||||
unordered_write=false
|
||||
wal_recovery_mode=kPointInTimeRecovery
|
||||
enable_write_thread_adaptive_yield=true
|
||||
write_thread_max_yield_usec=100
|
||||
advise_random_on_open=true
|
||||
info_log_level=DEBUG_LEVEL
|
||||
|
||||
|
||||
[CFOptions "default"]
|
||||
memtable_max_range_deletions=0
|
||||
compression_manager=nullptr
|
||||
compression_opts={checksum=false;max_dict_buffer_bytes=0;enabled=false;max_dict_bytes=0;max_compressed_bytes_per_kb=896;parallel_threads=1;zstd_max_train_bytes=0;level=32767;use_zstd_dict_trainer=true;strategy=0;window_bits=-14;}
|
||||
paranoid_memory_checks=false
|
||||
memtable_avg_op_scan_flush_trigger=0
|
||||
block_protection_bytes_per_key=0
|
||||
uncache_aggressiveness=0
|
||||
bottommost_file_compaction_delay=0
|
||||
memtable_protection_bytes_per_key=0
|
||||
compression_per_level=kNoCompression:kNoCompression:kLZ4Compression:kLZ4Compression:kLZ4Compression:kLZ4Compression:kLZ4Compression
|
||||
bottommost_compression=kDisableCompressionOption
|
||||
sample_for_compression=0
|
||||
prepopulate_blob_cache=kDisable
|
||||
blob_file_starting_level=0
|
||||
blob_compaction_readahead_size=0
|
||||
blob_garbage_collection_force_threshold=1.000000
|
||||
blob_garbage_collection_age_cutoff=0.250000
|
||||
table_factory=BlockBasedTable
|
||||
max_successive_merges=0
|
||||
max_write_buffer_number=6
|
||||
prefix_extractor=nullptr
|
||||
memtable_huge_page_size=0
|
||||
write_buffer_size=134217728
|
||||
strict_max_successive_merges=false
|
||||
arena_block_size=1048576
|
||||
memtable_op_scan_flush_trigger=0
|
||||
level0_file_num_compaction_trigger=2
|
||||
report_bg_io_stats=false
|
||||
inplace_update_num_locks=10000
|
||||
memtable_prefix_bloom_size_ratio=0.000000
|
||||
level0_stop_writes_trigger=36
|
||||
blob_compression_type=kNoCompression
|
||||
level0_slowdown_writes_trigger=20
|
||||
hard_pending_compaction_bytes_limit=274877906944
|
||||
target_file_size_multiplier=1
|
||||
paranoid_file_checks=false
|
||||
min_blob_size=0
|
||||
max_compaction_bytes=1677721600
|
||||
disable_auto_compactions=false
|
||||
experimental_mempurge_threshold=0.000000
|
||||
verify_output_flags=0
|
||||
last_level_temperature=kUnknown
|
||||
preserve_internal_time_seconds=0
|
||||
memtable_veirfy_per_key_checksum_on_seek=false
|
||||
soft_pending_compaction_bytes_limit=68719476736
|
||||
target_file_size_base=67108864
|
||||
enable_blob_files=false
|
||||
bottommost_compression_opts={checksum=false;max_dict_buffer_bytes=0;enabled=false;max_dict_bytes=0;max_compressed_bytes_per_kb=896;parallel_threads=1;zstd_max_train_bytes=0;level=32767;use_zstd_dict_trainer=true;strategy=0;window_bits=-14;}
|
||||
memtable_whole_key_filtering=false
|
||||
target_file_size_is_upper_bound=false
|
||||
max_bytes_for_level_base=536870912
|
||||
compaction_options_fifo={trivial_copy_buffer_size=4096;allow_trivial_copy_when_change_temperature=false;file_temperature_age_thresholds=;allow_compaction=false;age_for_warm=0;max_table_files_size=1073741824;}
|
||||
max_bytes_for_level_multiplier=10.000000
|
||||
max_bytes_for_level_multiplier_additional=1:1:1:1:1:1:1
|
||||
max_sequential_skip_in_iterations=8
|
||||
compression=kLZ4Compression
|
||||
default_write_temperature=kUnknown
|
||||
compaction_options_universal={reduce_file_locking=false;incremental=false;compression_size_percent=-1;allow_trivial_move=false;max_size_amplification_percent=200;max_merge_width=4294967295;stop_style=kCompactionStopStyleTotalSize;min_merge_width=2;max_read_amp=-1;size_ratio=1;}
|
||||
ttl=2592000
|
||||
periodic_compaction_seconds=0
|
||||
preclude_last_level_data_seconds=0
|
||||
blob_file_size=268435456
|
||||
enable_blob_garbage_collection=false
|
||||
cf_allow_ingest_behind=false
|
||||
min_write_buffer_number_to_merge=2
|
||||
sst_partitioner_factory=nullptr
|
||||
num_levels=7
|
||||
disallow_memtable_writes=false
|
||||
force_consistency_checks=true
|
||||
memtable_insert_with_hint_prefix_extractor=nullptr
|
||||
memtable_factory=SkipListFactory
|
||||
optimize_filters_for_hits=false
|
||||
level_compaction_dynamic_level_bytes=true
|
||||
compaction_style=kCompactionStyleLevel
|
||||
compaction_filter=nullptr
|
||||
default_temperature=kUnknown
|
||||
inplace_update_support=false
|
||||
merge_operator=nullptr
|
||||
bloom_locality=0
|
||||
comparator=leveldb.BytewiseComparator
|
||||
compaction_filter_factory=nullptr
|
||||
max_write_buffer_size_to_maintain=0
|
||||
compaction_pri=kMinOverlappingRatio
|
||||
persist_user_defined_timestamps=true
|
||||
|
||||
[TableOptions/BlockBasedTable "default"]
|
||||
fail_if_no_udi_on_open=false
|
||||
initial_auto_readahead_size=8192
|
||||
max_auto_readahead_size=262144
|
||||
metadata_cache_options={unpartitioned_pinning=kFallback;partition_pinning=kFallback;top_level_index_pinning=kFallback;}
|
||||
block_align=false
|
||||
read_amp_bytes_per_bit=0
|
||||
verify_compression=false
|
||||
detect_filter_construct_corruption=false
|
||||
whole_key_filtering=true
|
||||
user_defined_index_factory=nullptr
|
||||
filter_policy=nullptr
|
||||
super_block_alignment_space_overhead_ratio=128
|
||||
use_delta_encoding=true
|
||||
optimize_filters_for_memory=true
|
||||
partition_filters=false
|
||||
prepopulate_block_cache=kDisable
|
||||
pin_top_level_index_and_filter=true
|
||||
index_block_restart_interval=1
|
||||
block_size_deviation=10
|
||||
num_file_reads_for_auto_readahead=2
|
||||
format_version=6
|
||||
decouple_partitioned_filters=true
|
||||
checksum=kXXH3
|
||||
block_size=4096
|
||||
data_block_hash_table_util_ratio=0.750000
|
||||
index_shortening=kShortenSeparators
|
||||
block_restart_interval=16
|
||||
data_block_index_type=kDataBlockBinarySearch
|
||||
index_type=kBinarySearch
|
||||
super_block_alignment_size=0
|
||||
metadata_block_size=4096
|
||||
pin_l0_filter_and_index_blocks_in_cache=false
|
||||
no_block_cache=false
|
||||
cache_index_and_filter_blocks_with_high_priority=true
|
||||
cache_index_and_filter_blocks=false
|
||||
enable_index_compression=true
|
||||
flush_block_policy_factory=FlushBlockBySizePolicyFactory
|
||||
|
||||
450
dynamodb/types.odin
Normal file
450
dynamodb/types.odin
Normal file
@@ -0,0 +1,450 @@
|
||||
package dynamodb
|
||||
|
||||
import "core:fmt"
|
||||
import "core:strings"
|
||||
|
||||
// DynamoDB AttributeValue - the core data type
|
||||
Attribute_Value :: union {
|
||||
String, // S
|
||||
Number, // N (stored as string)
|
||||
Binary, // B (base64)
|
||||
Bool, // BOOL
|
||||
Null, // NULL
|
||||
String_Set, // SS
|
||||
Number_Set, // NS
|
||||
Binary_Set, // BS
|
||||
List, // L
|
||||
Map, // M
|
||||
}
|
||||
|
||||
String :: distinct string
|
||||
Number :: distinct string
|
||||
Binary :: distinct string
|
||||
Bool :: distinct bool
|
||||
Null :: distinct bool
|
||||
|
||||
String_Set :: distinct []string
|
||||
Number_Set :: distinct []string
|
||||
Binary_Set :: distinct []string
|
||||
List :: distinct []Attribute_Value
|
||||
Map :: distinct map[string]Attribute_Value
|
||||
|
||||
// Item is a map of attribute names to values
|
||||
Item :: map[string]Attribute_Value
|
||||
|
||||
// Key represents a DynamoDB key (partition key + optional sort key)
|
||||
Key :: struct {
|
||||
pk: Attribute_Value,
|
||||
sk: Maybe(Attribute_Value),
|
||||
}
|
||||
|
||||
// Free a key
|
||||
key_destroy :: proc(key: ^Key) {
|
||||
attr_value_destroy(&key.pk)
|
||||
if sk, ok := key.sk.?; ok {
|
||||
sk_copy := sk
|
||||
attr_value_destroy(&sk_copy)
|
||||
}
|
||||
}
|
||||
|
||||
// Extract key from item based on key schema
|
||||
key_from_item :: proc(item: Item, key_schema: []Key_Schema_Element) -> (Key, bool) {
|
||||
pk_value: Attribute_Value
|
||||
sk_value: Maybe(Attribute_Value)
|
||||
|
||||
for schema_elem in key_schema {
|
||||
attr, ok := item[schema_elem.attribute_name]
|
||||
if !ok {
|
||||
return {}, false
|
||||
}
|
||||
|
||||
// Validate that key is a scalar type (S, N, or B)
|
||||
#partial switch _ in attr {
|
||||
case String, Number, Binary:
|
||||
// Valid key type
|
||||
case:
|
||||
return {}, false
|
||||
}
|
||||
|
||||
// Deep copy the attribute value
|
||||
copied := attr_value_deep_copy(attr)
|
||||
|
||||
switch schema_elem.key_type {
|
||||
case .HASH:
|
||||
pk_value = copied
|
||||
case .RANGE:
|
||||
sk_value = copied
|
||||
}
|
||||
}
|
||||
|
||||
return Key{pk = pk_value, sk = sk_value}, true
|
||||
}
|
||||
|
||||
// Convert key to item
|
||||
key_to_item :: proc(key: Key, key_schema: []Key_Schema_Element) -> Item {
|
||||
item := make(Item)
|
||||
|
||||
for schema_elem in key_schema {
|
||||
attr_value: Attribute_Value
|
||||
|
||||
switch schema_elem.key_type {
|
||||
case .HASH:
|
||||
attr_value = key.pk
|
||||
case .RANGE:
|
||||
if sk, ok := key.sk.?; ok {
|
||||
attr_value = sk
|
||||
} else {
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
item[schema_elem.attribute_name] = attr_value_deep_copy(attr_value)
|
||||
}
|
||||
|
||||
return item
|
||||
}
|
||||
|
||||
// Extract raw byte values from key
|
||||
Key_Values :: struct {
|
||||
pk: []byte,
|
||||
sk: Maybe([]byte),
|
||||
}
|
||||
|
||||
key_get_values :: proc(key: ^Key) -> (Key_Values, bool) {
|
||||
pk_bytes: []byte
|
||||
|
||||
switch v in key.pk {
|
||||
case String:
|
||||
pk_bytes = transmute([]byte)string(v)
|
||||
case Number:
|
||||
pk_bytes = transmute([]byte)string(v)
|
||||
case Binary:
|
||||
pk_bytes = transmute([]byte)string(v)
|
||||
case:
|
||||
return {}, false
|
||||
}
|
||||
|
||||
sk_bytes: Maybe([]byte)
|
||||
if sk, ok := key.sk.?; ok {
|
||||
switch v in sk {
|
||||
case String:
|
||||
sk_bytes = transmute([]byte)string(v)
|
||||
case Number:
|
||||
sk_bytes = transmute([]byte)string(v)
|
||||
case Binary:
|
||||
sk_bytes = transmute([]byte)string(v)
|
||||
case:
|
||||
return {}, false
|
||||
}
|
||||
}
|
||||
|
||||
return Key_Values{pk = pk_bytes, sk = sk_bytes}, true
|
||||
}
|
||||
|
||||
// Key type
|
||||
Key_Type :: enum {
|
||||
HASH,
|
||||
RANGE,
|
||||
}
|
||||
|
||||
key_type_to_string :: proc(kt: Key_Type) -> string {
|
||||
switch kt {
|
||||
case .HASH: return "HASH"
|
||||
case .RANGE: return "RANGE"
|
||||
}
|
||||
return "HASH"
|
||||
}
|
||||
|
||||
key_type_from_string :: proc(s: string) -> (Key_Type, bool) {
|
||||
switch s {
|
||||
case "HASH": return .HASH, true
|
||||
case "RANGE": return .RANGE, true
|
||||
}
|
||||
return .HASH, false
|
||||
}
|
||||
|
||||
// Scalar attribute type
|
||||
Scalar_Attribute_Type :: enum {
|
||||
S, // String
|
||||
N, // Number
|
||||
B, // Binary
|
||||
}
|
||||
|
||||
scalar_type_to_string :: proc(t: Scalar_Attribute_Type) -> string {
|
||||
switch t {
|
||||
case .S: return "S"
|
||||
case .N: return "N"
|
||||
case .B: return "B"
|
||||
}
|
||||
return "S"
|
||||
}
|
||||
|
||||
scalar_type_from_string :: proc(s: string) -> (Scalar_Attribute_Type, bool) {
|
||||
switch s {
|
||||
case "S": return .S, true
|
||||
case "N": return .N, true
|
||||
case "B": return .B, true
|
||||
}
|
||||
return .S, false
|
||||
}
|
||||
|
||||
// Key schema element
|
||||
Key_Schema_Element :: struct {
|
||||
attribute_name: string,
|
||||
key_type: Key_Type,
|
||||
}
|
||||
|
||||
// Attribute definition
|
||||
Attribute_Definition :: struct {
|
||||
attribute_name: string,
|
||||
attribute_type: Scalar_Attribute_Type,
|
||||
}
|
||||
|
||||
// Projection type for indexes
|
||||
Projection_Type :: enum {
|
||||
ALL,
|
||||
KEYS_ONLY,
|
||||
INCLUDE,
|
||||
}
|
||||
|
||||
// Projection
|
||||
Projection :: struct {
|
||||
projection_type: Projection_Type,
|
||||
non_key_attributes: Maybe([]string),
|
||||
}
|
||||
|
||||
// Global secondary index
|
||||
Global_Secondary_Index :: struct {
|
||||
index_name: string,
|
||||
key_schema: []Key_Schema_Element,
|
||||
projection: Projection,
|
||||
}
|
||||
|
||||
// Local secondary index
|
||||
Local_Secondary_Index :: struct {
|
||||
index_name: string,
|
||||
key_schema: []Key_Schema_Element,
|
||||
projection: Projection,
|
||||
}
|
||||
|
||||
// Table status
|
||||
Table_Status :: enum {
|
||||
CREATING,
|
||||
UPDATING,
|
||||
DELETING,
|
||||
ACTIVE,
|
||||
INACCESSIBLE_ENCRYPTION_CREDENTIALS,
|
||||
ARCHIVING,
|
||||
ARCHIVED,
|
||||
}
|
||||
|
||||
table_status_to_string :: proc(status: Table_Status) -> string {
|
||||
switch status {
|
||||
case .CREATING: return "CREATING"
|
||||
case .UPDATING: return "UPDATING"
|
||||
case .DELETING: return "DELETING"
|
||||
case .ACTIVE: return "ACTIVE"
|
||||
case .INACCESSIBLE_ENCRYPTION_CREDENTIALS: return "INACCESSIBLE_ENCRYPTION_CREDENTIALS"
|
||||
case .ARCHIVING: return "ARCHIVING"
|
||||
case .ARCHIVED: return "ARCHIVED"
|
||||
}
|
||||
return "ACTIVE"
|
||||
}
|
||||
|
||||
// Table description
|
||||
Table_Description :: struct {
|
||||
table_name: string,
|
||||
key_schema: []Key_Schema_Element,
|
||||
attribute_definitions: []Attribute_Definition,
|
||||
table_status: Table_Status,
|
||||
creation_date_time: i64,
|
||||
item_count: u64,
|
||||
table_size_bytes: u64,
|
||||
global_secondary_indexes: Maybe([]Global_Secondary_Index),
|
||||
local_secondary_indexes: Maybe([]Local_Secondary_Index),
|
||||
}
|
||||
|
||||
// DynamoDB operation types
|
||||
Operation :: enum {
|
||||
CreateTable,
|
||||
DeleteTable,
|
||||
DescribeTable,
|
||||
ListTables,
|
||||
UpdateTable,
|
||||
PutItem,
|
||||
GetItem,
|
||||
DeleteItem,
|
||||
UpdateItem,
|
||||
Query,
|
||||
Scan,
|
||||
BatchGetItem,
|
||||
BatchWriteItem,
|
||||
TransactGetItems,
|
||||
TransactWriteItems,
|
||||
Unknown,
|
||||
}
|
||||
|
||||
operation_from_target :: proc(target: string) -> Operation {
|
||||
prefix :: "DynamoDB_20120810."
|
||||
if !strings.has_prefix(target, prefix) {
|
||||
return .Unknown
|
||||
}
|
||||
|
||||
op_name := target[len(prefix):]
|
||||
|
||||
switch op_name {
|
||||
case "CreateTable": return .CreateTable
|
||||
case "DeleteTable": return .DeleteTable
|
||||
case "DescribeTable": return .DescribeTable
|
||||
case "ListTables": return .ListTables
|
||||
case "UpdateTable": return .UpdateTable
|
||||
case "PutItem": return .PutItem
|
||||
case "GetItem": return .GetItem
|
||||
case "DeleteItem": return .DeleteItem
|
||||
case "UpdateItem": return .UpdateItem
|
||||
case "Query": return .Query
|
||||
case "Scan": return .Scan
|
||||
case "BatchGetItem": return .BatchGetItem
|
||||
case "BatchWriteItem": return .BatchWriteItem
|
||||
case "TransactGetItems": return .TransactGetItems
|
||||
case "TransactWriteItems": return .TransactWriteItems
|
||||
}
|
||||
|
||||
return .Unknown
|
||||
}
|
||||
|
||||
// DynamoDB error types
|
||||
DynamoDB_Error_Type :: enum {
|
||||
ValidationException,
|
||||
ResourceNotFoundException,
|
||||
ResourceInUseException,
|
||||
ConditionalCheckFailedException,
|
||||
ProvisionedThroughputExceededException,
|
||||
ItemCollectionSizeLimitExceededException,
|
||||
InternalServerError,
|
||||
SerializationException,
|
||||
}
|
||||
|
||||
error_to_response :: proc(err_type: DynamoDB_Error_Type, message: string) -> string {
|
||||
type_str: string
|
||||
|
||||
switch err_type {
|
||||
case .ValidationException:
|
||||
type_str = "com.amazonaws.dynamodb.v20120810#ValidationException"
|
||||
case .ResourceNotFoundException:
|
||||
type_str = "com.amazonaws.dynamodb.v20120810#ResourceNotFoundException"
|
||||
case .ResourceInUseException:
|
||||
type_str = "com.amazonaws.dynamodb.v20120810#ResourceInUseException"
|
||||
case .ConditionalCheckFailedException:
|
||||
type_str = "com.amazonaws.dynamodb.v20120810#ConditionalCheckFailedException"
|
||||
case .ProvisionedThroughputExceededException:
|
||||
type_str = "com.amazonaws.dynamodb.v20120810#ProvisionedThroughputExceededException"
|
||||
case .ItemCollectionSizeLimitExceededException:
|
||||
type_str = "com.amazonaws.dynamodb.v20120810#ItemCollectionSizeLimitExceededException"
|
||||
case .InternalServerError:
|
||||
type_str = "com.amazonaws.dynamodb.v20120810#InternalServerError"
|
||||
case .SerializationException:
|
||||
type_str = "com.amazonaws.dynamodb.v20120810#SerializationException"
|
||||
}
|
||||
|
||||
return fmt.aprintf(`{{"__type":"%s","message":"%s"}}`, type_str, message)
|
||||
}
|
||||
|
||||
// Deep copy an attribute value
|
||||
attr_value_deep_copy :: proc(attr: Attribute_Value) -> Attribute_Value {
|
||||
switch v in attr {
|
||||
case String:
|
||||
return String(strings.clone(string(v)))
|
||||
case Number:
|
||||
return Number(strings.clone(string(v)))
|
||||
case Binary:
|
||||
return Binary(strings.clone(string(v)))
|
||||
case Bool:
|
||||
return v
|
||||
case Null:
|
||||
return v
|
||||
case String_Set:
|
||||
ss := make([]string, len(v))
|
||||
for s, i in v {
|
||||
ss[i] = strings.clone(s)
|
||||
}
|
||||
return String_Set(ss)
|
||||
case Number_Set:
|
||||
ns := make([]string, len(v))
|
||||
for n, i in v {
|
||||
ns[i] = strings.clone(n)
|
||||
}
|
||||
return Number_Set(ns)
|
||||
case Binary_Set:
|
||||
bs := make([]string, len(v))
|
||||
for b, i in v {
|
||||
bs[i] = strings.clone(b)
|
||||
}
|
||||
return Binary_Set(bs)
|
||||
case List:
|
||||
list := make([]Attribute_Value, len(v))
|
||||
for item, i in v {
|
||||
list[i] = attr_value_deep_copy(item)
|
||||
}
|
||||
return List(list)
|
||||
case Map:
|
||||
m := make(map[string]Attribute_Value)
|
||||
for key, val in v {
|
||||
m[strings.clone(key)] = attr_value_deep_copy(val)
|
||||
}
|
||||
return Map(m)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Free an attribute value
|
||||
attr_value_destroy :: proc(attr: ^Attribute_Value) {
|
||||
switch v in attr {
|
||||
case String:
|
||||
delete(string(v))
|
||||
case Number:
|
||||
delete(string(v))
|
||||
case Binary:
|
||||
delete(string(v))
|
||||
case String_Set:
|
||||
for s in v {
|
||||
delete(s)
|
||||
}
|
||||
delete([]string(v))
|
||||
case Number_Set:
|
||||
for n in v {
|
||||
delete(n)
|
||||
}
|
||||
delete([]string(v))
|
||||
case Binary_Set:
|
||||
for b in v {
|
||||
delete(b)
|
||||
}
|
||||
delete([]string(v))
|
||||
case List:
|
||||
for item in v {
|
||||
item_copy := item
|
||||
attr_value_destroy(&item_copy)
|
||||
}
|
||||
delete([]Attribute_Value(v))
|
||||
case Map:
|
||||
for key, val in v {
|
||||
delete(key)
|
||||
val_copy := val
|
||||
attr_value_destroy(&val_copy)
|
||||
}
|
||||
delete(map[string]Attribute_Value(v))
|
||||
case Bool, Null:
|
||||
// Nothing to free
|
||||
}
|
||||
}
|
||||
|
||||
// Free an item
|
||||
item_destroy :: proc(item: ^Item) {
|
||||
for key, val in item {
|
||||
delete(key)
|
||||
val_copy := val
|
||||
attr_value_destroy(&val_copy)
|
||||
}
|
||||
delete(item^)
|
||||
}
|
||||
429
http.odin
Normal file
429
http.odin
Normal file
@@ -0,0 +1,429 @@
|
||||
package main
|
||||
|
||||
import "core:fmt"
|
||||
import "core:mem"
|
||||
import vmem "core:mem/virtual"
|
||||
import "core:net"
|
||||
import "core:strings"
|
||||
import "core:strconv"
|
||||
|
||||
// HTTP Method enumeration
|
||||
HTTP_Method :: enum {
|
||||
GET,
|
||||
POST,
|
||||
PUT,
|
||||
DELETE,
|
||||
OPTIONS,
|
||||
HEAD,
|
||||
PATCH,
|
||||
}
|
||||
|
||||
method_from_string :: proc(s: string) -> HTTP_Method {
|
||||
switch s {
|
||||
case "GET": return .GET
|
||||
case "POST": return .POST
|
||||
case "PUT": return .PUT
|
||||
case "DELETE": return .DELETE
|
||||
case "OPTIONS": return .OPTIONS
|
||||
case "HEAD": return .HEAD
|
||||
case "PATCH": return .PATCH
|
||||
}
|
||||
return .GET
|
||||
}
|
||||
|
||||
// HTTP Status codes
|
||||
HTTP_Status :: enum u16 {
|
||||
OK = 200,
|
||||
Created = 201,
|
||||
No_Content = 204,
|
||||
Bad_Request = 400,
|
||||
Unauthorized = 401,
|
||||
Forbidden = 403,
|
||||
Not_Found = 404,
|
||||
Method_Not_Allowed = 405,
|
||||
Conflict = 409,
|
||||
Payload_Too_Large = 413,
|
||||
Internal_Server_Error = 500,
|
||||
Service_Unavailable = 503,
|
||||
}
|
||||
|
||||
// HTTP Header
|
||||
HTTP_Header :: struct {
|
||||
name: string,
|
||||
value: string,
|
||||
}
|
||||
|
||||
// HTTP Request
|
||||
HTTP_Request :: struct {
|
||||
method: HTTP_Method,
|
||||
path: string,
|
||||
headers: []HTTP_Header,
|
||||
body: []byte,
|
||||
}
|
||||
|
||||
// Get header value by name (case-insensitive)
|
||||
request_get_header :: proc(req: ^HTTP_Request, name: string) -> Maybe(string) {
|
||||
for header in req.headers {
|
||||
if strings.equal_fold(header.name, name) {
|
||||
return header.value
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// HTTP Response
|
||||
HTTP_Response :: struct {
|
||||
status: HTTP_Status,
|
||||
headers: [dynamic]HTTP_Header,
|
||||
body: [dynamic]byte,
|
||||
}
|
||||
|
||||
response_init :: proc(allocator: mem.Allocator) -> HTTP_Response {
|
||||
return HTTP_Response{
|
||||
status = .OK,
|
||||
headers = make([dynamic]HTTP_Header, allocator),
|
||||
body = make([dynamic]byte, allocator),
|
||||
}
|
||||
}
|
||||
|
||||
response_set_status :: proc(resp: ^HTTP_Response, status: HTTP_Status) {
|
||||
resp.status = status
|
||||
}
|
||||
|
||||
response_add_header :: proc(resp: ^HTTP_Response, name: string, value: string) {
|
||||
append(&resp.headers, HTTP_Header{name = name, value = value})
|
||||
}
|
||||
|
||||
response_set_body :: proc(resp: ^HTTP_Response, data: []byte) {
|
||||
clear(&resp.body)
|
||||
append(&resp.body, ..data)
|
||||
}
|
||||
|
||||
// Request handler function type
|
||||
// Takes context pointer, request, and request-scoped allocator
|
||||
Request_Handler :: #type proc(ctx: rawptr, request: ^HTTP_Request, request_alloc: mem.Allocator) -> HTTP_Response
|
||||
|
||||
// Server configuration
|
||||
Server_Config :: struct {
|
||||
max_body_size: int, // default 100MB
|
||||
max_headers: int, // default 100
|
||||
read_buffer_size: int, // default 8KB
|
||||
enable_keep_alive: bool, // default true
|
||||
max_requests_per_connection: int, // default 1000
|
||||
}
|
||||
|
||||
default_server_config :: proc() -> Server_Config {
|
||||
return Server_Config{
|
||||
max_body_size = 100 * 1024 * 1024,
|
||||
max_headers = 100,
|
||||
read_buffer_size = 8 * 1024,
|
||||
enable_keep_alive = true,
|
||||
max_requests_per_connection = 1000,
|
||||
}
|
||||
}
|
||||
|
||||
// Server
|
||||
Server :: struct {
|
||||
allocator: mem.Allocator,
|
||||
endpoint: net.Endpoint,
|
||||
handler: Request_Handler,
|
||||
handler_ctx: rawptr,
|
||||
config: Server_Config,
|
||||
running: bool,
|
||||
socket: Maybe(net.TCP_Socket),
|
||||
}
|
||||
|
||||
server_init :: proc(
|
||||
allocator: mem.Allocator,
|
||||
host: string,
|
||||
port: int,
|
||||
handler: Request_Handler,
|
||||
handler_ctx: rawptr,
|
||||
config: Server_Config,
|
||||
) -> (Server, bool) {
|
||||
endpoint, endpoint_ok := net.parse_endpoint(fmt.tprintf("%s:%d", host, port))
|
||||
if !endpoint_ok {
|
||||
return {}, false
|
||||
}
|
||||
|
||||
return Server{
|
||||
allocator = allocator,
|
||||
endpoint = endpoint,
|
||||
handler = handler,
|
||||
handler_ctx = handler_ctx,
|
||||
config = config,
|
||||
running = false,
|
||||
socket = nil,
|
||||
}, true
|
||||
}
|
||||
|
||||
server_start :: proc(server: ^Server) -> bool {
|
||||
// Create listening socket
|
||||
socket, socket_err := net.listen_tcp(server.endpoint)
|
||||
if socket_err != nil {
|
||||
fmt.eprintfln("Failed to create listening socket: %v", socket_err)
|
||||
return false
|
||||
}
|
||||
|
||||
server.socket = socket
|
||||
server.running = true
|
||||
|
||||
fmt.printfln("HTTP server listening on %v", server.endpoint)
|
||||
|
||||
// Accept loop
|
||||
for server.running {
|
||||
conn, source, accept_err := net.accept_tcp(socket)
|
||||
if accept_err != nil {
|
||||
if server.running {
|
||||
fmt.eprintfln("Accept error: %v", accept_err)
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
// Handle connection in separate goroutine would go here
|
||||
// For now, handle synchronously (should spawn thread)
|
||||
handle_connection(server, conn, source)
|
||||
}
|
||||
|
||||
return true
|
||||
}
|
||||
|
||||
server_stop :: proc(server: ^Server) {
|
||||
server.running = false
|
||||
if sock, ok := server.socket.?; ok {
|
||||
net.close(sock)
|
||||
server.socket = nil
|
||||
}
|
||||
}
|
||||
|
||||
// Handle a single connection
|
||||
handle_connection :: proc(server: ^Server, conn: net.TCP_Socket, source: net.Endpoint) {
|
||||
defer net.close(conn)
|
||||
|
||||
request_count := 0
|
||||
for request_count < server.config.max_requests_per_connection {
|
||||
request_count += 1
|
||||
|
||||
// Growing arena for this request
|
||||
arena: vmem.Arena
|
||||
arena_err := vmem.arena_init_growing(&arena)
|
||||
if arena_err != .None {
|
||||
break
|
||||
}
|
||||
defer vmem.arena_destroy(&arena)
|
||||
|
||||
request_alloc := vmem.arena_allocator(&arena)
|
||||
|
||||
// TODO: Double check if we want *all* downstream allocations to use the request arena?
|
||||
old := context.allocator
|
||||
context.allocator = request_alloc
|
||||
defer context.allocator = old
|
||||
|
||||
request, parse_ok := parse_request(conn, request_alloc, server.config)
|
||||
if !parse_ok {
|
||||
break
|
||||
}
|
||||
|
||||
response := server.handler(server.handler_ctx, &request, request_alloc)
|
||||
|
||||
send_ok := send_response(conn, &response, request_alloc)
|
||||
if !send_ok {
|
||||
break
|
||||
}
|
||||
|
||||
// Check keep-alive
|
||||
keep_alive := request_get_header(&request, "Connection")
|
||||
if ka, ok := keep_alive.?; ok {
|
||||
if !strings.equal_fold(ka, "keep-alive") {
|
||||
break
|
||||
}
|
||||
} else if !server.config.enable_keep_alive {
|
||||
break
|
||||
}
|
||||
|
||||
// Arena is automatically freed here
|
||||
}
|
||||
}
|
||||
|
||||
// Parse HTTP request
|
||||
parse_request :: proc(
|
||||
conn: net.TCP_Socket,
|
||||
allocator: mem.Allocator,
|
||||
config: Server_Config,
|
||||
) -> (HTTP_Request, bool) {
|
||||
// Read request line and headers
|
||||
buffer := make([]byte, config.read_buffer_size, allocator)
|
||||
|
||||
bytes_read, read_err := net.recv_tcp(conn, buffer)
|
||||
if read_err != nil || bytes_read == 0 {
|
||||
return {}, false
|
||||
}
|
||||
|
||||
request_data := buffer[:bytes_read]
|
||||
|
||||
// Find end of headers (\r\n\r\n)
|
||||
header_end_idx := strings.index(string(request_data), "\r\n\r\n")
|
||||
if header_end_idx < 0 {
|
||||
return {}, false
|
||||
}
|
||||
|
||||
header_section := string(request_data[:header_end_idx])
|
||||
body_start := header_end_idx + 4
|
||||
|
||||
// Parse request line
|
||||
lines := strings.split_lines(header_section, allocator)
|
||||
if len(lines) == 0 {
|
||||
return {}, false
|
||||
}
|
||||
|
||||
request_line := lines[0]
|
||||
parts := strings.split(request_line, " ", allocator)
|
||||
if len(parts) < 3 {
|
||||
return {}, false
|
||||
}
|
||||
|
||||
method := method_from_string(parts[0])
|
||||
path := strings.clone(parts[1], allocator)
|
||||
|
||||
// Parse headers
|
||||
headers := make([dynamic]HTTP_Header, allocator)
|
||||
for i := 1; i < len(lines); i += 1 {
|
||||
line := lines[i]
|
||||
if len(line) == 0 {
|
||||
continue
|
||||
}
|
||||
|
||||
colon_idx := strings.index(line, ":")
|
||||
if colon_idx < 0 {
|
||||
continue
|
||||
}
|
||||
|
||||
name := strings.trim_space(line[:colon_idx])
|
||||
value := strings.trim_space(line[colon_idx+1:])
|
||||
|
||||
append(&headers, HTTP_Header{
|
||||
name = strings.clone(name, allocator),
|
||||
value = strings.clone(value, allocator),
|
||||
})
|
||||
}
|
||||
|
||||
// Read body if Content-Length present
|
||||
body: []byte
|
||||
|
||||
content_length_header := request_get_header_from_slice(headers[:], "Content-Length")
|
||||
if cl, ok := content_length_header.?; ok {
|
||||
content_length := strconv.parse_int(cl) or_else 0
|
||||
|
||||
if content_length > 0 && content_length <= config.max_body_size {
|
||||
// Check if we already have the body in buffer
|
||||
existing_body := request_data[body_start:]
|
||||
|
||||
if len(existing_body) >= content_length {
|
||||
// Body already in buffer
|
||||
body = make([]byte, content_length, allocator)
|
||||
copy(body, existing_body[:content_length])
|
||||
} else {
|
||||
// Need to read more
|
||||
body = make([]byte, content_length, allocator)
|
||||
copy(body, existing_body)
|
||||
|
||||
remaining := content_length - len(existing_body)
|
||||
body_written := len(existing_body)
|
||||
|
||||
for remaining > 0 {
|
||||
chunk_size := min(remaining, config.read_buffer_size)
|
||||
chunk := make([]byte, chunk_size, allocator)
|
||||
|
||||
n, err := net.recv_tcp(conn, chunk)
|
||||
if err != nil || n == 0 {
|
||||
return {}, false
|
||||
}
|
||||
|
||||
copy(body[body_written:], chunk[:n])
|
||||
body_written += n
|
||||
remaining -= n
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return HTTP_Request{
|
||||
method = method,
|
||||
path = path,
|
||||
headers = headers[:],
|
||||
body = body,
|
||||
}, true
|
||||
}
|
||||
|
||||
// Helper to get header from slice
|
||||
request_get_header_from_slice :: proc(headers: []HTTP_Header, name: string) -> Maybe(string) {
|
||||
for header in headers {
|
||||
if strings.equal_fold(header.name, name) {
|
||||
return header.value
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Send HTTP response
|
||||
send_response :: proc(conn: net.TCP_Socket, resp: ^HTTP_Response, allocator: mem.Allocator) -> bool {
|
||||
// Build response string
|
||||
builder := strings.builder_make(allocator)
|
||||
defer strings.builder_destroy(&builder)
|
||||
|
||||
// Status line
|
||||
strings.write_string(&builder, "HTTP/1.1 ")
|
||||
strings.write_int(&builder, int(resp.status))
|
||||
strings.write_string(&builder, " ")
|
||||
strings.write_string(&builder, status_text(resp.status))
|
||||
strings.write_string(&builder, "\r\n")
|
||||
|
||||
// Headers
|
||||
response_add_header(resp, "Content-Length", fmt.tprintf("%d", len(resp.body)))
|
||||
|
||||
for header in resp.headers {
|
||||
strings.write_string(&builder, header.name)
|
||||
strings.write_string(&builder, ": ")
|
||||
strings.write_string(&builder, header.value)
|
||||
strings.write_string(&builder, "\r\n")
|
||||
}
|
||||
|
||||
// End of headers
|
||||
strings.write_string(&builder, "\r\n")
|
||||
|
||||
// Send headers
|
||||
header_bytes := transmute([]byte)strings.to_string(builder)
|
||||
_, send_err := net.send_tcp(conn, header_bytes)
|
||||
if send_err != nil {
|
||||
return false
|
||||
}
|
||||
|
||||
// Send body
|
||||
if len(resp.body) > 0 {
|
||||
_, send_err = net.send_tcp(conn, resp.body[:])
|
||||
if send_err != nil {
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
return true
|
||||
}
|
||||
|
||||
// Get status text for status code
|
||||
status_text :: proc(status: HTTP_Status) -> string {
|
||||
switch status {
|
||||
case .OK: return "OK"
|
||||
case .Created: return "Created"
|
||||
case .No_Content: return "No Content"
|
||||
case .Bad_Request: return "Bad Request"
|
||||
case .Unauthorized: return "Unauthorized"
|
||||
case .Forbidden: return "Forbidden"
|
||||
case .Not_Found: return "Not Found"
|
||||
case .Method_Not_Allowed: return "Method Not Allowed"
|
||||
case .Conflict: return "Conflict"
|
||||
case .Payload_Too_Large: return "Payload Too Large"
|
||||
case .Internal_Server_Error: return "Internal Server Error"
|
||||
case .Service_Unavailable: return "Service Unavailable"
|
||||
}
|
||||
return "Unknown"
|
||||
}
|
||||
253
key_codec/key_codec.odin
Normal file
253
key_codec/key_codec.odin
Normal file
@@ -0,0 +1,253 @@
|
||||
package key_codec
|
||||
|
||||
import "core:bytes"
|
||||
import "core:encoding/varint"
|
||||
import "core:mem"
|
||||
|
||||
// Entity type prefix bytes for namespacing
|
||||
Entity_Type :: enum u8 {
|
||||
Meta = 0x01, // Table metadata
|
||||
Data = 0x02, // Item data
|
||||
GSI = 0x03, // Global secondary index
|
||||
LSI = 0x04, // Local secondary index
|
||||
}
|
||||
|
||||
// Encode a varint length prefix
|
||||
encode_varint :: proc(buf: ^bytes.Buffer, value: int) {
|
||||
temp: [10]byte
|
||||
n := varint.encode_u64(temp[:], u64(value))
|
||||
bytes.buffer_write(buf, temp[:n])
|
||||
}
|
||||
|
||||
// Decode a varint length prefix
|
||||
decode_varint :: proc(data: []byte, offset: ^int) -> (value: int, ok: bool) {
|
||||
if offset^ >= len(data) {
|
||||
return 0, false
|
||||
}
|
||||
|
||||
val, n := varint.decode_u64(data[offset^:])
|
||||
if n <= 0 {
|
||||
return 0, false
|
||||
}
|
||||
|
||||
offset^ += n
|
||||
return int(val), true
|
||||
}
|
||||
|
||||
// Build metadata key: [meta][table_name]
|
||||
build_meta_key :: proc(table_name: string) -> []byte {
|
||||
buf: bytes.Buffer
|
||||
bytes.buffer_init_allocator(&buf, 0, 256, context.allocator)
|
||||
|
||||
// Write entity type
|
||||
bytes.buffer_write_byte(&buf, u8(Entity_Type.Meta))
|
||||
|
||||
// Write table name with length prefix
|
||||
encode_varint(&buf, len(table_name))
|
||||
bytes.buffer_write_string(&buf, table_name)
|
||||
|
||||
return bytes.buffer_to_bytes(&buf)
|
||||
}
|
||||
|
||||
// Build data key: [data][table_name][pk_value][sk_value?]
|
||||
build_data_key :: proc(table_name: string, pk_value: []byte, sk_value: Maybe([]byte)) -> []byte {
|
||||
buf: bytes.Buffer
|
||||
bytes.buffer_init_allocator(&buf, 0, 512, context.allocator)
|
||||
|
||||
// Write entity type
|
||||
bytes.buffer_write_byte(&buf, u8(Entity_Type.Data))
|
||||
|
||||
// Write table name
|
||||
encode_varint(&buf, len(table_name))
|
||||
bytes.buffer_write_string(&buf, table_name)
|
||||
|
||||
// Write partition key
|
||||
encode_varint(&buf, len(pk_value))
|
||||
bytes.buffer_write(&buf, pk_value)
|
||||
|
||||
// Write sort key if present
|
||||
if sk, ok := sk_value.?; ok {
|
||||
encode_varint(&buf, len(sk))
|
||||
bytes.buffer_write(&buf, sk)
|
||||
}
|
||||
|
||||
return bytes.buffer_to_bytes(&buf)
|
||||
}
|
||||
|
||||
// Build table prefix for scanning: [data][table_name]
|
||||
build_table_prefix :: proc(table_name: string) -> []byte {
|
||||
buf: bytes.Buffer
|
||||
bytes.buffer_init_allocator(&buf, 0, 256, context.allocator)
|
||||
|
||||
// Write entity type
|
||||
bytes.buffer_write_byte(&buf, u8(Entity_Type.Data))
|
||||
|
||||
// Write table name
|
||||
encode_varint(&buf, len(table_name))
|
||||
bytes.buffer_write_string(&buf, table_name)
|
||||
|
||||
return bytes.buffer_to_bytes(&buf)
|
||||
}
|
||||
|
||||
// Build partition prefix for querying: [data][table_name][pk_value]
|
||||
build_partition_prefix :: proc(table_name: string, pk_value: []byte) -> []byte {
|
||||
buf: bytes.Buffer
|
||||
bytes.buffer_init_allocator(&buf, 0, 512, context.allocator)
|
||||
|
||||
// Write entity type
|
||||
bytes.buffer_write_byte(&buf, u8(Entity_Type.Data))
|
||||
|
||||
// Write table name
|
||||
encode_varint(&buf, len(table_name))
|
||||
bytes.buffer_write_string(&buf, table_name)
|
||||
|
||||
// Write partition key
|
||||
encode_varint(&buf, len(pk_value))
|
||||
bytes.buffer_write(&buf, pk_value)
|
||||
|
||||
return bytes.buffer_to_bytes(&buf)
|
||||
}
|
||||
|
||||
// Build GSI key: [gsi][table_name][index_name][gsi_pk][gsi_sk?]
|
||||
build_gsi_key :: proc(table_name: string, index_name: string, gsi_pk: []byte, gsi_sk: Maybe([]byte)) -> []byte {
|
||||
buf: bytes.Buffer
|
||||
bytes.buffer_init_allocator(&buf, 0, 512, context.allocator)
|
||||
|
||||
// Write entity type
|
||||
bytes.buffer_write_byte(&buf, u8(Entity_Type.GSI))
|
||||
|
||||
// Write table name
|
||||
encode_varint(&buf, len(table_name))
|
||||
bytes.buffer_write_string(&buf, table_name)
|
||||
|
||||
// Write index name
|
||||
encode_varint(&buf, len(index_name))
|
||||
bytes.buffer_write_string(&buf, index_name)
|
||||
|
||||
// Write GSI partition key
|
||||
encode_varint(&buf, len(gsi_pk))
|
||||
bytes.buffer_write(&buf, gsi_pk)
|
||||
|
||||
// Write GSI sort key if present
|
||||
if sk, ok := gsi_sk.?; ok {
|
||||
encode_varint(&buf, len(sk))
|
||||
bytes.buffer_write(&buf, sk)
|
||||
}
|
||||
|
||||
return bytes.buffer_to_bytes(&buf)
|
||||
}
|
||||
|
||||
// Build LSI key: [lsi][table_name][index_name][pk][lsi_sk]
|
||||
build_lsi_key :: proc(table_name: string, index_name: string, pk: []byte, lsi_sk: []byte) -> []byte {
|
||||
buf: bytes.Buffer
|
||||
bytes.buffer_init_allocator(&buf, 0, 512, context.allocator)
|
||||
|
||||
// Write entity type
|
||||
bytes.buffer_write_byte(&buf, u8(Entity_Type.LSI))
|
||||
|
||||
// Write table name
|
||||
encode_varint(&buf, len(table_name))
|
||||
bytes.buffer_write_string(&buf, table_name)
|
||||
|
||||
// Write index name
|
||||
encode_varint(&buf, len(index_name))
|
||||
bytes.buffer_write_string(&buf, index_name)
|
||||
|
||||
// Write partition key
|
||||
encode_varint(&buf, len(pk))
|
||||
bytes.buffer_write(&buf, pk)
|
||||
|
||||
// Write LSI sort key
|
||||
encode_varint(&buf, len(lsi_sk))
|
||||
bytes.buffer_write(&buf, lsi_sk)
|
||||
|
||||
return bytes.buffer_to_bytes(&buf)
|
||||
}
|
||||
|
||||
// Key decoder for reading binary keys
|
||||
Key_Decoder :: struct {
|
||||
data: []byte,
|
||||
pos: int,
|
||||
}
|
||||
|
||||
decoder_init :: proc(data: []byte) -> Key_Decoder {
|
||||
return Key_Decoder{data = data, pos = 0}
|
||||
}
|
||||
|
||||
decoder_read_entity_type :: proc(decoder: ^Key_Decoder) -> (Entity_Type, bool) {
|
||||
if decoder.pos >= len(decoder.data) {
|
||||
return .Meta, false
|
||||
}
|
||||
|
||||
entity_type := Entity_Type(decoder.data[decoder.pos])
|
||||
decoder.pos += 1
|
||||
return entity_type, true
|
||||
}
|
||||
|
||||
decoder_read_segment :: proc(decoder: ^Key_Decoder) -> (segment: []byte, ok: bool) {
|
||||
// Read length
|
||||
length := decode_varint(decoder.data, &decoder.pos) or_return
|
||||
|
||||
// Read data
|
||||
if decoder.pos + length > len(decoder.data) {
|
||||
return nil, false
|
||||
}
|
||||
|
||||
// Return slice (owned by caller via context.allocator)
|
||||
segment = make([]byte, length, context.allocator)
|
||||
copy(segment, decoder.data[decoder.pos:decoder.pos + length])
|
||||
decoder.pos += length
|
||||
|
||||
return segment, true
|
||||
}
|
||||
|
||||
decoder_read_segment_borrowed :: proc(decoder: ^Key_Decoder) -> (segment: []byte, ok: bool) {
|
||||
// Read length
|
||||
length := decode_varint(decoder.data, &decoder.pos) or_return
|
||||
|
||||
// Return borrowed slice
|
||||
if decoder.pos + length > len(decoder.data) {
|
||||
return nil, false
|
||||
}
|
||||
|
||||
segment = decoder.data[decoder.pos:decoder.pos + length]
|
||||
decoder.pos += length
|
||||
|
||||
return segment, true
|
||||
}
|
||||
|
||||
decoder_has_more :: proc(decoder: ^Key_Decoder) -> bool {
|
||||
return decoder.pos < len(decoder.data)
|
||||
}
|
||||
|
||||
// Decode a data key back into components
|
||||
Decoded_Data_Key :: struct {
|
||||
table_name: string,
|
||||
pk_value: []byte,
|
||||
sk_value: Maybe([]byte),
|
||||
}
|
||||
|
||||
decode_data_key :: proc(key: []byte) -> (result: Decoded_Data_Key, ok: bool) {
|
||||
decoder := decoder_init(key)
|
||||
|
||||
// Read and verify entity type
|
||||
entity_type := decoder_read_entity_type(&decoder) or_return
|
||||
if entity_type != .Data {
|
||||
return {}, false
|
||||
}
|
||||
|
||||
// Read table name
|
||||
table_name_bytes := decoder_read_segment(&decoder) or_return
|
||||
result.table_name = string(table_name_bytes)
|
||||
|
||||
// Read partition key
|
||||
result.pk_value = decoder_read_segment(&decoder) or_return
|
||||
|
||||
// Read sort key if present
|
||||
if decoder_has_more(&decoder) {
|
||||
sk := decoder_read_segment(&decoder) or_return
|
||||
result.sk_value = sk
|
||||
}
|
||||
|
||||
return result, true
|
||||
}
|
||||
137
main.odin
Normal file
137
main.odin
Normal file
@@ -0,0 +1,137 @@
|
||||
package main
|
||||
|
||||
import "core:fmt"
|
||||
import "core:mem"
|
||||
import "core:os"
|
||||
import "core:strconv"
|
||||
//import "core:strings" // I know we'll use in future but because we're not right now, compiler is complaining
|
||||
import "rocksdb"
|
||||
|
||||
Config :: struct {
|
||||
host: string,
|
||||
port: int,
|
||||
data_dir: string,
|
||||
verbose: bool,
|
||||
}
|
||||
|
||||
main :: proc() {
|
||||
// Parse configuration
|
||||
config := parse_config()
|
||||
|
||||
// Print banner
|
||||
print_banner(config)
|
||||
|
||||
// Create data directory
|
||||
os.make_directory(config.data_dir)
|
||||
|
||||
// Initialize storage engine
|
||||
db, err := rocksdb.db_open(config.data_dir, true)
|
||||
if err != .None {
|
||||
fmt.eprintln("Failed to initialize storage:", err)
|
||||
os.exit(1)
|
||||
}
|
||||
defer rocksdb.db_close(&db)
|
||||
|
||||
fmt.printfln("Storage engine initialized at %s", config.data_dir)
|
||||
fmt.printfln("Starting DynamoDB-compatible server on %s:%d", config.host, config.port)
|
||||
|
||||
// Create HTTP server
|
||||
server_config := default_server_config()
|
||||
|
||||
// For now, use a simple echo handler until we implement the full DynamoDB handler
|
||||
server, server_ok := server_init(
|
||||
context.allocator,
|
||||
config.host,
|
||||
config.port,
|
||||
handle_http_request,
|
||||
&db,
|
||||
server_config,
|
||||
)
|
||||
|
||||
if !server_ok {
|
||||
fmt.eprintln("Failed to initialize HTTP server")
|
||||
os.exit(1)
|
||||
}
|
||||
defer server_stop(&server)
|
||||
|
||||
fmt.println("Ready to accept connections!")
|
||||
|
||||
// Start server (blocks)
|
||||
if !server_start(&server) {
|
||||
fmt.eprintln("Server failed to start")
|
||||
os.exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
// Temporary HTTP request handler
|
||||
// TODO: Replace with full DynamoDB handler once dynamodb/handler.odin is implemented
|
||||
handle_http_request :: proc(ctx: rawptr, request: ^HTTP_Request, request_alloc: mem.Allocator) -> HTTP_Response {
|
||||
//db := cast(^rocksdb.DB)ctx // I know we'll use in future but because we're not right now, compiler is complaining
|
||||
|
||||
response := response_init(request_alloc)
|
||||
response_add_header(&response, "Content-Type", "application/x-amz-json-1.0")
|
||||
response_add_header(&response, "x-amzn-RequestId", "local-request-id")
|
||||
|
||||
// Get X-Amz-Target header
|
||||
target := request_get_header(request, "X-Amz-Target")
|
||||
|
||||
if t, ok := target.?; ok {
|
||||
// Echo back the operation for now
|
||||
body := fmt.aprintf("{{\"operation\":\"%s\",\"status\":\"not_implemented\"}}", t)
|
||||
response_set_body(&response, transmute([]byte)body)
|
||||
} else {
|
||||
response_set_status(&response, .Bad_Request)
|
||||
response_set_body(&response, transmute([]byte)string("{\"error\":\"Missing X-Amz-Target header\"}"))
|
||||
}
|
||||
|
||||
return response
|
||||
}
|
||||
|
||||
parse_config :: proc() -> Config {
|
||||
config := Config{
|
||||
host = "0.0.0.0",
|
||||
port = 8002,
|
||||
data_dir = "./data",
|
||||
verbose = false,
|
||||
}
|
||||
|
||||
// Environment variables
|
||||
if port_str, env_ok := os.lookup_env("JORMUN_PORT"); env_ok {
|
||||
if port, parse_ok := strconv.parse_int(port_str); parse_ok {
|
||||
config.port = port
|
||||
}
|
||||
}
|
||||
|
||||
if host, ok := os.lookup_env("JORMUN_HOST"); ok {
|
||||
config.host = host
|
||||
}
|
||||
|
||||
if data_dir, ok := os.lookup_env("JORMUN_DATA_DIR"); ok {
|
||||
config.data_dir = data_dir
|
||||
}
|
||||
|
||||
if verbose, ok := os.lookup_env("JORMUN_VERBOSE"); ok {
|
||||
config.verbose = verbose == "1"
|
||||
}
|
||||
|
||||
// TODO: Parse command line arguments
|
||||
|
||||
return config
|
||||
}
|
||||
|
||||
print_banner :: proc(config: Config) {
|
||||
banner := `
|
||||
╔═══════════════════════════════════════════════╗
|
||||
║ ║
|
||||
║ ╦╔═╗╦═╗╔╦╗╦ ╦╔╗╔╔╦╗╔╗ ║
|
||||
║ ║║ ║╠╦╝║║║║ ║║║║ ║║╠╩╗ ║
|
||||
║ ╚╝╚═╝╩╚═╩ ╩╚═╝╝╚╝═╩╝╚═╝ ║
|
||||
║ ║
|
||||
║ DynamoDB-Compatible Database ║
|
||||
║ Powered by RocksDB + Odin ║
|
||||
║ ║
|
||||
╚═══════════════════════════════════════════════╝
|
||||
`
|
||||
fmt.println(banner)
|
||||
fmt.printfln(" Port: %d | Data Dir: %s\n", config.port, config.data_dir)
|
||||
}
|
||||
6
ols.json
Normal file
6
ols.json
Normal file
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"$schema": "https://raw.githubusercontent.com/DanielGavin/ols/master/misc/ols.schema.json",
|
||||
"enable_document_symbols": true,
|
||||
"enable_hover": true,
|
||||
"enable_snippets": true
|
||||
}
|
||||
3337
project_context.txt
Normal file
3337
project_context.txt
Normal file
File diff suppressed because it is too large
Load Diff
369
rocksdb/rocksdb.odin
Normal file
369
rocksdb/rocksdb.odin
Normal file
@@ -0,0 +1,369 @@
|
||||
package rocksdb
|
||||
|
||||
import "core:c"
|
||||
import "core:fmt"
|
||||
|
||||
foreign import rocksdb "system:rocksdb"
|
||||
|
||||
// RocksDB C API types
|
||||
RocksDB_T :: distinct rawptr
|
||||
RocksDB_Options :: distinct rawptr
|
||||
RocksDB_WriteOptions :: distinct rawptr
|
||||
RocksDB_ReadOptions :: distinct rawptr
|
||||
RocksDB_WriteBatch :: distinct rawptr
|
||||
RocksDB_Iterator :: distinct rawptr
|
||||
RocksDB_FlushOptions :: distinct rawptr
|
||||
|
||||
// Error type
|
||||
Error :: enum {
|
||||
None,
|
||||
OpenFailed,
|
||||
WriteFailed,
|
||||
ReadFailed,
|
||||
DeleteFailed,
|
||||
InvalidArgument,
|
||||
Corruption,
|
||||
NotFound,
|
||||
IOError,
|
||||
Unknown,
|
||||
}
|
||||
|
||||
// Database handle with options
|
||||
DB :: struct {
|
||||
handle: RocksDB_T,
|
||||
options: RocksDB_Options,
|
||||
write_options: RocksDB_WriteOptions,
|
||||
read_options: RocksDB_ReadOptions,
|
||||
}
|
||||
|
||||
// Foreign C functions
|
||||
@(default_calling_convention = "c")
|
||||
foreign rocksdb {
|
||||
// Database operations
|
||||
rocksdb_open :: proc(options: RocksDB_Options, path: cstring, errptr: ^cstring) -> RocksDB_T ---
|
||||
rocksdb_close :: proc(db: RocksDB_T) ---
|
||||
|
||||
// Options
|
||||
rocksdb_options_create :: proc() -> RocksDB_Options ---
|
||||
rocksdb_options_destroy :: proc(options: RocksDB_Options) ---
|
||||
rocksdb_options_set_create_if_missing :: proc(options: RocksDB_Options, val: c.uchar) ---
|
||||
rocksdb_options_increase_parallelism :: proc(options: RocksDB_Options, total_threads: c.int) ---
|
||||
rocksdb_options_optimize_level_style_compaction :: proc(options: RocksDB_Options, memtable_memory_budget: c.uint64_t) ---
|
||||
rocksdb_options_set_compression :: proc(options: RocksDB_Options, compression: c.int) ---
|
||||
|
||||
// Write options
|
||||
rocksdb_writeoptions_create :: proc() -> RocksDB_WriteOptions ---
|
||||
rocksdb_writeoptions_destroy :: proc(options: RocksDB_WriteOptions) ---
|
||||
|
||||
// Read options
|
||||
rocksdb_readoptions_create :: proc() -> RocksDB_ReadOptions ---
|
||||
rocksdb_readoptions_destroy :: proc(options: RocksDB_ReadOptions) ---
|
||||
|
||||
// Put/Get/Delete
|
||||
rocksdb_put :: proc(db: RocksDB_T, options: RocksDB_WriteOptions, key: [^]byte, keylen: c.size_t, val: [^]byte, vallen: c.size_t, errptr: ^cstring) ---
|
||||
rocksdb_get :: proc(db: RocksDB_T, options: RocksDB_ReadOptions, key: [^]byte, keylen: c.size_t, vallen: ^c.size_t, errptr: ^cstring) -> [^]byte ---
|
||||
rocksdb_delete :: proc(db: RocksDB_T, options: RocksDB_WriteOptions, key: [^]byte, keylen: c.size_t, errptr: ^cstring) ---
|
||||
|
||||
// Flush
|
||||
rocksdb_flushoptions_create :: proc() -> RocksDB_FlushOptions ---
|
||||
rocksdb_flushoptions_destroy :: proc(options: RocksDB_FlushOptions) ---
|
||||
rocksdb_flush :: proc(db: RocksDB_T, options: RocksDB_FlushOptions, errptr: ^cstring) ---
|
||||
|
||||
// Write batch
|
||||
rocksdb_writebatch_create :: proc() -> RocksDB_WriteBatch ---
|
||||
rocksdb_writebatch_destroy :: proc(batch: RocksDB_WriteBatch) ---
|
||||
rocksdb_writebatch_put :: proc(batch: RocksDB_WriteBatch, key: [^]byte, keylen: c.size_t, val: [^]byte, vallen: c.size_t) ---
|
||||
rocksdb_writebatch_delete :: proc(batch: RocksDB_WriteBatch, key: [^]byte, keylen: c.size_t) ---
|
||||
rocksdb_writebatch_clear :: proc(batch: RocksDB_WriteBatch) ---
|
||||
rocksdb_write :: proc(db: RocksDB_T, options: RocksDB_WriteOptions, batch: RocksDB_WriteBatch, errptr: ^cstring) ---
|
||||
|
||||
// Iterator
|
||||
rocksdb_create_iterator :: proc(db: RocksDB_T, options: RocksDB_ReadOptions) -> RocksDB_Iterator ---
|
||||
rocksdb_iter_destroy :: proc(iter: RocksDB_Iterator) ---
|
||||
rocksdb_iter_seek_to_first :: proc(iter: RocksDB_Iterator) ---
|
||||
rocksdb_iter_seek_to_last :: proc(iter: RocksDB_Iterator) ---
|
||||
rocksdb_iter_seek :: proc(iter: RocksDB_Iterator, key: [^]byte, keylen: c.size_t) ---
|
||||
rocksdb_iter_seek_for_prev :: proc(iter: RocksDB_Iterator, key: [^]byte, keylen: c.size_t) ---
|
||||
rocksdb_iter_valid :: proc(iter: RocksDB_Iterator) -> c.uchar ---
|
||||
rocksdb_iter_next :: proc(iter: RocksDB_Iterator) ---
|
||||
rocksdb_iter_prev :: proc(iter: RocksDB_Iterator) ---
|
||||
rocksdb_iter_key :: proc(iter: RocksDB_Iterator, klen: ^c.size_t) -> [^]byte ---
|
||||
rocksdb_iter_value :: proc(iter: RocksDB_Iterator, vlen: ^c.size_t) -> [^]byte ---
|
||||
|
||||
// Memory management
|
||||
rocksdb_free :: proc(ptr: rawptr) ---
|
||||
}
|
||||
|
||||
// Compression types
|
||||
ROCKSDB_NO_COMPRESSION :: 0
|
||||
ROCKSDB_SNAPPY_COMPRESSION :: 1
|
||||
ROCKSDB_ZLIB_COMPRESSION :: 2
|
||||
ROCKSDB_BZIP2_COMPRESSION :: 3
|
||||
ROCKSDB_LZ4_COMPRESSION :: 4
|
||||
ROCKSDB_LZ4HC_COMPRESSION :: 5
|
||||
ROCKSDB_ZSTD_COMPRESSION :: 7
|
||||
|
||||
// Open a database
|
||||
db_open :: proc(path: string, create_if_missing := true) -> (DB, Error) {
|
||||
options := rocksdb_options_create()
|
||||
if options == nil {
|
||||
return {}, .Unknown
|
||||
}
|
||||
|
||||
// Set create if missing
|
||||
rocksdb_options_set_create_if_missing(options, create_if_missing ? 1 : 0)
|
||||
|
||||
// Performance optimizations
|
||||
rocksdb_options_increase_parallelism(options, 4)
|
||||
rocksdb_options_optimize_level_style_compaction(options, 512 * 1024 * 1024)
|
||||
rocksdb_options_set_compression(options, ROCKSDB_LZ4_COMPRESSION)
|
||||
|
||||
// Create write and read options
|
||||
write_options := rocksdb_writeoptions_create()
|
||||
if write_options == nil {
|
||||
rocksdb_options_destroy(options)
|
||||
return {}, .Unknown
|
||||
}
|
||||
|
||||
read_options := rocksdb_readoptions_create()
|
||||
if read_options == nil {
|
||||
rocksdb_writeoptions_destroy(write_options)
|
||||
rocksdb_options_destroy(options)
|
||||
return {}, .Unknown
|
||||
}
|
||||
|
||||
// Open database
|
||||
err: cstring
|
||||
path_cstr := fmt.ctprintf("%s", path)
|
||||
handle := rocksdb_open(options, path_cstr, &err)
|
||||
if err != nil {
|
||||
defer rocksdb_free(rawptr(err)) // Cast it here and now so we don't deal with issues from FFI down the line
|
||||
rocksdb_readoptions_destroy(read_options)
|
||||
rocksdb_writeoptions_destroy(write_options)
|
||||
rocksdb_options_destroy(options)
|
||||
return {}, .OpenFailed
|
||||
}
|
||||
|
||||
return DB{
|
||||
handle = handle,
|
||||
options = options,
|
||||
write_options = write_options,
|
||||
read_options = read_options,
|
||||
}, .None
|
||||
}
|
||||
|
||||
// Close database
|
||||
db_close :: proc(db: ^DB) {
|
||||
rocksdb_readoptions_destroy(db.read_options)
|
||||
rocksdb_writeoptions_destroy(db.write_options)
|
||||
rocksdb_close(db.handle)
|
||||
rocksdb_options_destroy(db.options)
|
||||
}
|
||||
|
||||
// Put key-value pair
|
||||
db_put :: proc(db: ^DB, key: []byte, value: []byte) -> Error {
|
||||
err: cstring
|
||||
rocksdb_put(
|
||||
db.handle,
|
||||
db.write_options,
|
||||
raw_data(key),
|
||||
c.size_t(len(key)),
|
||||
raw_data(value),
|
||||
c.size_t(len(value)),
|
||||
&err,
|
||||
)
|
||||
if err != nil {
|
||||
defer rocksdb_free(rawptr(err)) // Cast it here and now so we don't deal with issues from FFI down the line
|
||||
return .WriteFailed
|
||||
}
|
||||
return .None
|
||||
}
|
||||
|
||||
// Get value by key (returns owned slice - caller must free)
|
||||
db_get :: proc(db: ^DB, key: []byte) -> (value: []byte, err: Error) {
|
||||
errptr: cstring
|
||||
value_len: c.size_t
|
||||
|
||||
value_ptr := rocksdb_get(
|
||||
db.handle,
|
||||
db.read_options,
|
||||
raw_data(key),
|
||||
c.size_t(len(key)),
|
||||
&value_len,
|
||||
&errptr,
|
||||
)
|
||||
|
||||
if errptr != nil {
|
||||
defer rocksdb_free(rawptr(errptr)) // Cast it here and now so we don't deal with issues from FFI down the line
|
||||
return nil, .ReadFailed
|
||||
}
|
||||
|
||||
if value_ptr == nil {
|
||||
return nil, .NotFound
|
||||
}
|
||||
|
||||
// Copy the data and free RocksDB's buffer
|
||||
result := make([]byte, value_len, context.allocator)
|
||||
copy(result, value_ptr[:value_len])
|
||||
rocksdb_free(rawptr(value_ptr)) // Cast it here and now so we don't deal with issues from FFI down the line
|
||||
|
||||
return result, .None
|
||||
}
|
||||
|
||||
// Delete key
|
||||
db_delete :: proc(db: ^DB, key: []byte) -> Error {
|
||||
err: cstring
|
||||
rocksdb_delete(
|
||||
db.handle,
|
||||
db.write_options,
|
||||
raw_data(key),
|
||||
c.size_t(len(key)),
|
||||
&err,
|
||||
)
|
||||
if err != nil {
|
||||
defer rocksdb_free(rawptr(err)) // Cast it here and now so we don't deal with issues from FFI down the line
|
||||
return .DeleteFailed
|
||||
}
|
||||
return .None
|
||||
}
|
||||
|
||||
// Flush database
|
||||
db_flush :: proc(db: ^DB) -> Error {
|
||||
flush_opts := rocksdb_flushoptions_create()
|
||||
if flush_opts == nil {
|
||||
return .Unknown
|
||||
}
|
||||
defer rocksdb_flushoptions_destroy(flush_opts)
|
||||
|
||||
err: cstring
|
||||
rocksdb_flush(db.handle, flush_opts, &err)
|
||||
if err != nil {
|
||||
defer rocksdb_free(rawptr(err)) // Cast it here and now so we don't deal with issues from FFI down the line
|
||||
return .IOError
|
||||
}
|
||||
return .None
|
||||
}
|
||||
|
||||
// Write batch
|
||||
WriteBatch :: struct {
|
||||
handle: RocksDB_WriteBatch,
|
||||
}
|
||||
|
||||
// Create write batch
|
||||
batch_create :: proc() -> (WriteBatch, Error) {
|
||||
handle := rocksdb_writebatch_create()
|
||||
if handle == nil {
|
||||
return {}, .Unknown
|
||||
}
|
||||
return WriteBatch{handle = handle}, .None
|
||||
}
|
||||
|
||||
// Destroy write batch
|
||||
batch_destroy :: proc(batch: ^WriteBatch) {
|
||||
rocksdb_writebatch_destroy(batch.handle)
|
||||
}
|
||||
|
||||
// Add put operation to batch
|
||||
batch_put :: proc(batch: ^WriteBatch, key: []byte, value: []byte) {
|
||||
rocksdb_writebatch_put(
|
||||
batch.handle,
|
||||
raw_data(key),
|
||||
c.size_t(len(key)),
|
||||
raw_data(value),
|
||||
c.size_t(len(value)),
|
||||
)
|
||||
}
|
||||
|
||||
// Add delete operation to batch
|
||||
batch_delete :: proc(batch: ^WriteBatch, key: []byte) {
|
||||
rocksdb_writebatch_delete(
|
||||
batch.handle,
|
||||
raw_data(key),
|
||||
c.size_t(len(key)),
|
||||
)
|
||||
}
|
||||
|
||||
// Clear batch
|
||||
batch_clear :: proc(batch: ^WriteBatch) {
|
||||
rocksdb_writebatch_clear(batch.handle)
|
||||
}
|
||||
|
||||
// Write batch to database
|
||||
batch_write :: proc(db: ^DB, batch: ^WriteBatch) -> Error {
|
||||
err: cstring
|
||||
rocksdb_write(db.handle, db.write_options, batch.handle, &err)
|
||||
if err != nil {
|
||||
defer rocksdb_free(rawptr(err)) // Cast it here and now so we don't deal with issues from FFI down the line
|
||||
return .WriteFailed
|
||||
}
|
||||
return .None
|
||||
}
|
||||
|
||||
// Iterator
|
||||
Iterator :: struct {
|
||||
handle: RocksDB_Iterator,
|
||||
}
|
||||
|
||||
// Create iterator
|
||||
iter_create :: proc(db: ^DB) -> (Iterator, Error) {
|
||||
handle := rocksdb_create_iterator(db.handle, db.read_options)
|
||||
if handle == nil {
|
||||
return {}, .Unknown
|
||||
}
|
||||
return Iterator{handle = handle}, .None
|
||||
}
|
||||
|
||||
// Destroy iterator
|
||||
iter_destroy :: proc(iter: ^Iterator) {
|
||||
rocksdb_iter_destroy(iter.handle)
|
||||
}
|
||||
|
||||
// Seek to first
|
||||
iter_seek_to_first :: proc(iter: ^Iterator) {
|
||||
rocksdb_iter_seek_to_first(iter.handle)
|
||||
}
|
||||
|
||||
// Seek to last
|
||||
iter_seek_to_last :: proc(iter: ^Iterator) {
|
||||
rocksdb_iter_seek_to_last(iter.handle)
|
||||
}
|
||||
|
||||
// Seek to key
|
||||
iter_seek :: proc(iter: ^Iterator, target: []byte) {
|
||||
rocksdb_iter_seek(iter.handle, raw_data(target), c.size_t(len(target)))
|
||||
}
|
||||
|
||||
// Check if iterator is valid
|
||||
iter_valid :: proc(iter: ^Iterator) -> bool {
|
||||
return rocksdb_iter_valid(iter.handle) != 0
|
||||
}
|
||||
|
||||
// Move to next
|
||||
iter_next :: proc(iter: ^Iterator) {
|
||||
rocksdb_iter_next(iter.handle)
|
||||
}
|
||||
|
||||
// Move to previous
|
||||
iter_prev :: proc(iter: ^Iterator) {
|
||||
rocksdb_iter_prev(iter.handle)
|
||||
}
|
||||
|
||||
// Get current key (returns borrowed slice)
|
||||
iter_key :: proc(iter: ^Iterator) -> []byte {
|
||||
klen: c.size_t
|
||||
key_ptr := rocksdb_iter_key(iter.handle, &klen)
|
||||
if key_ptr == nil {
|
||||
return nil
|
||||
}
|
||||
return key_ptr[:klen]
|
||||
}
|
||||
|
||||
// Get current value (returns borrowed slice)
|
||||
iter_value :: proc(iter: ^Iterator) -> []byte {
|
||||
vlen: c.size_t
|
||||
value_ptr := rocksdb_iter_value(iter.handle, &vlen)
|
||||
if value_ptr == nil {
|
||||
return nil
|
||||
}
|
||||
return value_ptr[:vlen]
|
||||
}
|
||||
Reference in New Issue
Block a user