zyna-db/README.md

# ZynamoDB

A DynamoDB-compatible database built with **Zig** and **RocksDB**.

## Why Zig?

Zig was chosen over C++ for several reasons:

1. **Built-in Memory Safety** - Compile-time safety checks without garbage collection
2. **Seamless C Interop** - RocksDB's C API can be imported directly with `@cImport`
3. **Simple Build System** - `build.zig` replaces complex CMake/Makefile configurations
4. **No Hidden Control Flow** - Explicit error handling, no exceptions
5. **Modern Tooling** - Built-in test framework, documentation generator, and package manager

## Features

### Implemented Operations
- ✅ CreateTable
- ✅ DeleteTable
- ✅ DescribeTable
- ✅ ListTables
- ✅ PutItem
- ✅ GetItem
- ✅ DeleteItem
- ✅ Query (basic)
- ✅ Scan

### Planned Operations
- 🚧 UpdateItem
- 🚧 BatchGetItem
- 🚧 BatchWriteItem
- 🚧 TransactGetItems
- 🚧 TransactWriteItems
- 🚧 Global Secondary Indexes
- 🚧 Local Secondary Indexes

## Quick Start

### Using Docker (Recommended)

```bash
# Build the development container
docker-compose build dev

# Start a shell in the container
docker-compose run --rm dev

# Inside the container:
zig build run
```

### Native Build (requires Zig 0.13+ and RocksDB)

```bash
# Install dependencies (Ubuntu/Debian)
sudo apt install librocksdb-dev libsnappy-dev liblz4-dev libzstd-dev

# Build and run
zig build run
```

## Usage

### Starting the Server

```bash
# Default (port 8000)
zig build run

# Custom port
zig build run -- --port 8080

# Custom data directory
zig build run -- --data-dir /var/lib/zynamodb
```

### Using AWS CLI

```bash
# Create a table
aws dynamodb create-table \
    --endpoint-url http://localhost:8000 \
    --table-name Users \
    --key-schema AttributeName=pk,KeyType=HASH \
    --attribute-definitions AttributeName=pk,AttributeType=S \
    --billing-mode PAY_PER_REQUEST

# Put an item
aws dynamodb put-item \
    --endpoint-url http://localhost:8000 \
    --table-name Users \
    --item '{"pk":{"S":"user123"},"name":{"S":"Alice"},"email":{"S":"alice@example.com"}}'

# Get an item
aws dynamodb get-item \
    --endpoint-url http://localhost:8000 \
    --table-name Users \
    --key '{"pk":{"S":"user123"}}'

# Scan the table
aws dynamodb scan \
    --endpoint-url http://localhost:8000 \
    --table-name Users

# List tables
aws dynamodb list-tables --endpoint-url http://localhost:8000
```

### Using Python (boto3)

```python
import boto3

# Connect to local ZynamoDB
dynamodb = boto3.client(
    'dynamodb',
    endpoint_url='http://localhost:8000',
    region_name='us-east-1',
    aws_access_key_id='fake',
    aws_secret_access_key='fake'
)

# Create table
dynamodb.create_table(
    TableName='Products',
    KeySchema=[{'AttributeName': 'pk', 'KeyType': 'HASH'}],
    AttributeDefinitions=[{'AttributeName': 'pk', 'AttributeType': 'S'}],
    BillingMode='PAY_PER_REQUEST'
)

# Put item
dynamodb.put_item(
    TableName='Products',
    Item={
        'pk': {'S': 'prod-001'},
        'name': {'S': 'Widget'},
        'price': {'N': '29.99'}
    }
)

# Get item
response = dynamodb.get_item(
    TableName='Products',
    Key={'pk': {'S': 'prod-001'}}
)
print(response.get('Item'))
```

## Development

### Project Structure

```
dynamodb-compat/
├── Dockerfile              # Dev container with Zig + RocksDB
├── docker-compose.yml      # Container orchestration
├── build.zig               # Zig build configuration
├── Makefile                # Convenience commands
├── src/
│   ├── main.zig            # Entry point
│   ├── rocksdb.zig         # RocksDB C bindings
│   ├── http.zig            # HTTP server
│   ├── bench.zig           # Performance benchmarks
│   └── dynamodb/
│       ├── types.zig       # DynamoDB protocol types
│       ├── storage.zig     # Storage engine (RocksDB mapping)
│       └── handler.zig     # API request handlers
└── tests/
    └── integration.zig     # Integration tests
```

### Build Commands

```bash
# Build
make build              # Debug build
make release            # Optimized release

# Test
make test               # Unit tests
make test-integration   # Integration tests
make test-all           # All tests

# Run
make run                # Start server
make run-port PORT=8080 # Custom port

# Benchmark
make bench              # Run benchmarks

# Docker
make docker-build       # Build container
make docker-shell       # Open shell
make docker-test        # Run tests in container
```

### Running Tests

```bash
# Unit tests
zig build test

# Integration tests
zig build test-integration

# With Docker
docker-compose run --rm dev zig build test
```

### Running Benchmarks

```bash
zig build bench

# Or with make
make bench
```

## Architecture

### Storage Model

Data is stored in RocksDB with the following key prefixes:

| Prefix | Purpose | Format |
|--------|---------|--------|
| `_meta:` | Table metadata | `_meta:{table_name}` |
| `_data:` | Item data | `_data:{table}:{pk}` or `_data:{table}:{pk}:{sk}` |
| `_gsi:` | Global secondary index | `_gsi:{table}:{index}:{pk}:{sk}` |
| `_lsi:` | Local secondary index | `_lsi:{table}:{index}:{pk}:{sk}` |

### HTTP Server

- Custom HTTP/1.1 implementation using Zig's stdlib
- Thread-per-connection model (suitable for moderate load)
- Parses `X-Amz-Target` header to route DynamoDB operations

### DynamoDB Protocol

- JSON request/response format
- Standard DynamoDB error responses
- Compatible with AWS SDKs (boto3, AWS CLI, etc.)

## Configuration

### Command Line Options

| Option | Description | Default |
|--------|-------------|---------|
| `-p, --port` | HTTP port | 8000 |
| `-h, --host` | Bind address | 0.0.0.0 |
| `-d, --data-dir` | RocksDB data directory | ./data |
| `-v, --verbose` | Enable verbose logging | false |

### Environment Variables

| Variable | Description |
|----------|-------------|
| `DYNAMODB_PORT` | Override port |
| `ROCKSDB_DATA_DIR` | Override data directory |

## Performance

Preliminary benchmarks on development hardware:

| Operation | Ops/sec |
|-----------|---------|
| PutItem | ~15,000 |
| GetItem | ~25,000 |
| Scan (1K items) | ~50,000 |

Run `make bench` for actual numbers on your hardware.

## Comparison with DynamoDB Local

| Feature | ZynamoDB | DynamoDB Local |
|---------|----------|----------------|
| Language | Zig | Java |
| Storage | RocksDB | SQLite |
| Memory | ~10MB | ~200MB+ |
| Startup | Instant | 2-5 seconds |
| Persistence | Yes | Optional |

## License

MIT

## Contributing

Contributions welcome! Please read the contributing guidelines first.

Areas that need work:
- UpdateItem with expression parsing
- Batch operations
- Secondary indexes
- Streams support
- Better JSON parsing (currently simplified)