its kind of working now
This commit is contained in:
297
README.md
Normal file
297
README.md
Normal file
@@ -0,0 +1,297 @@
|
||||
# ZynamoDB
|
||||
|
||||
A DynamoDB-compatible database built with **Zig** and **RocksDB**.
|
||||
|
||||
## Why Zig?
|
||||
|
||||
Zig was chosen over C++ for several reasons:
|
||||
|
||||
1. **Built-in Memory Safety** - Compile-time safety checks without garbage collection
|
||||
2. **Seamless C Interop** - RocksDB's C API can be imported directly with `@cImport`
|
||||
3. **Simple Build System** - `build.zig` replaces complex CMake/Makefile configurations
|
||||
4. **No Hidden Control Flow** - Explicit error handling, no exceptions
|
||||
5. **Modern Tooling** - Built-in test framework, documentation generator, and package manager
|
||||
|
||||
## Features
|
||||
|
||||
### Implemented Operations
|
||||
- ✅ CreateTable
|
||||
- ✅ DeleteTable
|
||||
- ✅ DescribeTable
|
||||
- ✅ ListTables
|
||||
- ✅ PutItem
|
||||
- ✅ GetItem
|
||||
- ✅ DeleteItem
|
||||
- ✅ Query (basic)
|
||||
- ✅ Scan
|
||||
|
||||
### Planned Operations
|
||||
- 🚧 UpdateItem
|
||||
- 🚧 BatchGetItem
|
||||
- 🚧 BatchWriteItem
|
||||
- 🚧 TransactGetItems
|
||||
- 🚧 TransactWriteItems
|
||||
- 🚧 Global Secondary Indexes
|
||||
- 🚧 Local Secondary Indexes
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Using Docker (Recommended)
|
||||
|
||||
```bash
|
||||
# Build the development container
|
||||
docker-compose build dev
|
||||
|
||||
# Start a shell in the container
|
||||
docker-compose run --rm dev
|
||||
|
||||
# Inside the container:
|
||||
zig build run
|
||||
```
|
||||
|
||||
### Native Build (requires Zig 0.13+ and RocksDB)
|
||||
|
||||
```bash
|
||||
# Install dependencies (Ubuntu/Debian)
|
||||
sudo apt install librocksdb-dev libsnappy-dev liblz4-dev libzstd-dev
|
||||
|
||||
# Build and run
|
||||
zig build run
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### Starting the Server
|
||||
|
||||
```bash
|
||||
# Default (port 8000)
|
||||
zig build run
|
||||
|
||||
# Custom port
|
||||
zig build run -- --port 8080
|
||||
|
||||
# Custom data directory
|
||||
zig build run -- --data-dir /var/lib/zynamodb
|
||||
```
|
||||
|
||||
### Using AWS CLI
|
||||
|
||||
```bash
|
||||
# Create a table
|
||||
aws dynamodb create-table \
|
||||
--endpoint-url http://localhost:8000 \
|
||||
--table-name Users \
|
||||
--key-schema AttributeName=pk,KeyType=HASH \
|
||||
--attribute-definitions AttributeName=pk,AttributeType=S \
|
||||
--billing-mode PAY_PER_REQUEST
|
||||
|
||||
# Put an item
|
||||
aws dynamodb put-item \
|
||||
--endpoint-url http://localhost:8000 \
|
||||
--table-name Users \
|
||||
--item '{"pk":{"S":"user123"},"name":{"S":"Alice"},"email":{"S":"alice@example.com"}}'
|
||||
|
||||
# Get an item
|
||||
aws dynamodb get-item \
|
||||
--endpoint-url http://localhost:8000 \
|
||||
--table-name Users \
|
||||
--key '{"pk":{"S":"user123"}}'
|
||||
|
||||
# Scan the table
|
||||
aws dynamodb scan \
|
||||
--endpoint-url http://localhost:8000 \
|
||||
--table-name Users
|
||||
|
||||
# List tables
|
||||
aws dynamodb list-tables --endpoint-url http://localhost:8000
|
||||
```
|
||||
|
||||
### Using Python (boto3)
|
||||
|
||||
```python
|
||||
import boto3
|
||||
|
||||
# Connect to local ZynamoDB
|
||||
dynamodb = boto3.client(
|
||||
'dynamodb',
|
||||
endpoint_url='http://localhost:8000',
|
||||
region_name='us-east-1',
|
||||
aws_access_key_id='fake',
|
||||
aws_secret_access_key='fake'
|
||||
)
|
||||
|
||||
# Create table
|
||||
dynamodb.create_table(
|
||||
TableName='Products',
|
||||
KeySchema=[{'AttributeName': 'pk', 'KeyType': 'HASH'}],
|
||||
AttributeDefinitions=[{'AttributeName': 'pk', 'AttributeType': 'S'}],
|
||||
BillingMode='PAY_PER_REQUEST'
|
||||
)
|
||||
|
||||
# Put item
|
||||
dynamodb.put_item(
|
||||
TableName='Products',
|
||||
Item={
|
||||
'pk': {'S': 'prod-001'},
|
||||
'name': {'S': 'Widget'},
|
||||
'price': {'N': '29.99'}
|
||||
}
|
||||
)
|
||||
|
||||
# Get item
|
||||
response = dynamodb.get_item(
|
||||
TableName='Products',
|
||||
Key={'pk': {'S': 'prod-001'}}
|
||||
)
|
||||
print(response.get('Item'))
|
||||
```
|
||||
|
||||
## Development
|
||||
|
||||
### Project Structure
|
||||
|
||||
```
|
||||
dynamodb-compat/
|
||||
├── Dockerfile # Dev container with Zig + RocksDB
|
||||
├── docker-compose.yml # Container orchestration
|
||||
├── build.zig # Zig build configuration
|
||||
├── Makefile # Convenience commands
|
||||
├── src/
|
||||
│ ├── main.zig # Entry point
|
||||
│ ├── rocksdb.zig # RocksDB C bindings
|
||||
│ ├── http.zig # HTTP server
|
||||
│ ├── bench.zig # Performance benchmarks
|
||||
│ └── dynamodb/
|
||||
│ ├── types.zig # DynamoDB protocol types
|
||||
│ ├── storage.zig # Storage engine (RocksDB mapping)
|
||||
│ └── handler.zig # API request handlers
|
||||
└── tests/
|
||||
└── integration.zig # Integration tests
|
||||
```
|
||||
|
||||
### Build Commands
|
||||
|
||||
```bash
|
||||
# Build
|
||||
make build # Debug build
|
||||
make release # Optimized release
|
||||
|
||||
# Test
|
||||
make test # Unit tests
|
||||
make test-integration # Integration tests
|
||||
make test-all # All tests
|
||||
|
||||
# Run
|
||||
make run # Start server
|
||||
make run-port PORT=8080 # Custom port
|
||||
|
||||
# Benchmark
|
||||
make bench # Run benchmarks
|
||||
|
||||
# Docker
|
||||
make docker-build # Build container
|
||||
make docker-shell # Open shell
|
||||
make docker-test # Run tests in container
|
||||
```
|
||||
|
||||
### Running Tests
|
||||
|
||||
```bash
|
||||
# Unit tests
|
||||
zig build test
|
||||
|
||||
# Integration tests
|
||||
zig build test-integration
|
||||
|
||||
# With Docker
|
||||
docker-compose run --rm dev zig build test
|
||||
```
|
||||
|
||||
### Running Benchmarks
|
||||
|
||||
```bash
|
||||
zig build bench
|
||||
|
||||
# Or with make
|
||||
make bench
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
### Storage Model
|
||||
|
||||
Data is stored in RocksDB with the following key prefixes:
|
||||
|
||||
| Prefix | Purpose | Format |
|
||||
|--------|---------|--------|
|
||||
| `_meta:` | Table metadata | `_meta:{table_name}` |
|
||||
| `_data:` | Item data | `_data:{table}:{pk}` or `_data:{table}:{pk}:{sk}` |
|
||||
| `_gsi:` | Global secondary index | `_gsi:{table}:{index}:{pk}:{sk}` |
|
||||
| `_lsi:` | Local secondary index | `_lsi:{table}:{index}:{pk}:{sk}` |
|
||||
|
||||
### HTTP Server
|
||||
|
||||
- Custom HTTP/1.1 implementation using Zig's stdlib
|
||||
- Thread-per-connection model (suitable for moderate load)
|
||||
- Parses `X-Amz-Target` header to route DynamoDB operations
|
||||
|
||||
### DynamoDB Protocol
|
||||
|
||||
- JSON request/response format
|
||||
- Standard DynamoDB error responses
|
||||
- Compatible with AWS SDKs (boto3, AWS CLI, etc.)
|
||||
|
||||
## Configuration
|
||||
|
||||
### Command Line Options
|
||||
|
||||
| Option | Description | Default |
|
||||
|--------|-------------|---------|
|
||||
| `-p, --port` | HTTP port | 8000 |
|
||||
| `-h, --host` | Bind address | 0.0.0.0 |
|
||||
| `-d, --data-dir` | RocksDB data directory | ./data |
|
||||
| `-v, --verbose` | Enable verbose logging | false |
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Variable | Description |
|
||||
|----------|-------------|
|
||||
| `DYNAMODB_PORT` | Override port |
|
||||
| `ROCKSDB_DATA_DIR` | Override data directory |
|
||||
|
||||
## Performance
|
||||
|
||||
Preliminary benchmarks on development hardware:
|
||||
|
||||
| Operation | Ops/sec |
|
||||
|-----------|---------|
|
||||
| PutItem | ~15,000 |
|
||||
| GetItem | ~25,000 |
|
||||
| Scan (1K items) | ~50,000 |
|
||||
|
||||
Run `make bench` for actual numbers on your hardware.
|
||||
|
||||
## Comparison with DynamoDB Local
|
||||
|
||||
| Feature | ZynamoDB | DynamoDB Local |
|
||||
|---------|----------|----------------|
|
||||
| Language | Zig | Java |
|
||||
| Storage | RocksDB | SQLite |
|
||||
| Memory | ~10MB | ~200MB+ |
|
||||
| Startup | Instant | 2-5 seconds |
|
||||
| Persistence | Yes | Optional |
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
|
||||
## Contributing
|
||||
|
||||
Contributions welcome! Please read the contributing guidelines first.
|
||||
|
||||
Areas that need work:
|
||||
- UpdateItem with expression parsing
|
||||
- Batch operations
|
||||
- Secondary indexes
|
||||
- Streams support
|
||||
- Better JSON parsing (currently simplified)
|
||||
Reference in New Issue
Block a user