fix storage issues

TODO.md

@@ -1,6 +1,6 @@
 # JormunDB (Odin rewrite) — TODO
 
-This tracks the rewrite from Zig (ZynamoDB) → Odin (JormunDB), and what’s left to stabilize + extend.
+This tracks the rewrite from Zig (ZynamoDB) → Odin (JormunDB), and what's left to stabilize + extend.
 
 ## Status Snapshot
 
@@ -19,52 +19,57 @@ This tracks the rewrite from Zig (ZynamoDB) → Odin (JormunDB), and what’s le
 ---
 
 ## Now (MVP correctness + polish)
-Goal: “aws cli works reliably for CreateTable/ListTables/PutItem/GetItem/DeleteItem/Scan/Query” with correct DynamoDB-ish responses.
+Goal: "aws cli works reliably for CreateTable/ListTables/PutItem/GetItem/DeleteItem/Scan/Query" with correct DynamoDB-ish responses.
 
 ### 1) HTTP + routing hardening
 - [ ] Audit request parsing boundaries:
-  - Max body size enforcement
+  - Max body size enforcement (config exists, need to verify enforcement path)
   - Missing/invalid headers → correct DynamoDB error types
   - Content-Type handling (be permissive but consistent)
-- [ ] Ensure **all request-scoped allocations** come from the request arena (no accidental long-lived allocs)
-- [ ] Standardize error responses:
-  - `__type` formatting
-  - `message` field consistency
-  - status code mapping per error type
+- [x] Ensure **all request-scoped allocations** come from the request arena (no accidental long-lived allocs)
+  - Verified: `handle_connection` in http.odin sets `context.allocator = request_alloc`
+  - Long-lived data (table metadata, locks) explicitly uses `engine.allocator`
+- [x] Standardize error responses:
+  - `__type` formatting — done, uses `com.amazonaws.dynamodb.v20120810#ErrorType`
+  - `message` field consistency — done
+  - Status code mapping per error type — **DONE**: centralized `handle_storage_error` + `make_error_response` now maps InternalServerError→500, everything else→400
+  - Missing X-Amz-Target now returns `SerializationException` (matches real DynamoDB)
 
 ### 2) Storage correctness edge cases
-- [ ] Table metadata durability + validation:
-  - reject duplicate tables
-  - reject invalid key schema (no HASH, multiple HASH, etc.)
-- [ ] Item validation against key schema:
-  - missing PK/SK errors
-  - type mismatch errors (S/N/B)
+- [x] Table metadata durability + validation:
+  - [x] Reject duplicate tables — done in `create_table` (checks existing meta key)
+  - [x] Reject invalid key schema — done in `parse_key_schema` (no HASH, multiple HASH, etc.)
+- [x] Item validation against key schema:
+  - [x] Missing PK/SK errors — done in `key_from_item`
+  - [x] Type mismatch errors (S/N/B) — **DONE**: new `validate_item_key_types` proc checks item key attr types against AttributeDefinitions
 - [ ] Deterministic encoding tests:
-  - key codec round-trip
-  - TLV item encode/decode round-trip (nested maps/lists/sets)
+  - [ ] Key codec round-trip
+  - [ ] TLV item encode/decode round-trip (nested maps/lists/sets)
 
 ### 3) Query/Scan pagination parity
-- [ ] Make pagination behavior match Zig version + AWS CLI expectations:
-  - `Limit`
-  - `ExclusiveStartKey`
-  - `LastEvaluatedKey` generation (and correct key-type reconstruction)
-- [ ] Add “golden” pagination tests:
-  - query w/ sort key ranges
-  - scan limit + resume loop
+- [x] Make pagination behavior match AWS CLI expectations:
+  - [x] `Limit` — done
+  - [x] `ExclusiveStartKey` — done (parsed via JSON object lookup with key schema type reconstruction)
+  - [x] `LastEvaluatedKey` generation — **FIXED**: now saves key of *last returned item* (not next unread item); only emits when more results exist
+- [ ] Add "golden" pagination tests:
+  - [ ] Query w/ sort key ranges
+  - [ ] Scan limit + resume loop
 
 ### 4) Expression parsing reliability
-- [ ] Remove brittle string-scanning for `KeyConditionExpression` extraction:
-  - Parse expression fields via JSON object lookup (handles whitespace/ordering safely)
+- [x] Remove brittle string-scanning for `KeyConditionExpression` extraction:
+  - **DONE**: `parse_key_condition_expression_string` uses JSON object lookup (handles whitespace/ordering safely)
 - [ ] Add validation + better errors for malformed expressions
-- [ ] Expand operator coverage as needed (BETWEEN/begins_with already planned)
+- [x] Expand operator coverage: BETWEEN and begins_with are implemented in parser
+- [x] **Sort key condition filtering in query** — **DONE**: `query()` now accepts optional `Sort_Key_Condition` and applies it (=, <, <=, >, >=, BETWEEN, begins_with)
 
 ---
 
 ## Next (feature parity with Zig + API completeness)
 ### 5) UpdateItem / conditional logic groundwork
+- [x] `UpdateItem` handler registered in router (currently returns clear "not yet supported" error)
 - [ ] Implement `UpdateItem` (initially minimal: SET for scalar attrs)
 - [ ] Add `ConditionExpression` support for Put/Delete/Update (start with simple comparisons)
-- [ ] Define internal “update plan” representation (parsed ops → applied mutations)
+- [ ] Define internal "update plan" representation (parsed ops → applied mutations)
 
 ### 6) Response completeness / options
 - [ ] `ReturnValues` handling where relevant (NONE/ALL_OLD/UPDATED_NEW etc. — even partial support is useful)
@@ -81,8 +86,38 @@ Goal: “aws cli works reliably for CreateTable/ListTables/PutItem/GetItem/Delet
 
 ---
 
+## Bug Fixes Applied This Session
+
+### Pagination (scan + query)
+**Bug**: `last_evaluated_key` was set to the key of the *next unread* item (the item at `count == limit`). When the client resumed with that key as `ExclusiveStartKey`, it would seek-then-skip, **dropping one item** from the result set.
+
+**Fix**: Now tracks the key of the *last successfully returned* item. Only emits `LastEvaluatedKey` when we confirm there are more items beyond the returned set (via `has_more` flag).
+
+### Sort key condition filtering
+**Bug**: `query()` performed a partition-prefix scan but never applied the sort key condition (=, <, BETWEEN, begins_with, etc.) from `KeyConditionExpression`. All items in the partition were returned regardless of sort key predicates.
+
+**Fix**: `query()` now accepts an optional `Sort_Key_Condition` parameter. The handler extracts it from the parsed `Key_Condition` and passes it through. `evaluate_sort_key_condition()` compares the item's SK attribute against the condition using string comparison (matching DynamoDB's lexicographic semantics for S/N/B keys).
+
+### Write locking
+**Bug**: `put_item` and `delete_item` acquired *shared* (read) locks. Multiple concurrent writes to the same table could interleave without mutual exclusion.
+
+**Fix**: Both now acquire *exclusive* (write) locks via `sync.rw_mutex_lock`. Read operations (`get_item`, `scan`, `query`) continue to use shared locks.
+
+### delete_table item cleanup
+**Bug**: `delete_table` only deleted the metadata key, leaving all data items orphaned in RocksDB.
+
+**Fix**: Before deleting metadata, `delete_table` now iterates over all keys with the table's data prefix and deletes them individually.
+
+### Item key type validation
+**New**: `put_item` now validates that the item's key attribute types match the table's `AttributeDefinitions`. E.g., if PK is declared as `S`, putting an item with a numeric PK is rejected with `Invalid_Key`.
+
+### Error response standardization
+**Fix**: Centralized all storage-error-to-HTTP-error mapping in `handle_storage_error`. InternalServerError maps to HTTP 500; all client errors (validation, not-found, etc.) map to HTTP 400. Missing `X-Amz-Target` now returns `SerializationException` to match real DynamoDB behavior.
+
+---
+
 ## Later (big features)
-These align with the “Future Enhancements” list in ARCHITECTURE.md.
+These align with the "Future Enhancements" list in ARCHITECTURE.md.
 
 ### 8) Secondary indexes
 - [ ] Global Secondary Indexes (GSI)
@@ -111,6 +146,7 @@ These align with the “Future Enhancements” list in ARCHITECTURE.md.
 ---
 
 ## Housekeeping
-- [ ] Fix TODO hygiene: keep this file short and “actionable”
+- [x] Fix TODO hygiene: keep this file short and "actionable"
+  - Added "Bug Fixes Applied" section documenting what changed and why
 - [ ] Add a CONTRIBUTING quick checklist (allocator rules, formatting, tests)
-- [ ] Add “known limitations” section in README (unsupported DynamoDB features)
+- [ ] Add "known limitations" section in README (unsupported DynamoDB features)