feat: migrate python sdk from grpc to fibp binary protocol

Closed
VI
2mo ago
+3834-2362

Summary

  • Replace entire gRPC transport layer with native FIBP (Fila Binary Protocol) implementation
  • Add fila/fibp/ module with primitives (Reader/Writer), opcodes (42 opcodes, 18 error codes), and codec (encode/decode for all opcodes)
  • Add fila/conn.py with synchronous Connection and async AsyncConnection classes handling TCP framing, handshake, keepalive (Ping/Pong), and continuation frames
  • Rewrite client.py and async_client.py to use FIBP connections with leader-hint reconnection
  • Rewrite batcher.py (Auto/Linger accumulators) to use Connection instead of gRPC stubs
  • Add admin methods (create/delete queue, stats, config, redrive) and auth methods (API key CRUD, ACL management) to both sync and async clients
  • Add new error types mapped from FIBP error codes: UnauthorizedError, ForbiddenError, NotLeaderError, ChannelFullError, etc.
  • Add ConsumeMessage fields: weight, throttle_keys, enqueued_at, leased_at
  • Delete fila/v1/ (generated protobuf), proto/ directory
  • Remove grpcio and protobuf from dependencies (zero external deps now)
  • Bump version to 0.3.0
  • 49 unit tests passing (34 new codec/primitives tests + rewritten batcher/type tests)

Test plan

  • python -m pytest tests/test_fibp.py -- 34 codec/primitives unit tests
  • python -m pytest tests/test_batcher.py -- 12 batcher unit tests with mock connections
  • python -m pytest tests/test_enqueue_integration.py::TestAccumulatorModeTypes -- 4 type unit tests
  • ruff check fila/ tests/ -- all checks pass
  • Integration tests (test_client.py, test_enqueue_integration.py) require a running fila-server with FIBP support

Summary by cubic

Migrated the Python SDK from gRPC to the FIBP (Fila Binary Protocol) over TCP, aligning codecs and result types with the wire spec. Hardened frame parsing and consume behavior per review feedback.

  • New Features

    • Native FIBP codec and sync/async connections with handshake, keepalive, continuation frames.
    • Clients speak FIBP with leader‑hint reconnect; consume handles Delivery‑before‑ConsumeOk via a pushback buffer.
    • enqueue_many returns per‑message results; background accumulators (AccumulatorMode.AUTO/Linger) run on the new connection.
    • Admin/auth: queue CRUD, stats/config/redrive, API key CRUD, ACL management (sync and async) with spec‑aligned codecs and typed results (StatsResult, QueueInfo, AclEntry, ApiKeyInfo).
    • All result frames decode error_code; errors map to typed exceptions (UnauthorizedError, ForbiddenError, NotLeaderError, ChannelFullError, etc.).
    • ConsumeMessage adds weight, throttle_keys, enqueued_at, leased_at.
    • Delivery decode diagnostics added; mypy dict types tightened.
    • Correct hot‑path opcode assignments: Enqueue(0x10), EnqueueResult(0x11), Consume(0x12), ConsumeOk(0x13), Delivery(0x14), CancelConsume(0x15), Ack(0x16), AckResult(0x17), Nack(0x18), NackResult(0x19).
    • Hardening: bounds checks in string/bytes readers to catch truncated frames; batcher now fails unresolved futures if server returns fewer results; removed wrapper that swallowed decode errors in consume; subscribe pushback only accepts Delivery and raises on unexpected opcodes; broadened auth rejection test to accept any FilaError.
    • Version bumped to 0.3.0.

  • Migration

    • Requires a FIBP‑enabled fila‑server on port 5555.
    • Remove grpcio/protobuf and any fila.v1.* or proto/* imports; the SDK has no external deps now.
    • API key is sent during the FIBP handshake (not via gRPC metadata).
    • If you rely on batching, use AccumulatorMode or Linger; defaults remain sensible.
    • For per‑message enqueue errors, read them from enqueue_many results; single enqueue preserves exception mapping via EnqueueError and typed errors.