fix: removed most of schema encode/decode overhead

Closed
HA
12d ago
+1410-3259

Summary by cubic

Switched event transport from Avro/Schema Registry to FastStream’s native Pydantic JSON with 1 topic per event class, removed the EventStore/KafkaEventService, and replaced event_type with topic across API and DB. UnifiedProducer is now EventPublisher that persists idempotently; DLQ processes typed events and metadata via headers. CI pre-pulls the Python runtime image for reliable startup.

  • Performance

    • Dropped Schema Registry and custom decoders; handlers use typed Pydantic models directly, cutting serialization overhead.
    • Removed EventStore and eager schema init; EventPublisher persists before publish; Kubernetes worker no longer waits on the image pre-puller; CI pre-pulls python:3.11-slim.
    • DLQ now uses BaseEvent payloads with header metadata, simplifying parsing and retries.

  • Migration

    • Replace DomainEvent/DomainEventAdapter with BaseEvent; remove SchemaRegistryManager and related APIs/tests.
    • Rename UnifiedProducer to EventPublisher; delete KafkaEventService.
    • Replace event_type with topic everywhere: EventDocument.event_type -> topic, API params event_types -> topics, SSE uses topic strings.
    • Topics are derived via BaseEvent.topic(); remove mapping/registry helpers and EventType enum usage.
    • Add IdempotencyMiddleware for FastStream consumers.

Summary by CodeRabbit

  • Refactor

    • Events moved to topic-based models and Avro/schema-registry paths removed; publishing simplified to a single publisher flow.

  • New Features

    • Events persisted idempotently before publish; DLQ/retry emits richer headers and trace info.
    • Idempotency middleware added to messaging pipeline.
    • Kubernetes worker pre-pulls runtime images; CI pre-pulls test runtime image.

  • Bug Fixes

    • Invalid events are skipped and counted; duplicate stores are no-ops.

  • Tests

    • Tests updated to topic/BaseEvent model; legacy schema-registry and event-store e2e tests removed.

  • Chores

    • Schema registry auth setting removed; tuned test K8s resource/timeouts.