Skip to content

Purpose and scope of this Document

This document outlines the best practices for utilizing the KaaS (Kafka as a Service) platform, with a specific focus on schemas and metadata. It serves as a reference for users, development teams, and stakeholders to understand the effective use of Kafka within their applications while emphasizing key concepts, schemas, metadata, and design considerations.

Schemas

When designing the event-driven part of your application, it is crucial to consider the importance of schemas. Schemas help maintain data consistency by establishing a standardized structure for the messages being exchanged.

  • Consistency:
    • Defining the structure of your data ensures that all messages follow a consistent format, simplifying processing and analysis.
  • Compatibility:
    • Schemas act as a clear contract that outlines the expected data format, making it easier to maintain compatibility between different versions of your applications.
  • Schema Validation:
    • Well-defined schemas allow for automatic validation of messages, ensuring that only valid data enters your system, thus enhancing reliability.
  • Data Evolution:
    • Schemas support modifications over time while maintaining backward and forward compatibility.
  • Automated Code Generation:
    • Facilitates the generation of serialization/deserialization libraries automatically, allowing rapid development.
  • Performance:
    • Using formats like Avro, Protobuf, or JSON Schema can improve performance due to reduced data sizes and optimized serialization/deserialization routines.

Metadata

Business metadata adds context to events, making it easier for consumers to grasp their intended purpose and relation to the organization. It enhances communication between different applications by including details such as event type, source, and user information.

Effective use of metadata may include:

  1. Identifying relevant metadata and including it in the message schema.
  2. Keeping metadata lightweight by including only necessary information.
  3. Adopting consistent formats for metadata fields.
  4. Separating metadata from the payload to simplify processing and versioning.

Avro

While ksqlDB does not enforce a specific serialization format, utilizing Apache Avro when working with it is highly recommended due to:

  • Schema Evolution:
    • Avro supports modifications while maintaining backward and forward compatibility.
  • Compact Binary Format:
    • It reduces the size of serialized data, aiding in performance.
  • Confluent Schema Registry Integration:
    • Enhances schema management capabilities.
  • Code Generation:
    • Facilitates easy production and consumption of Avro events across various programming languages.

Message design considerations

Large messages

Kafka is designed to handle many small messages rather than a few large ones. Large events can pose significant challenges such as:

  • Increased network bandwidth consumption leading to higher latency.
  • Slower message processing and storage management complexities.

To mitigate these issues, consider breaking down large events into smaller chunks or uploading large binary files to an external system (e.g., S3) and including a reference in the event message.

1
2
3
4
5
6
7
8
{
  "type": "record",
  "name": "XRayScan",
  "location": {
    "bucket": "12346790-something-something",
    "object_key": "123/789/08ed7d8e-430f-4331-b9d2-f4f00a69a20d"
  }
}