Skip to content

Purpose and scope of this Document

This document provides an integrated view of the KaaS platform, consolidating its functionalities, components, and architecture. It illustrates how different components work together to ensure efficient data management, observability, security, and compliance throughout the platform lifecycle.

KaaS Platform Objectives

  • The KaaS platform is designed as a cohesive architecture to address modern data streaming challenges. By integrating user access controls, automated pipelines, environment segmentation, and compliance mechanisms, it ensures secure, scalable, and efficient data management.
  • KaaS can connect to various data sources and sinks and can support almost all Kafka Clients like Python, NodeJs.
  • Key objectives include:
    • Support for streaming data IN and OUT of Kafka clusters with ease, and also facilitate data storage with retention policies.
    • Greater data sharing & insights to foster collaboration and informed decision-making.
    • Ensure robust access control, enhancing security, data protection and preventing unauthorized access.
    • Utilizes CI/CD methodologies for infrastructure and application deployment, optimizing setup processes for better efficiency.
    • Support of development teams with resources and assistance for quicker project execution.

KaaS Technical Design

alt text

KaaS Access Management

alt text

Detailed Description of the KaaS IT Infrastructure Landscape

IT Service Component Name Description and Purpose
Kafka Cluster The Kafka cluster consists of brokers that serve as storage servers. Topics are replicated across these brokers for redundancy, with producers writing messages and consumers reading them. It is managed in a controlled cloud environment.
Schema Registry Maintains a database of predefined message format rules (schemas) to ensure consistent data processing between producers and consumers.
Management Plane Oversees administrative tasks, while the Data Plane handles data flow, ensuring operational separation for efficiency.
Network Connectivity The Confluent cloud is connected to the NN Corp network via AWS Transit Gateway to establish high-level connectivity. Transit Gateway connectivity is detailed in the diagram below.
CI/CD for Infrastructure as Code Automates platform configuration using templates, replacing manual processes to improve efficiency and reduce errors.
CI/CD for Application Pipelines Streamlines application deployment workflows, ensuring rapid and reliable project setup.
Access Management User roles and access are provisioned using Active Directory (AD) groups in NovoAccess. Service accounts are created for projects with necessary RBAC to access Kafka resources.
Monitoring Mulesoft Anypoint Applications forward audit logs to Splunk , Dashboards and alerts are configured in Grafana for confluent metrics and audit logs.

KaaS Environment and Cluster Overview

alt text

Note

To quickly get started with data/event streaming for proofs of concept (POCs) or preliminary explorations, use Playground Environment in the Prod Org, which is provisioned using DEV clusters (dev_eu1_cluster_0 and dev_eu1_public_default) in DEV environment.