| Title | Kafka as a Service - Requirement Specification |
|---|---|
| Version | 5.0 |
| Approval Pull Request (PR) | PR251979 |
Requirement Specification¶
Requirement Specification structure - Wiki page and Work Items¶
The Requirement Specification will be a combination of an ADO Wiki page and separate ADO work items, one per requirement.
The Wiki page will hold general information and introduction to KaaS. The Wiki page will be subject to acknowledgement by Product Manager by approving the ADO branch of the Wiki page that holds the approved requirements.
Individual requirements will be documented in ADO as individual Work Items with the title RS (Requirement Specification) and acknowledged by Product Owner. The ADO Work Item type (metadata) will be URS and therefore links with ADO ID will include the name URS.
Each requirement will be assessed for potential test verification activities. These assessments will be described as acceptance criteria in the requirements. The acceptance criteria will act as the test scoping / coverage and therefore reduce the need for formal test strategies. The acceptance criteria will remain in the requirements for the duration of the lifetime of the requirement.
All requirements are interlinked and will not be grouped into functional nor non-functional requirements.
Purpose¶
This Requirement Specification (RS) describes the requirements for the Kafka as a Service (KaaS) Platform.
Scope¶
Scope of Services provided by the KaaS platform¶
The Platform Kafka as a Service will provide these services:
- Platform pipeline
- Connectivity to and from Confluent Cloud from NN CORP
- Team provisioning pipeline
- Access Management
- Application pipeline
- Pipelines, automating software build and release
- Stream Governance pipeline
- Pipeline for linking business metadata and Tags
- Processes and organisation of the platform
- Connectors
- Pre-built, managed Apache Kafka connectors, that make it easier to connect to data source and targets
- Flink
- utilizing its powerful event processing features and rich ecosystem to handle real-time data processing and analytics effectively
- Client Quota for Service Accounts
Scope of KaaS platform, from a technical point of view¶
The KaaS platform will have these components in scope
- Confluent Cloud, which hosts Kafka clusters (Topic storage and management) and a Control Center of the platform (access, monitoring etc.)
- Confluent Kafka services, such as Kafka Connect (move data in/out of Kafka)
Out of Scope for the Requirement Specification¶
- On-premise (in Corp) Kafka cluster
- If and when Novo Nordisk decides to add this to the scope, the implementation plan will require an update.
- LoB Confluent applications
- Once the platform is qualified and released for use, teams (customers/LoB) will be responsible for implementing their own Confluent applications. Teams may be like IDEA Infra and CDR within the Clinical Development area.
- LoB Confluent applications are not considered to be part of the integration platform.
- As per the IT Risk Assessment, the platform supports Events (data) that may fall in the categories of being GxP, GDPR and sensitive data and the Events (Data) will be the responsibility of the teams / projects using the platform. The platform in itself does not store any data. Onboarding Agreements between KaaS and teams/projects/customers will describe the split of responsibilities and that platform team can be contacted in case of need of insight into platform audit log.
- Events retention time is set per topic (configuration of topics is governed by Teams and not the Platform organisation)
Abbreviations and definitions¶
See Abbreviations and definitions
Description of Kafka as a Service (KaaS)¶
The purpose of the Kafka as a Service (KaaS) Platform is to create a foundational IT Infrastructure platform that will enable LoB teams across Novo Nordisk to achieve event-driven data integration between IT systems, supporting near-real time data exchange and data-driven decision making. Further, to prevent the creation of “narrow sighted” solutions (project-by-project), which prevents Novo Nordisk in executing our digital strategy and business ambitions of (re)using data in new and efficient ways, the platform supports exposing existing events for re-use and consumptions of new-use cases.
KaaS combines messaging, storage, and stream processing to allow storage and analysis of both historical and real-time data. KaaS can connect to data sources and sinks (targets) (e.g. RDBMS, Cloud object stores, Message queues, document stores, cloud data warehouses etc.)
The general purpose of the platform is to realize the following points:
- Greater data sharing & insights
- Ease of use & access to integration services
- Standardisation of platforms
- Supported development teams
- Support of new event-driven (publish/subscribe) integration patterns
The combination of these elements will enable a faster time to market approach combined with instant product release.
The initial release of the KaaS Platform will be implemented in Novo Nordisk as depicted and described below, and in accordance with the KaaS Implementation Plan [6].
High level diagram - KaaS¶
- An Event Producer publishes an event (data record) to the Confluent Kafka Event broker
- The event can be validated in the Schema registry, to ensure that it follows the right data structure/format
- Schema registry is part of 'Stream Catalog' - a self-service Discovery Overview, that allows teams to classify, organize and navigate in schemas (through meta data - not actual data records)
- An Event Consumer reads events from the Confluent Kafka Event broker
- A 'KaaS Management' MAP application forwards Platform Audit Logs and Metrics to Splunk
- The solution has the following supporting pipelines;
- Infrastructure as code (platform-pipeline) for provisioning platform resources and common components
- Team Provisioning Pipeline for onboarding and setting up access and resources for new groups of users (teams)
- Application Pipeline Template (Team Product template) for enabling teams to develop and deploy their data products (applications) on the platform
- Stream Governance Pipeline for linking business metadata and Tags by teams for their data products
- Security is handled with SSO and OpenID Connect with Azure Active Directory
- A Pilot application will be used to verify the automation and processes defined on the platform;
- (MuleSoft) MAP KaaS Pilot application to act as producer and consumer of events
- KaaS Pilot application utilizing the 'Application Pipeline Template' to create topics and verify Schema Registry in Confluent
- Confluent cloud is connected to NN Corp network via AWS Transit Gateway for establishing the high-level connectivity. Transit Gateway connectivity is detailed through below diagram.
Description of the KaaS IT Infrastructure landscape¶
The overall description of the IT Infrastructure landscape is provided in below table:
| IT Service component name | Description and purpose |
|---|---|
| Apache Kafka Cluster (one or more brokers – Kafka servers) | The Kafka cluster consist of brokers which serves as storage servers, where topics are replicated across brokers for redundancy, as well as producers (writing Topic messages) and consumers (reading Topic messages). In this first implementation the Kafka cluster will be hosted and operated by Confluent in a controlled cloud environment and managed by NN. The solution is part of Confluent Cloud. |
| Schema Registry | Is a server process that stores and maintains a database of the topic schemas – the predefined message format “rules”, to ensure data is processed in a compatible way between producers and consumers. |
| Confluent Control Center | The enterprise administrative tool suite. This includes resource provisioning and management, monitoring of health and performance, access tracking via audit logs. |
| Mulesoft Anypoint Applications | KaaS Management application responsible for forwarding logs to Splunk |
| Confluent Metrics and Logs | The alerts and Dashboards representing Confluent metrics and logs are displayed in Bifrost |
| CI/CD Development flow for Infrastructure as Code | CI/CD as part of infrastructure as code (infrastructure will be set as templates, by example Terraform, instead of configuring the platform manually) |
| CI/CD Development flow for application pipelines | CI/CD development flow for application. Pipelines will setup for projects |
| Confluent application development | Confluent will be used as supporting tools for operating the platform |
| Pilot Application | Pilot application will be created based on MAP and Confluent application for testing application pipeline and connectivity in MAP to and from Kafka and schema registry |
| Connectors | Pre-built, managed Apache Kafka connectors, that make it easier to connect to data source and targets |
| Flink | Utilizing its powerful event processing features and rich ecosystem to handle real-time data processing and analytics effectively |
Assessments¶
KaaS is categorized as IT Infrastructure in accordance with Manage IT infrastructure [1] and the KaaS IT Risk Assessment [5]. Risk assessment has been carried out and provided input to the requirements.
Business processes¶
KaaS supports the Novo Nordisk IT Area and the business process S3 IT.
Requirement Classification¶
The requirements will be created in ADO and will be classified according to ADO meta data to support grouping of requirements.
Requirements¶
| ID | Work Item Type | URS (Status Dynamically Updated) |
|---|---|---|
| 1318367 | URS | #1318367 |
| 1337204 | URS | #1337204 |
| 1374619 | URS | #1374619 |
| 1374669 | URS | #1374669 |
| 1374692 | URS | #1374692 |
| 1417078 | URS | #1417078 |
| 1421694 | URS | #1421694 |
| 1422189 | URS | #1422189 |
| 1422220 | URS | #1422220 |
| 1422279 | URS | #1422279 |
| 1933853 | URS | #1933853 |
| 1422561 | URS | #1422561 |
| 1425495 | URS | #1425495 |
| 1445237 | URS | #1445237 |
| 1520146 | URS | #1520146 |
| 1533706 | URS | #1533706 |
| 1932759 | URS | #1932759 |
| 3144028 | URS | #3144028 |
| 3185126 | URS | #3185126 |
| 3351539 | URS | #3351539 |
| ## Traceability between Requirements | ||
| ### Traceability (Static list) between Requirements (RS) and (SDR) | ||
| The requirements are grouped into products. For each product a solution design exists. Traceability can be found here in Solution Designs |
References¶
- Q216303 Manage IT Infrastructure
- Q187655 Manage Information Security in IT Solutions
- Q204010 Manage data integrity in IT solutions
- Q187218 Ownership of IT systems and IT infrastructure
- ITR0014383 IT Risk Assessment - ITRA | Novo Nordisk ServiceNow
- F-00883817 KaaS Implementation Plan
Change Log¶
| Version no. | CHG no. | Approved Pull Request | Date | Change Description |
|---|---|---|---|---|
| 1.0 | CHG0100614 | PR92280 | Date of acknowledgement | New document |
| 2.0 | CHG0122187 | PR129700 | Refer PR | Stream Governance Pipeline details added to RS wiki |
| 3.0 | CHG0184608 | PR228852 | Refer PR | Added requirement for Connector enablement in UI |
| 4.0 | CHG0201019 | PR241529 | Refer PR | Added requirement for Flink enablement |
| 5.0 | CHG0207193 | PR251979 | Refer PR | Added requirement for Client Quota |