Title	Kafka as a Service Solution Design - Logging
Version	2.0
Approval Pull Request (PR)	PR119510

Solution Design - Logging¶

Purpose & Scope¶

This Solution Design covers Logging. See the Solution Designs section for an overview of the entire KaaS Solution.

Abbreviations and definitions¶

See Abbreviations and definitions

Solution Design Requirements¶

ID	Name
1	Audit Log
2	Splunk Logging Setup

Platform Audit Log¶

Changes made by users and services accounts within Confluent Cloud organisations are logged through an Audit Log service. Each auditable event record includes information about who tried to do what, when they tried, and whether or not the system gave permission to proceed

Confluent Cloud audit logs are retained for seven days on an independent Kafka cluster. These audit log records cannot be modified, deleted, or produced directly to the audit log topic. In order to retain the records longer, logs will be forwarded and archived in Splunk, through the use of a MAP application.

Note: Default Confluent Audit Logs are stored in US, but no sensitive data is logged. More information about the default setup can be found on Confluent documentation here

Splunk Logging Setup¶

Data is moved from the default audit-topic cluster with the kaas-audit-log-forwarder MAP application. Detailed design and documentation is stored in the MAP ADO project 'MAP-CS (Cross-services)' here: GIT: kaas-audit-log-forwarder

Solution overview¶

The solution is based on the concept of batch messages consumed with Kafka connector as a messages poller. All the consumed batches are processed in loop and transferred to Splunk one by one to ensure that messages are processed in correct order.

Batch of consumed messages is committed whenever all the messages are properly transferred to Splunk index.

KaaS Log Forwarder

Splunk Access¶

Splunk is delivered as a service by GITO (C&CI) team. Logs are retained for 2 years across all indexes.

Three indexes are created, where all Platform roles - KaaS Administrator, KaaS Operator and KaaS Supporter - have access to the indexes(s);

KaaS_audit_index
KaaS_prd_index
KaaS_qa_index
KaaS_tst_index
KaaS_dev_index

Data Audit Log¶

Events (Data) on the platform can only be appended to a topic (log). It is not possible to change or delete individual data records. Changes to Topic retention is captured in the audit log.

You can remove data by changing the retention period of the topic. Data will automatically be removed when the retention period expires.

Traceability between Requirements¶

Traceability between Requirements (URS) and (SDR)¶

ID	Work Item Type	URS (Status Dynamically Updated)	GIT SDR
1417078	URS	#1417078	Solution Design Requirement: Platform Audit Log
1520146	URS	#1520146	Solution Design Requirement: Data Audit Log

Change Log¶

Version no.	CHG no.	Approved Pull Request	Date	Change Description
1.0	CHG0100614	PR102957	Date of acknowledgement	New document
2.0	N/A	PR119510	See PR	Updated Splunk retention to 2 years, after dialog with Architects, QA, IT Security and Data Privacy