Kafka as a Service - Verification Strategy¶
Purpose¶
This document describes the Verification strategy for the release 1.0 of the Kafka as a Service (KaaS) Platform. Further information on the purpose and scope of KaaS and the release can be found in the Agile IT Implementation Plan and Report ref. [1].
The verification will be done according to the Manage IT Infrastructure [2] and Verification of IT Infrastructure [3]
Abbreviations and definitions¶
See Abbreviations and definitions
Verification Strategy considerations¶
As stated in the Agile IT Implementation Plan and Report [1] the following aspects have been considered while developing this strategy:
- The identified life cycle controls from the IT risk assessment
- Requirements created in according to ‘Manage IT Infrastructure’
- A Pilot Application ('Data Product' and API) is designed and tested for functionality, deployment and management as part of the platform functional testing.
- Shift left testing where applicable for any development activities concerning integrations and APIs that are to be tested, designed, scoped, and documented as part of the solution
In addition, focus will be on the KaaS Pilot application and Access Management that are key parts to verify that the platform works as intended with the correct roles and privileges assigned to projects being onboarded.
IT risk assessment¶
As stated in the Agile IT Implementation Plan and Report [1], the IT risk assessment outcome can be summarised into two main areas that must be addressed via implementation and verification activities:
- Data considerations
- Data will be the responsibility of the teams / projects using the platform and therefore agreements must be in place to ensure roles and responsibilities are in place to support this setup
- Confluent cloud is an iPaaS solution and assists in data transitioning. The platform as such does not store or hold the responsibility of storage of data. The onboarding projects using the platform are responsible for DI and GDPR controls
- The risk is mitigated by the F-00920036 KaaS - Project on-boarding agreement - Template v1.0 (section 4.1.1) that must be signed by projects before being onboarded
- Two requirements have been created to support logging requirements (Platform and Audit Logs). These will be verified.
- Security / privilege controls
- RBAC setup/permissions and governance must be in place, to ensure separation of duties between platform activities, environments and teams maintaining their data streams and applications
- Multiple Access Management requirements concerning novoAccess, AzureAD, SSO and Password management (PIM) have been created to mitigate this risk. These requirements will be verified.
IT risk has been considered while developing the Requirements for the platform.
Requirements¶
Requirements have been created and each requirement has been assessed for test verification activities. These assessments are described as acceptance criteria in the requirements. The acceptance criteria acts as the test scoping / coverage and therefore reduce the need for formal test strategies.
RS requirements are implicitly verified by the test done by Unit, Integration and OV testing. Traceability will be established.
KaaS Requirements can be found here.
The platform is made out of different products. The requirements are grouped into these products. For each product a solution design exists.
A product as defined in the RS will be a combination of different components. These components can consist of APIs, configurations, roles, environments, etc. The components are grouped into an ADO page per product and named Solution Design and holds Solution Design Requirements.
Products / Solution Designs¶
- Access Management
- Confluent Cloud
- Pilot Applications
- Application pipeline template
- Team provisioning pipeline
- Azure Pipeline agent
- Logging
A list of Solution Designs can be found in the Solution Designs section. In the list a link can be found to each Solution Design.
Below conceptual drawing depicts the Products and their interrelationship. The drawing is made to visualise and create a better understanding for non KaaS SMEs. The implementation and verification activities and sequence of events required to implement the solution is further described in Table 1 - Product, Test plans and Sequence of Test.
Pilot Application(s) - KaaS and MAP & Shift left testing¶
A Pilot application is created to simulate a customer project being onboarded to the platform. This consist of a KaaS Pilot Application and a MAP KaaS Pilot Application.
The combined Pilot application Product testing is considered for end-to-end testing.
As the MAP application is developed, Code review and Unit test will be performed for the MAP KaaS Pilot Application. Unit test will be performed early and be considered as shift left testing. The application will be integration tested on the QA environment where integration testing verifies the technical backend and OV testing will verify the UI testing.
Verification Approach¶
This document will focus on providing the reader an understanding of: - Verification approach across the platform - Environments - Test organisation & roles and responsibilities - How to understand traceability from Requirements to test
Verification approach across the platform¶
As mentioned above, the platform is made out of different products. This is the verification approach for Products:
- Test plans will be made per product
- There can multiple Test plans per product
- Some Products must have parts of the Product verified as a pre-requisite for other Products. E.g. IV has to be performed as it is a dependency for another Product, but OV can wait as OV has a dependency on yet another Product
- Test Plans can include multiple Products
- At the time of creation of the Verification Strategy no such plans exist, but this document allows for the flexibility to do so, if needed
- Some Products are tested as part of other Products
- E.g. Azure Pipeline Agent is verified indirectly via Product testing of Team Provisioning Pipeline and KaaS Pilot Application
- Rationales for leveraging other Test to Verify a Product must be stated in Test Plans
- A combined Test plan for Pilot Application(s) act as the end-to-end test as discussed in the Agile IT Implementation Plan and Report [1]
- The verification approach along with the test types identified to take place for a given product must be stated in the Test Plans
- Section 'Overall Acceptance Criteria' below must be adhered to in Test plans
- Rationales for deferring from the Overall 'Acceptance Criteria' must be stated in Test Plans
- Products must at least have Installation Verification in QA and Prod environments
- Products that have API code developed must have Code review, Unit Test and Integration Test
- Products that have Terraform code developed must have Code review of pipelines
- Section 'Overall Acceptance Criteria' below must be adhered to in Test plans
- Test plans can reference this Verification Strategy document to avoid duplication of information and sections
- Test plans will list traceability for the Product requirements to test cases, as further described in section 'How to understand traceability from Requirements to test' below
- Test plans will have a sequential order to be run in. This is defined in 'Table 1 Products, Test plans and Sequence of Test' as a number in column 'Test Plan ID + Sequence Number in ( )'
Release management¶
All Products must be verified before the Kafka as a Service Platform is released for use and available for customers / projects to be onboarded to the platform.
The Product Logging will have technical go-live (running in PRD), as it needs to capture logs for pipeline actions. It will be used by the project team only and not released for use with customers / projects.
| Product | Requirements in scope | Test Plan ID + Sequence Number in ( ) |
|---|---|---|
| Access Management | #1318367 #1337204 #1374619 #1374669 #1374692 #1421694 |
#1699873 (2) #1703044 (9) |
| Confluent Cloud | #1422279 #1422561 #1425495 |
#1827361 (3) #1704657 (10) |
| Pilot application (KaaS Pilot application) | #1422220 | #1701743 (7) |
| Pilot application (MAP KaaS Pilot application) | #1422189 | #1701743 (7) |
| Application Pipeline Template | #1422289 | #1701723 (6) |
| Team Provisioning Pipeline | #1445237 | #1701597 (5) |
| Azure Pipeline Agent | #1533706 | #1701569 (4) |
| Logging | #1417078 #1520146 |
#1701171 (1) #1703080 (8) |
Table 1 - Product, Test plans and Sequence of Test
Environments¶
| Environment name | Purpose of env. | Who | Activities | Strategy (activities) |
|---|---|---|---|---|
| Development | Development and/or configuration and setup of system functionality and/or API’s | Project team API developers Confluent | Development | Development and/or configuration and setup according to the guidelines |
| Test | Test environment for system functionality and/or developed APIs | Project team API Developers | Test | Test of development and/or functionality according to guidelines. Verification documented in ADO via Product Test plans |
| QA | Quality assurance environment similar to the production environment | Project team System management | Formal testing | Full test and verification documented in ADO via Product Test plans |
| Production | Production environment, which is under configuration and change control. | System management | Operation Deployment of new API’s and system functionality | Operation and Maintenance |
Table 2 - Environments
Test organisation & roles and responsibilities¶
The following roles are identified to support verification activities.
| Roles | Responsibilities | Initials |
|---|---|---|
| PO | Review and approve Test plans | RSNI |
| Lead Architect | Review and approve Test cases Review and approve Test defects |
AVNO |
| QC | Review and approve Test runs Review and approve Test defects Create and perform: Test execution review, conclusion and approval Test case Approve entering next Test phase |
JEQ PKUE |
| Test Designer SME | Create Test plans Create Test cases |
MVQS KZNT ZQEM |
| Tester SME | Execute Test cases Create Test defects |
MVQS KZNT ZQEM JEQ PKUE |
| IT QA | Supervision only - do not have any part in the verification activities | N/A |
Table 3 - Test organisation & roles and responsibilities
Roles and responsibilities are described in the Wiki page KaaS Artefacts, Reviews and Approvals.
Training of Testers¶
Training for using DevOps testing module has been done as peer training from Test Manager to testers.
In addition testers must be trained according to Kafka as a Service Azure DevOps Testing and Kafka as a Service Test Defect Management.
Training must be documented as per instructions on the WIKI page Kafka as a Service Azure DevOps Testing.
Agile IT Implementation Plan and Report [1] section 7.14 states that testers must be trained in Q0300381 - Validation Deviations. This is replaced by the local instruction Kafka as a Service Test Defect Management.
How to understand traceability from Requirements to test¶
Traceability will be established:
- between RS and Solution Designs requirements.
- between specifications requirements to relevant verification test cases and test results.
Requirements and specifications will be traceable via relations in Azure DevOps in the Solution Designs section of the Wiki.
Test case traceability will be created by tracing test cases and test runs to requirements in an Excel overview to support Test Plan traceability. Traceability will be documented, attached to Test plan, and reviewed as part of the Test plans (before test execution) and in the 'Test execution review, conclusion and approval Test case' (after test execution) to list successful runs.
Test execution¶
Pre-requisites¶
Before testing is started the following pre-requisites must be in place: - Test case is approved - Test plan is approved - Test plan must show traceability from RS -> SD -> Test case - RS, SD, FS, DS, Operation Guide must be in approved stated via a Pull Request - Code review is approved via Pull Request, in accordance with the Code review guideline - CIL is approved via Pull Request
Defect Management¶
Defect management follows the standard Kafka Test Defect Management process defined in the Kafka WIKI: Defect Management
Overall Acceptance Criteria¶
Product testing must be completed in an environment before the Product testing can be initiated in the next environment.
Deployment of builds can be done through Dev, Test, QA and Prod environments at the same time, as long as it is verified that the build being released to Prod is verified in QA environment first and released for Prod.
- All development is subject to code review, unit test and integration test
- Unit test must be completed in Test environment
- Integration test can be completed in Test environment
- Integration test can be completed in QA environment
- Performance test can be completed in QA environment
- Unit and Integration test must be reported in ADO, via Test case and run, before IV is started
- IV must be completed in QA environment and reported before build is deployed to Prod
- Any Integration and/or Performance test are be completed in QA environment and reported before build is deployed to Prod environment
- OV must be completed in QA environment and reported before build is deployed to Prod environment
- p-IV must confirm the same deployment in Prod environment as for QA environment
- p-IV must be completed in Prod environment and reported before Infrastructure Release
A test phase (from one environment to the next) is completed when:
- All test cases have been executed and the results have been evaluated against relevant Acceptance Criteria and/or Expected Results
- All Test Defects have been resolved and approved by relevant roles
- The results have been reported in the 'Test execution review, conclusion and approval Test case' and acknowledged by relevant roles
References¶
- F-00883817 KaaS Implementation Plan
- Q216303 Manage IT Infrastructure
- Q0779370 Verification of IT Infrastructure
- Q187655 Manage IT Security
- Q204010 Manage Data Integrity in IT Systems
- Q187218 Ownership of IT solutions
- F-00869898 KaaS IT Risk Assessment
- F-00925673 KaaS Design review Report