| Title | Kafka as a Service Solution Design - Azure Pipeline Agent |
|---|---|
| Version | 4.0 |
| Approval Pull Request (PR) | PR157981 |
Solution Design - Azure Pipeline Agent¶
Purpose & Scope¶
This Solution Design covers Azure Pipeline agent, used to ensure connectivity from ADO pipelines to the private Confluent Cloud network. See the Solution Designs section for an overview of the entire KaaS Solution.
Abbreviations and definitions¶
See Abbreviations and definitions
Solution Design Requirements¶
| ID | Name |
|---|---|
| 1 | Azure Pipeline Agent |
| 2 | Self-hosted Rationale |
Azure Pipeline Agent¶
Our Azure Pipeline Agent solution, deployed in the novonordiskit ADO organization, provides a dynamic, scalable, and secure build environment for managing tasks associated with Confluent Managed Kafka clusters residing in a closed AWS network. The solution combines persistent warm pool baseline agents for immediate availability with on-demand agents for specialized workloads. Agents are deployed as Docker containers on AWS ECS Fargate with self-terminating lifecycle management. More detailed information can be found here: GIT: pipeline-agent
Warm Pool Baseline Agents: The solution maintains a pool of 5 pre-configured agents during India and Europe business hours (02:30-18:00 CET, Mon-Fri), scaling to 1 agent outside working hours. This eliminates the 2-3 minute container startup overhead, enabling pipelines to start immediately. Step-based autoscaling responds within 60 seconds when additional capacity is needed (up to 20 agents maximum), triggered by CloudWatch metrics monitoring agent availability in real-time. Important! Please note that this is initial setup and we might adjust scaling numbers in the future!
On-Demand Agents: For specialized workloads requiring custom Docker images or specific configurations, the solution continues to support ephemeral on-demand agents that spin up for individual jobs and self-terminate upon completion.
Network setup¶
Addressing limited access to Kafka clusters, the agent allows secure VPC peering or Transit Gateway connections, enabling on-demand internal network access without compromising security.
Docker containers¶
The Azure Agent is hosted in a Docker container on AWS ECS Fargate, with a custom entrypoint script for precise control, including cleanup methods and timeouts for self-termination.
Scalability & Resource Efficiency¶
The solution supports two scaling approaches:
Warm Pool Autoscaling: Persistent baseline agents scale horizontally based on workload (CloudWatch metrics track agent availability, triggering scale-out within 60 seconds when capacity drops below target). This provides immediate job execution while maintaining cost efficiency through scheduled scaling (reduced capacity outside business hours).
On-Demand Provisioning: For specialized workloads, agents can still be provisioned on-demand, optimizing resource usage by creating containers only when needed and self-terminating after job completion.
By utilizing Azure DevOps, Docker, AWS ECS, CloudWatch, and Terraform, this solution ensures high-performance, adaptable, and secure agents that address unique challenges in network access, scalability, and resource optimization.
In order to allow Agent Pool access from Team/project ADO projects, security settings needs to be setup initially by the Central team and a ADO Project Admin. See more details here: ADO configuration guide
Self-Hosted Rationale¶
The main rationale for using self-hosted agents is to enable access to private networks through transit gateway, by hosting agents on our private AWS network that is peered to the Confluent managed AWS VPC with transit gateway. This allows secure communication with and configuration of Kafka clusters without publicly exposing them on the internet.
The warm pool approach further optimizes this by maintaining pre-configured agents ready for immediate use, eliminating the 2-3 minute startup delay of on-demand container provisioning. This significantly reduces pipeline execution time (40-50% improvement) while maintaining the benefits of containerized environments (consistent build environment, configuration drift elimination).
For specialized workloads requiring custom dependencies, the solution continues to support ephemeral on-demand agents, providing flexibility without the operational burden of maintaining static VMs. The combination of warm pool baseline agents and on-demand agents delivers both immediate availability and cost-effective scalability.
Traceability between Requirements¶
Traceability between Requirements (URS) and (SDR)¶
| ID | Work Item Type | URS (Status Dynamically Updated) | GIT SDR |
|---|---|---|---|
| 1533706 | URS | #1533706 | Solution Design Requirement: Azure Pipeline Agent |
Change Log¶
| Version no. | CHG no. | Approved Pull Request | Date | Change Description |
|---|---|---|---|---|
| 1.0 | CHG0100614 | PR102956 | Date of acknowledgement | New document |
| 2.0 | CHG0100614 | PR111467 | See PR | Updated broken links |
| 3.0 | N/A | PR119510 | See PR | Updated broken links as part of wiki restructure |
| 4.0 | CHG0126415 | PR157981 | See PR | Updated link to new repo |
| 5.0 | CHG0214068 | PR259730 | See PR | Updated link to new repo |