Overview of AMQ Streams
AMQ Streams is RedHat’s fully supported Apache Kafka offering on top of Kubernetes. At Irori we like working with AMQ Streams for several reasons.
Ease of maintenance
AMQ Streams is based on the Operator framework in Kubernetes. By supplying a declarative configuration of the wanted cluster setup, the operator takes care of executing the necessary steps to achieve that goal.The Cluster operator can also help with tasks such as TLS certificate renewal (for self managed certificates) and rolling upgrades of the cluster.
Team self service
Besides the Cluster Operator there is also the Topic- and User-Operators which handle declarative configurations of topics, users and permissions. We prefer hooking this up to a GitOps tool such as ArgoCD. This enables teams to manage their own topic configurations in Git. Depending on organization policies the access permissions to the same topics can be managed by the team letting other teams open pull requests to gain access.
Co-locating streaming applications
In order to do data pipelines in Kafka, the Kafka Streams framework is a very powerful tool to use. Since Kafka Streams applications usually both consume and produce data from and to the Kafka cluster it is beneficial to run them close to that cluster to avoid high latency and unnecessary egress traffic. Since AMQ Streams is running on a Kubernetes cluster a good way to achieve this is to provide means to also run Kafka Streams based workloads in the same cluster.
Author: Daniel Oldgren – Solution Architect
