How to choose the right replication topology and understand the moving parts
Apache Kafka has become the backbone of real time data infrastructure across most industries. As deployments scale and become business critical, one question shows up sooner or later: how do we reliably move data between Kafka clusters?
At Irori client sites, we often run Kafka on Kubernetes using Strimzi, the Kubernetes native operator for deploying, managing, and configuring Kafka clusters. Strimzi is also the upstream foundation for vendor supported Kafka on Kubernetes offerings from Red Hat and IBM, which makes it a pragmatic choice in many enterprise environments. MirrorMaker 2 is supported in Strimzi as a first class resource, which lets us define and operate replication topologies alongside our Kafka clusters, using the same GitOps and platform patterns as everything else.
In this blog series we walk through what MirrorMaker 2 is, how we use it for disaster recovery and cluster migration, and the tradeoffs that matter when you design replication across clusters. Part 1 covers the building blocks and how to choose a topology. Part 2 goes deep on failover and recovery mechanics and the pitfalls we see in production.
What is MirrorMaker 2?
In simple terms, MirrorMaker 2 is Kafka’s built-in cross-cluster replication tool. At its core, it consumes messages from topics in a source cluster, and then produces them into corresponding topics in a target cluster in real-time. However, it’s much more than a simple pipe moving messages from A to B, MM2 is a sophisticated distributed application built on the Kafka Connect framework, and by leveraging this eco system it provides more than simply data copying. Some features include:
- Intelligent Offset Mapping: MM2 doesn’t only copy bytes, it understands consumer progress by tracking where your applications are in the stream and translates the positions so they make sense in the new cluster
- Native High Availability: Since it runs as a Kafka Connect Cluster, it is inherently distributed. This means that if one MM2 node fails, the workload is automatically distributed across the remaining nodes
- Automatic Topology Management: It can automatically detect new topics on the source cluster and create them on the target cluster with correct configurations, and also sync Access Control Lists (ACLs)
- Observability: MM2 uses built-in heartbeats and checkpoints that allow for granular monitoring of replication latency, making it easy to track how far behind your secondary cluster might be
Key components
Under the hood, MirrorMaker 2 is not a single process, but a collection of specialized Kafka Connect workers. When you deploy a KafkaMirrorMaker2 resource in Strimzi, the operator orchestrates these workers to run as a distributed cluster. This architecture allows MM2 to parallelize the replication of hundreds of topics across multiple pods.
|
Components |
Description |
|
MirrorSourceConnector |
Mainly handles data replication. It discovers topics and partitions in the source cluster and then creates or updates them in the target cluster. It also syncs topic configuration and ACLs |
|
MirrorCheckpointConnector |
This connector is responsible for syncing consumer group offsets. It keeps track of consumer groups, creates internal topics to store checkpoint information, and polls consumer group offsets from the source cluster and translates them to the target cluster. |
|
MirrorHeartbeatConnector |
Monitors connectivity by periodically emitting health/heartbeat signals between the clusters |
Selecting a mirroring setup
Every organisation that wants to set up Kafka mirroring must first decide: why are we doing this? What are the scenarios we want to support or protect against, and how much time, energy and money are we willing to spend?
Typically the use cases revolve around disaster recovery, availability or migration.
Two common aspects to consider for disaster recovery are Recovery Time Objective (RTO) and Recovery Point Objective (RPO). In short:
- RTO – how fast do we want to be up and running again in a failure mode?
- RPO – how much data-loss can be tolerated
Instinctively, of course, you want to ensure full up time and no data-loss, but when you start pulling the threads in the analysis, we often find that some data loss can be tolerated or mitigated in other ways, if we instead ensure short RTO – having the live data flows come online quickly after a disaster.
Another way to look at it is what kind of scenarios we want to protect against. Mirroring might only be one tool in your tool belt, but some scenarios to keep in mind are:
- Geographic site goes down (power, misconfiguration, disaster)
- Bug or misconfiguration during upgrade of a cluster
- IT breach
- Human error
- Hardware failure
We will discuss some possible setups below, and in almost all, there are two clusters involved. The analysis of the scenarios above can drive you to establish some more properties that differ for the clusters you set up mirroring between:
- Geographical separation
- Different failure domains (power supply, network connection, hardware)
- Different access restrictions (e.g. strictly locked down second site)
- Different upgrade/deployment mechanisms and routines (don’t auto patch both clusters simultaneously)
Replication topologies
In almost all cases, you are designing around two clusters. The topology determines where producers and consumers connect during normal operation, and what happens when the primary site is unhealthy.
Active-active topology
In this topology clusters mirror topics simultaneously to each other, each source cluster typically creating a copy of its topics with a prefix in the target cluster. For example:
This setup is simple from the point of view of Kafka and Mirror Maker 2, in that it is always constant, and you know clearly which site is the master for each topic. But in essence, you have two completely different topics. There are no ordering guarantees between messages in one compared to the other, even if the same keys and partition counts are used. This pushes more complexity to the clients: they have to be aware of both topics, and be able to resolve ordering issues between them, and also possibly fail-over between two different Kafka producers/consumers in the client code, depending on site availability.
But we have several positive aspects:
- Near zero RTO and RPO: If one site goes down, clients can just fail-over to the other, with no particular action taken on the cluster side, and no waiting, and they can expect to find all data there (subject to mirroring lag)
- Low produce and consume latency for clients connecting to the nearest site
- Independent sites: clients operating toward the same site are not dependent on what is happening in other sites/regions
Active-passive topology
In this topology, data is replicated unidirectionally from a designated source cluster to a secondary target cluster that remains passive:
Consumers and producers interact with the source cluster eu-west1, while MM2 replicates all data over to the eu-north1 cluster.
This represents the “normal operation” state for an active-passive topology. However, in the event of a critical failure or a planned migration, the operational steps—as well as the resulting RTO and RPO—will differ depending on whether you follow a recovery or a failover strategy.
Part 2 will go into those strategies in detail, including what it takes to make consumer groups resume correctly, what “offset translation” really means in practice, and what commonly breaks under load.
Closing and what’s next
MirrorMaker 2 is a powerful tool, but the topology choice determines where the complexity sits: on the platform side, on the client side, or in the operational runbooks.
If you take one thing from Part 1, it should be this: start from scenarios and RTO/RPO, not from “we should replicate Kafka”.
In Part 2, we’ll cover active passive recovery vs failover, consumer offset translation, replication lag under load, topic naming policies, and a pragmatic test approach that actually validates your DR plan.
