Active passive strategies and the pitfalls you need to plan for
In Part 1, we covered the building blocks of MirrorMaker 2 and how to think about topology selection. In Part 2, we go deeper on active passive disaster recovery patterns, what changes operationally between recovery and failover, and the behaviors that tend to surprise teams in production.
Active passive topology, recovery vs failover
In an active passive setup, data is replicated unidirectionally from a designated source cluster to a secondary target cluster that remains passive. Consumers and producers interact with the source cluster while MirrorMaker 2 replicates the data to the target cluster. This is the normal operation state. In a critical failure or a planned migration, the operational steps and the resulting RTO and RPO differ depending on whether you follow a recovery or a failover strategy.
Active passive (recovery)
One implementation of an active passive topology we have used is focused on recovery. In case of a critical failure, the goal is to restore the primary cluster, and recover lost data in the source cluster by mirroring back the missing data from the target cluster. Practically, this involves stopping producers and clients, clearing residual data, and temporarily reversing the MirrorMaker 2 flow so that lost data is recovered from the target cluster into the source cluster.
With this setup, the RTO scales linearly with the amount of data being recovered. If only a lightweight topic needs recovery this can be quick, but if an entire cluster must be recovered the RTO increases rapidly. The RPO is generally low because MirrorMaker 2 is asynchronously replicating traffic in near real time, typically only seconds behind.
This setup can be seen as a trade off where built in Kafka redundancy is just about enough, but in extreme scenarios you still do not want to lose data, so you accept a higher RTO to get a more cost efficient solution. One tweak in this topology is enabling Tiered Storage in the active cluster where cold data is persisted in a blob storage service with high uptime and strong disaster robustness. Then, effectively, the RTO only scales with hot data size.
Active passive (failover)
While the recovery model is about restoring data to the primary cluster, the failover model is about moving operations to the secondary cluster permanently, or until the primary is fully rebuilt. This is common in blue green scenarios or during a critical site failure where the primary cluster cannot be quickly restored. In short, the blue cluster is the source cluster and green is the target cluster mirroring the source as closely as possible. In a migration or critical failure, traffic is redirected to the green cluster, which takes over responsibility of being the source cluster. The target cluster is on standby until failover is necessary.
The RTO is much faster here than in the recovery model, since the data is already present in the target cluster during failover. The recovery time is basically the time it takes for clients to restart and point to the new cluster. The RPO remains low, and can be zero in a planned migration where producers are stopped to let replication catch up before commencing the migration.
While this model is faster, it places more complexity on the platform team during failover. You rely on MirrorMaker 2 having correctly and timely translated consumer group offsets, and that no client is saving or using offsets in an unexpected way. Redirecting traffic introduces additional complexity too. A load balancer can help, but it becomes another possible point of failure.
Common pitfalls
While MirrorMaker 2 is built to handle cross-cluster replication automatically, it isn’t a “set and forget” tool. Moving data between clusters introduces technical challenges that don’t exist when working with a single environment.
Some of the challenges one might encounter – like offsets not matching or replication lag – are not necessarily errors, but rather a result of how MM2 handles data across distributed systems. In this section, we will cover common pitfalls and behaviors that often cause confusion, and how to plan around them.
Offset translation is not exact
A common point of confusion is that the offsets between the source and target clusters are not 1 to 1. A message that has offset 123 in the source cluster might have another offset in the target cluster. In the case that a consumer needs to fail over, MM2 will map it to a consumer group offset in the target cluster that accounts for the mapping of message offsets.
With offsets not necessarily being 1 to 1 between the source and target clusters, if a consumer tries to switch by simply using its source cluster offset, it will end up at the wrong position—which results in either skipping data or having to process large volumes of duplicate messages.
This is why the Checkpoint connector is so critical, since it handles the offset translation between the source and target cluster so that consumers can continue consuming from the correct offset. It is noteworthy that this translation is not exact, and it is important to ensure consumers are idempotent so they can handle small amounts of duplicate messages.
Replication lag spikes under load
Since MM2 is a distributed application running on the cluster, it consumes resources just like any other application. Due to this it is possible that it falls behind under heavy load, by seconds or even minutes depending on the quantity of data being replicated.
Topic naming policy
Another “gotcha” that can occur is when using the default naming policies and realising that all replicated topics are prefixed with the name of the source cluster. For example, the topic called my-topic in the source cluster would be replicated to source.my-topic in the secondary cluster. This would be a point of conflict in the case of doing a failover, where all consumers have been configured to use non-prefixed names.
This is of course designed with infinite loops for more complex replication topologies in mind (such as active-active configurations), but when it comes to DR and cluster migration scenarios this can be a cause for headaches.
To prevent MM2 from prefixing the topics, adding the following will ensure all topics names remain the same across the clusters:
replication.policy.class: “org.apache.kafka.connect.mirror.IdentityReplicationPolicy”
Conclusion
MirrorMaker 2 is a powerful tool to the Kafka ecosystem, but as we have seen, it isn’t a case of “set and forget”. There is no “one size fits all”, the choice between active-active and active-passive topologies depends greatly on the RTO/RPO, risks, cost, time, and most importantly, needs. No matter what topology you decide to go with, the goal is the same: resilience. Whether you’re implementing a robust disaster recovery setup or a large-scale cluster migration, the success of your strategy depends on understanding MirrorMaker 2’s underlying mechanism and the trade-offs of each solution.
Let’s Build Resilient Systems Together
At Irori, we specialise in taking these complex challenges, and turning them into stable, production-ready solutions. We’ve helped many clients navigate the nuances of Strimzi and MirrorMaker 2, by helping them build robust and resilient systems that actually survive real-world failures.
If you are considering a migration, designing a disaster recovery strategy, or simply want to ensure that your current Kafka setup is as solid as possible, we would be happy to help. Feel free to reach out to discuss your architecture or to learn more about how we can aid you in your journey towards a more resilient data platform.
