You know that old monolithic application the whole enterprise relies on. It works and gets the job done, even though none of your current colleagues understand how. Suddenly your managers want you to bring the company into the age of AI, and they want it now. Here you are, and all they gave you was access to this legacy monolithic codebase.
Rewriting the monolith is unrealistic. It takes a lot of time and puts the whole business at risk since everything relies on it. And it might not even be achievable. A more pragmatic move: turn the monolith database into a real-time event stream with Change Data Capture. The tools that get you there are Debezium and Kafka.
Why this matters
Whatever you plan to do with the data downstream, three things tend to matter.
First, latency. For use cases like fraud detection or recommendations, real-time events are the whole point. Even for less time-critical reporting, the difference between data that’s seconds old and data that’s a day old reshapes what’s possible.
Second, history. Model training, backtesting, reproducibility, audit trails. All of these need the data to be accessible long after the event happened.
Third, structure. Data needs to be governed by consistent schemas so that pipelines stay stable and efficient, without every consumer reinterpreting or transforming what it receives.
CDC streams give you all three out of the box. That’s why this pattern keeps showing up wherever someone is trying to get more out of an old database than the database itself was built to deliver.
How it works
What you need to start is access to the database transaction log. The transaction log is different depending on what database you are running, but in short it’s a log containing every operation against the database. For PostgreSQL it’s called the WAL (Write-Ahead Log). The concept is similar for any database, even though the connection mechanics differ.
Debezium runs inside Kafka Connect and uses a database-specific connector to read the changes. It reacts to committed transaction log entries, typically insert, update and delete records. It does not care about queries or other actions that don’t produce a change to an entity. It acts like a replication client and streams the changes to Kafka.
In this article we’re assuming a PostgreSQL database. The concepts are the same across databases, but the setup details differ.
A few prerequisites before you start. In the PostgreSQL case:
- A running Kafka cluster with Kafka Connect configured, and some specific topics, users and ACLs:
- Kafka Connect specific topics
- A schema history topic to track changes to the database schema
- CDC topics for the change events, for example postgres-public-orders
- Kafka users with the right access to the topics above
- PostgreSQL configured for logical replication:
wal_level = logical max_replication_slots = <minimum number of connectors> max_wal_senders = <minimum number of active consumers + 2>
- Every table you want to stream changes from has a primary key.
- A dedicated Debezium connector user with the permissions: LOGIN, REPLICATION, SELECT, CONNECT.
With that in place, the first step is getting Debezium to connect to PostgreSQL as a replication client. The first time it connects, a couple of things happen that are worth understanding.
The connector creates a replication slot in PostgreSQL, if one does not already exist. This slot tracks which WAL entries Debezium has processed. PostgreSQL uses it to decide which WAL entries need to be retained, ensuring no events are missed. The flip side: if the connector goes down, PostgreSQL keeps accumulating WAL entries waiting to be consumed. This can grow large enough to fill up your disk. Monitoring replication slot lag is non-negotiable.
Debezium will also set up a publication for the tables it needs to track. This can be created manually or set up automatically by Debezium itself.
When the initial setup is in place, Debezium performs an initial snapshot of the database in an isolated mode to ensure a consistent version of the data. It also notes the current log sequence number in the WAL so it knows where to pick up streaming after the snapshot finishes. The snapshot reads all selected table rows and writes them as events to Kafka.
Worth knowing: Debezium supports several snapshot modes. initial is the default and does what’s described above. never skips the snapshot entirely and starts from the current WAL position, which is risky if consumers need the full historical state. schema_only and when_needed exist for specific scenarios. Pick deliberately.
After the initial load, Debezium streams change data continuously by reading from the WAL, starting from the noted log sequence. At that point you have a working CDC pipeline.
One more thing worth saying out loud. The schema history topic mentioned above is what lets Debezium reason about DDL applied to the source database over time. Adding columns is usually safe and downstream consumers see new fields appear. Dropping or renaming columns is where things break, and where having a contract with your consumers matters more than the CDC tooling itself. Same goes for delivery guarantees: Debezium is at-least-once, so consumers must tolerate duplicates.
Other use cases
So, is CDC only useful for getting your monolith into the age of AI? Of course not. There are plenty of other situations where it earns its keep.
Take audit logging. Instead of making the application track who did what on which data, CDC can capture every insert, update and delete at the database level. That keeps auditing as a separate concern from the application itself.
One limitation to be upfront about: CDC captures the database user that made the change, which in most applications is a shared non-human service account. If you need to track which application user or human triggered the change, you have to be deliberate. A common approach is to add an updated_by column to your tables and have the application write the authenticated user’s ID alongside every mutation. CDC then picks it up naturally as part of the row change. A more robust pattern is the outbox pattern, where the application writes an explicit event record in the same transaction as the business operation, including full context like user ID, session and correlation ID. That gives you rich, application-aware audit events without polluting your domain tables.
CDC also shines as a tool for legacy migration, used as a bridge between the new and the old. In practice you put Debezium in front of the old database and have it start producing change events. In phase one, every event comes from the old system, and the new service only reads these events and saves them. In phase two, all read requests are served by the new system. Writes still technically go to the old monolith, but through CDC the new service stays aware of the state.
The trickiest phase is when writes have moved to the new service but parts of the monolith still depend on that data. Here CDC runs in reverse: the new service’s database streams changes back to the monolith, keeping it fed with the data it still needs. This bi-directional sync is temporary, but it’s what makes the migration safe. You’re never in a position where the monolith is serving stale data to parts of the system that haven’t been migrated yet. In a later phase you migrate the remaining writes to a new microservice.
In a real scenario there are more flows than just read and write, but the principle is the same. You migrate step by step, one flow at a time, instead of taking the risk of a big bang cutover.
When not to use it
CDC is powerful but not free. Two things to weigh before committing.
The first is how dependent you actually are on every change event. Let’s say you have a table that gets hammered with updates. Is every change something you need to publish as an event? The risk is producing millions of events on busy tables, which means higher storage costs in Kafka and a lot of CPU usage on the consumers. You should always be able to justify that real-time changes are something the business can actually use. Even if the event-driven story sounds more modern and aligned with your IT strategy, sometimes you only need the current state, and only occasionally.
The second is operational overhead. Running CDC with Kafka and Debezium means managing a Kafka cluster, with everything that entails. You also need to maintain the Debezium connectors and do proper lifecycle management. If you don’t already run Kafka in your organization, this means onboarding a whole new platform. If Kafka fits your broader strategy, fine. If you’re standing up Kafka for the CDC use case alone, think twice.
Conclusion
The beauty of CDC is that it meets you where you stand. You don’t need to refactor anything. You don’t need to understand every corner of that legacy codebase. You just need access to the transaction log.
That said, CDC is not a silver bullet. It comes with operational overhead, eventual consistency has limits, and a rapidly changing schema will cause you pain. The best results come from teams that are deliberate about when and where they apply it, not from those who reach for it reflexively because it sounds modern.
If you’re sitting in front of that monolith right now wondering where to start, start small. Pick one table the business cares about, get it streaming, and let the value speak for itself. The rest tends to follow.
