Blog · Transactions
Apache Kafka's ghost messages: aborted transactions and read_uncommitted
A producer opens a transaction, writes 20 messages, something goes wrong and the transaction is aborted. For the application those 20 messages never existed. Yet you open the topic in your favourite client and there they are, neatly in a row, as if nothing happened.
It is not a broker bug. It is a matter of isolation level, and it is worth understanding before you spend an hour hunting for an order that is "in the topic but not in the database".
What stays in the log
In Apache Kafka the messages of a transaction land in the log straight away, as they are written. The broker does not wait for the commit to store them. At the end of the transaction it appends a marker (a control record): COMMIT or ABORT.
So an aborted transaction leaves in the log:
- its messages, which stay there until retention or compaction removes them;
- an ABORT marker, which is not shown as a message but takes up an offset.
The reader decides what to do with those messages.
Two ways to read
The setting is the consumer's isolation.level:
| Value | What it sees |
|---|---|
read_uncommitted | everything in the log, including aborted transactions and those still open |
read_committed | only non-transactional messages and those of committed transactions; it stops at the last stable offset (LSO) |
The surprising detail: the defaults differ between libraries. Apache Kafka's Java consumer starts with read_uncommitted. librdkafka, the C library behind many clients in other languages, starts with read_committed.
So two programs reading the same topic, both "with default settings", can show different numbers.
A concrete test
In our correctness bench we wrote a topic with 40 committed messages and 20 from an aborted transaction, then asked two clients to count them:
| Question | Expected | Client with read_committed | Client with read_uncommitted |
|---|---|---|---|
| Messages of the aborted transaction | 0 | 0 | 20 |
| The whole topic | 40 | 40 | 60 |
The second client is not misreading: it reads the whole log. But if you are trying to understand what really happened in your application, those 20 messages are noise, and sometimes dangerous noise.
The last stable offset
With read_committed there is a side effect worth knowing: the consumer does not go past the last stable offset, the first offset that belongs to a transaction still open. If a producer opens a transaction and hangs, read_committed readers stop there, even if committed messages from other producers come after it.
If a partition "does not move" for a transactional consumer while its high watermark keeps growing, the first thing to check is a transaction left open. It closes by itself when the producer's transaction.timeout.ms expires.
How to spot it
- Check the isolation level of the client you are looking at the data with, not just the application's.
- Offsets have gaps: every transaction marker takes an offset. If a number is missing between one message and the next, there is a marker there.
- Compare the counts: if "end minus start" does not match the number of messages read, there are markers, aborted transactions or compacted messages in between.
In Topicup
Topicup reads with read_committed: what you see is what consumer applications using librdkafka's default settings see. A message from an aborted transaction appears neither when reading nor when searching.