Event Sourcing (16): Event Semantics and Idempotent for Decentralized Systems

July 6 23:14~24:00; July 7 00:00~01:39

截圖 2022-07-07 上午1.38.24

▲I’m almost exhausted XD

foreword

The reason why villagers use Event Sourcing and CQRS is to develop microservices in many cases. The microservice architecture is a distributed system. Compared with the centralized system, the distributed system has the characteristics of heterogeneity, easy expansion, robustness, high fault tolerance and low coupling between system modules. However, the development of decentralized systems is relatively complex and prone to “errors” if you are not careful.

In DDD, the state between Aggregates reaches eventual consistency through domain events . Similarly, in the microservice architecture, the state synchronization between microservices also achieves eventual consistency through events or messages. However, in a decentralized system, the event delivery itself may also be lost, resulting in the receiver not receiving the event and thus failing to achieve eventual consistency.

This episode describes the properties that event delivery and event handlers must possess in order to properly achieve eventual consistency in a decentralized system.

***

Guaranteed order of events

First of all, the order of the events themselves cannot be messed up, otherwise the state of the event receiver will be wrong. When the event is written to the Event Store, the Event Store itself must ensure that the events in the same event stream must be sorted according to the order of occurrence (writing time point), which is basically no problem. Seeing this, the villagers may think: “Since the events have been sorted in the Event Store, why would the order be out of order?”

Please refer to Figure 1. The sequence of events in the Event Store is correct at the beginning. Suppose this event is an event generated by the User Management Bounded Context in ezKanban, such as UserCreated, UserRenamed, UserEmailChanged, etc. In ezKanban, the Kanban Board Bounded Context (the Core Domain of ezKanban) needs to display the user name on the screen, so it will listen to domain events such as UserCreated and UserRenamed, and then create a cache of User data on its own local end. Therefore, User Management Bounded Context will transmit the internal UserCreated domain events and write them to Topic A of Pulsar.

Suppose that in the Kanban Board Bounded Context, two consumer (event handler) programs are started to read the data of Topic A at the same time in order to “seek speed”. This kind of consumer is called Competing Consumer in the message-oriented architecture, and they will rush to process the data in the Topic. Consumer A and Consumer B in Figure 1 will write another event to the Topic Result after processing the event from the Topic User. Now Consumer A gets the two messages 1 and 3, and Consumer B gets the two messages 2 and 4. Because they execute at different speeds, the order of events in Topic B becomes 2, 1, 4, 3, in a different order from the original event.

截圖 2022-07-07 上午12.11.50

▲Figure 1: Schematic diagram of Event out of order

If today Topic A stores the transfer request, and Topic B stores the result of the transfer, the order of events is usually not important in this case. However, if you want to pass events through the Event Broker and hope that the receiver achieves eventual consistency, you must pay attention not to accidentally cause the event sequence to be chaotic during the event delivery process.

***

At least once

In addition to the order of events cannot be wrong, another requirement for event delivery is that the event must be guaranteed to be seen at least once by the receiver .

Why so much trouble? Why not specify exactly once ? Because it’s hard to do. Please refer to Figure 2. After the Consumer reads Even 1 from Topic A, it crashes before it finishes. After the Consumer restarts, because Even 1 has been read, the latest event in Topic A becomes Even 2. But Even 1 has not been processed by the Consumer, which means that Even 1 has since disappeared from the earth.

截圖 2022-07-07 上午12.27.16

▲Figure 2: After the Consumer reads the event, it fails before it is finished

***

It is also very simple to solve the problem of the disappearance of the event, that is, after the Consumer reads the event, the Topic will not delete the event immediately, until the Consumer has finished processing and notified (ack) the Topic, then the Topic will delete the event, as shown in Figure 3 Show. In Figure 3, in case 1, the Consumer immediately crashes after reading event 1. But because the Consumer has no ack, Topic A still exists in Event 1. Case 2 When the Consumer restarts, event 1 can be read again (this is at least once, at least once, at most unlimited). Case 2 Event 1 will be removed from Topic A when the Consumer is executed and acked.

截圖 2022-07-07 上午12.34.30

▲Figure 3: Events that are read out after the Consumer notifies the Topic will be deleted from the Topic

***

Idempotent

Now a new problem is coming. Although the ack method can be used, the event will be received by the consumer at least once, but what if the consumer repeatedly receives the same event? For example getting duplicate chargeback requests? Then it really becomes a line commonly used by “fraud groups”: “The system is set incorrectly, causing repeated deductions XD”.

Please refer to Figure 4, in case 4 the Consumer receives event 1 and has also done things (update system state), just before it is about to ack, it is pawned, so no ack is successful. Case 5 When it restarts next time, event 1 is processed again, which is obviously not OK. Therefore, Consumer (Event Handler) needs to have Idempotent.

截圖 2022-07-07 上午12.46.30

▲Figure 4: Event 1 is processed twice by Consumer

Simply put, Idempotent refers to the same operation, even if repeated execution will not affect the system state. For example, multiplying any number by 1 does not change the final result, so the operation “multiplying by 1” is idempotent. In software development, traditional CRUD operations, basically RUD are idempotent. Needless to say, reading data N times will not change the system state, and updating and deleting the same data with a fixed value N times will not change the system state. But C (new) is not an idempotent.

Consumer needs to have the meaning of idempotent, which means that Consumer can magically make two identical new effects become one. How to do it? In principle, let the Consumer record the event number (event id) it has read. Every time you read an event from a topic, first go to your local database to query the event, have you seen it before? If there is, directly ack to continue to touch the next stroke, if not, it will be processed normally, as shown in Figure 5.

There is a detail to note in Figure 5. The Consumer stores the processed domain event and the system state update caused by touching the event. These two operations must be placed in the same transaction, otherwise it may happen again, and the Consumer changes. State but it is too late to record the event; or an error condition where the event is recorded but it is too late to save the state.

截圖 2022-07-07 上午1.10.34

▲Figure 5: Storing processed events to achieve idempotent

***

lots of details

Seeing this, I asked the villagers to look back at < Consumer Event Source (10): Implementing Projector >, when Teddy ㄅ ㄧ ㄡˋ. As shown in Figure 6, Projector is a kind of Consumer, so the issues of event ordering, at least once and Idempotent mentioned in this episode must be considered when implementing Projector.

截圖 2022-07-07 上午1.19.55

▲Figure 6: Projector of Read Model needs to have Idempotent

***

next episode preview

If the villagers use EventStoreDB, it is both an Event Store and a simple Event Broker, so it supports event ordering and at least once. Kafka and Pulsar are even more powerful Event Brokers, and of course they have support. However, if the villagers use Rational Database or NoSQL “hand-carved Event Store”, they need to ensure event ordering and at least once. The next episode will talk about how to achieve event ordering and at least once in the case of hand-carved Event Store.

***

Youzo’s inner monologue: As a Maker, you must be the Event Store’s own XD.

This article is reprinted from https://teddy-chen-tw.blogspot.com/2022/07/16idempotent.html
This site is for inclusion only, and the copyright belongs to the original author.

Leave a Comment