Distributed Transaction in Java Microservices using SAGA Pattern

What is Distributed Transactions in Microservices?

Let’s understand first what the distributed transaction is and how the distributed transactions work in the micro-services. Suppose you have a client and the client in turn connects to the microservices and the services in turn connect to the database right. So, here the database is co-located, and you can do all the transactions. You can lock the table you can create update delete the records because it is located at same location. Now, if you have a database that is located in different locations, let's see how it can be done.

Microservices accelerates software development through parallelization

Now considering the scenario where a case like you have a client which is connected to the payment services and payment services are connecting to a database. Now you can see in the below image that there is another service called Notification microservice which in turn connects to a different database and this notification service and the database can be located at different locations or it can be out of a different network. In this case, there is no way we can do a transaction between two different databases or two different tables which are located at two different databases. We cannot apply any of the ACID transactions here.

So, here in this is how we are going to achieve this. In this case, we have a very interesting design pattern called “SAGAS” where we can use that pattern to apply the distribution transaction to a table that is located in different places. Now how the distributed transactions using Sagas design pattern works in the micro service-based architecture.

Suppose you have a client. The client intern connects to the micro services, which is a Payment micro service. A payment service, in turn, connects to the database where you maintain all your payment details.

Now I want to do payment and the payment request goes into the payment service, and it gets created in that database. So, now we are going to maintain a state in the table where I call it a “processing” state where payment can either succeed or failed state after receiving the notification from the bank.

Now can we fulfill this payment here? No, because we have not received the notification from the bank whether that payment has been a success or failure. As I told you the notification service is in a different database and the service is also located at a different location. So, how do I make sure that notification is available for this payment request?

So, what I'm going to do here is I am going to send a message from the payment microservice to the Notification service. The Notification service, in turn, checks for the customer’s account balance information and makes sure the payment is done or not. If the payment is not done, then it sends again back to the payment services with the message, which we call the Payment Failed. So, the process state gets changed into the "Failed" state in the payment info Database.

The order is completely canceled as payment is failed, which is similar to what we call is to the rollback. Let's take in the happy path, where users the user does the same payment request where he places a payment request, and it goes to the payment service and if the payment is successful in this case and the payment database got updated to "SUCCESS" state. Now in this happy path case, we noticed that there is a payment done by the customer, which is confirmed by the Notification service. So, the notification service sends the reverse message to the Payment service saying that Payment is successful and approved.

So, what happens over here is the "processed" state becomes "Success" and in this case, you can fulfill the payment process. It is how the whole transaction works, and this is how we can achieve the distributed transaction.

SAGA Pattern Use Cases in Microservices

The saga design pattern is for data consistency in a microservice architecture. A simple use case of online purchase of Stocks and also will explain here in this blog why saga design pattern to be used, different implementations of saga design pattern and disadvantages and this advantage of saga design pattern with the help of this use case.

Here in the above diagram, we have four microservice applications, and each of these microservice has a dedicated Database. So, I am mentioning the online purchase of Stocks through Zerodha or any other broker may it be. Suppose, Application 1 is the UI service, Application 2 is the StockManagement service, Application 3 is the Payment service and application 4 is the Notification service which gives you either the purchase of Stock is done successfully or not.

As a user, when going for the purchase of stock the request first goes through UI service and the UI service request for selection of stocks. The request moves forward to application 2 or StockManagement service and stock selected. Then depending upon the trading day and the value of the stock, the user go for the payment of the stock. So, the request moves forward to payment service.

In the payment service, there are two scenarios new possible: either a successful transaction or a decline of the transaction due to any reason. In both the scenario, the request should move forward to the Notification service if the transaction is successful or it is a failure. Both kinds of states should we notify the user. From the Notification service, the user will get the response in the UI service whether the stock purchase is successful or not. If you look at the simple use case scenario, we have four microservice and a single-use case of purchase of stock, the request is going through different microservices and involving data of different microservices.

Here, you can see a single request is spanning over multiple microservices. When a single request expands over multiple services needs for the automaticity of the transaction and isolation of concurrent transaction is required. By the term automaticity of the transaction, means that the local transaction could either be successful or should not happen. It means if the transaction is failed, it would require to be rollback. By the term isolation of concurrent transaction, means it may happen a situation like a user is logging in by a laptop or by an app and from both the devices user is trying to purchase the same stocks or different stocks.

Usually, services expose APIs that are invoked by Saga. The transient state of the request process between microservices as I mentioned earlier, and the payment service let's imagine the transaction is going on and money is deducted. But due to some reason, the transaction comes to failure. In this case, your money is deducted but the transaction is still displaying failure in UI.

In this case, the amount that is deducted should be returned to the user. This transient state of the request processed between services needs to be persisted in case of failure also. Moreover, in case of failure, there should be a rollback.

These points all are related to data inconsistency. To maintain data consistency, we use the SAGA design pattern.

Details of Saga Design Pattern for distributed transactions in Java Microservices

In Simple terms, SAGE design is nothing but a sequence of the local transaction. As I have mentioned earlier in the request for the purchase of online stock is going through four of the microservice. All the combination of those local transactions is called SAGE. In the Sage design pattern, we have each transaction to a particular microservice is treated as a Local transaction. So, we have here a sequence of local transactions and each local transaction updates data within a single service. The transaction could be sequential or parallel that means it can be synchronous or asynchronous. This design pattern is preferred for long-running Transactions.

If you have only two microservices or three microservices in your entire backend application, then the Saga design pattern is not a preferable approach. In case your request is going through five microservices, then data consistency can look at the SAGA design pattern.

How to Implement Saga in your Java Microservices application?

There are different approaches for the implementation of SAGA design patterns but mostly two are used.

There are different approaches to the implementation of SAGA design patterns but, mostly two are used, which has been explained by the Spring Boot developers.

• Event-based Approach
• Orchestration-based Approach

Event-based Approach

The event-based approach is also called the choreography-based approach. So, if you recall the same use case that I already mention earlier i.e. of online purchase of stocks. We have UI service StockManagement service, Payment service and Notification service. In the case of the event-based approach, each of the microservice generates an event and another microservice listened to that event and performs an action based on the event generation.

Again, when a user tries to purchase the stock, it will first go through with a UI service. The UI service in case of event-based approach implementation generates an event called selectStock() and the StockManagement service listens to this event. So, on listening to this event, the stock is selected, and it will further generate the makePayment() event. Now the payment service will listen to makePayment() event and after the success or the failure of the payment, the payment service will generate a paymentNotification() event.

This payment notification event will contain either success or failure. The Notification service will listen to this paymentNotification event, and the Notification service generates an event called purchaseComplete() in case of success, it is stock purchased successfully, and in case of failure, it will be stock purchased failure and finally, the UI service listened to the Notification service.

Choreography/Event Based Design

Here simply, they are one-to-one relations between events and the Microservices. It may happen that one of your microservice is creating an event and multiple microservices are listening to that event. It may kind of create a cyclic dependency between the microservices and this could be a disadvantage of the event or choreography-based approach. In an Event-based approach, each of the local transactions after successful completion will publish the event that is called domain event, which in turn invokes the local transaction in further succeeding services.

Rollback/Failure Scenario

If there is a failure in the case of an event-based approach. Let me explain now in case of failure of the transaction, we have to prepare for rollback or compensating action. But remember roll back and compensation both are different terms. By the term rollback we mean we are trying to revert the transaction as if it had never happened but in case of compensating, we are admitting that a transaction has happened and it has failed and we are trying to do something else to compensate that transaction.

Assume here at the payment service, your money is deducted but due to some reason, the transaction failed. So, the payment service will generate an event called makePaymenetFailed() and as the money has been deducted from your account and the payment is also getting failed and to compensate this action, this event should be listened to by notification service and by the payment service to revert the deducted amount.

So, the makePaymenetFailed() event will listen by the payment service and that will return or refund to the user or any the microservice that is doing this process and simultaneously by the notification service that payment has failed. After the paymentReceivedFailed() event by the notification service, the notification service will generate an event called stockPurchaseFailed() that will be listened to by the UI service and the response will finally display to the user.

Orchestration-based Approach

In Orchestration or command-based pattern or approach, the microservices do not directly interact with each other. Instead of that, we have a common orchestrator to perform such action. Again, taking the same case of online stock purchase. The user invokes the request for online purchase of stock it will go through UI service. UI service will generate a command purchaseStock(). Instead of directly going through the StockManagement service, the request will go through a central Orchestrator.

The orchestrator on receiving of purchaseStock() command will generate the command selectStock(), and this command will be listened to by the stock management service in the backend, and the stock service will reply, with the next command saying stockChoosen(), and again this will go to the orchestrator.

Upon receiving of stockChoosen() command, Orchestrator will further generate another command saying makePayment() and it will be listened to by your payment service. When the payment service will process the payment transaction then on success or failure of the transaction, it will generate a command called “paymentNotification()”. If the payment is successful and this response will be listened to by the Orchestrator and upon receiving this response, the orchestrator will further generate the next command called PaymentReceived() for the notification service. This payment confirmation will be listened to by the notification service and the notification service give a response to the orchestrator.

If the payment is successful, then the stockPurchaseSuccessful() command is sent to the orchestrator and the orchestrator again responds to the UI service that the stock is purchased successfully, and it will be displayed on the user screen.

If you look at the above diagram, in an orchestrated-based approach, there is a central Orchestrator. This Orchestrator is an additional service that we have implemented apart from the purchase-Stock architecture. So, this is an extra overhead in case we are going to implement an Orchestrator or command based.

If I mention the implementation of Orchestrator, it could be an orchestrating tool like Camunda or you can use a Finite-state machine in Java to build this Orchestrator.

Saga Creation Pattern in Java Microservices

Create PurchaseStock Saga

@SuppressWarnings("common-java:DuplicatedBlocks") public class PurchaseStockSaga implements Saga<PurchaseStockSagaState,PurchaseStockSagaData> { private SagaStateMachine<PurchaseStockSagaState,PurchaseStockSagaData> stockStateMachine; @PostConstruct public void initializeStateMachine(){ this.stockStateMachine = starting(this::initialize) .inState(PurchaseStockSagaState.PURCHASE_STOCK_STARTED) .replyFrom(PaymentServiceProxy.makePayment) .onReply(Success.class, this::handleStockPurchaseSuccessful) .onReply(Failure.class, this::handlePaymentFailed); } @Override public SagaStateMachine<PurchaseStockSagaState,PurchaseStockSagaData> getStockStateMachine(){ return stockStateMachine; }

These are the methods to be invoked when Saga is created. It is nothing but the initialize method. Then for each one of the states, how reactive reply coming back from microservices and then on basis of that specific method to be invoked.

If we are implementing a microservice with the help of a spring boot, there should be limitations on the number of services. There is only one thing we can do, or we can try to do that is store the state of the transaction may be in a database or message broker like Kafka and after some time again try to initiate. So, these retry mechanisms and storing the state of the transaction for the transaction itself in database or message broker you can try if we are more focusing on the compensation action rather than rollback.