Implementing FIFO Pattern in Biz Talk 2010

1. Problem You are implementing an integration point where message order must be maintained. Messages must be delivered from your source system to your destination system in first-in/first-out (FIFO) sequence. 2. Solution In scenarios where FIFO-ordered delivery is required, sequential convoys handle the race condition that occurs as BizTalk attempts to process subscriptions for messages received at the same time. Ordered message delivery is a common requirement that necessitates the use of sequential convoys. For example, FIFO processing of messages is usually required for financial transactions. It is easy to see why ordered delivery is required when looking at a simple example of a deposit and withdrawal from a bank account. If a customer has $0.00 in her account, makes a deposit of $10.00, and then makes a withdrawal of $5.00, it is important that these transactions are committed in the correct order. If the withdrawal transaction occurs first, the customer will likely be informed that she has insufficient funds, even though she has just made her deposit. Sequential convoys are implemented by message correlation and ordered delivery flags in BizTalk Server, as outlined in the following steps. Open the project that contains the schema. (We assume that an XSD schema used to define a financial transaction message is already created.) Add a new orchestration to the project, and give it a descriptive name. In our example, the orchestration is named SequentialConvoyOrchestration. Create a new message, and specify the name and type. In our example, we create a message named FinancialTransactionMessage, which is defined by the FinancialTransactionSchema schema. In the Orchestration View window, expand the Types node of the tree view so that the Correlation Types folder is visible. Right-click the Correlation Types folder, and select New Correlation Type, which creates a correlation type and launches the Correlation Properties dialog box. In the Correlation Properties dialog box, select the properties that the convoy's correlation set will be based on. In our example, we select the BTS.ReceivePortName property, which indicates which receive port the message was received through. Click the new correlation type, and give it a descriptive name in the Properties window. In our example, the correlation type is named ReceivePortNameCorrelationType. In the Orchestration View window, right-click the Correlation Set folder, select New Correlation Set, and specify a name and correlation type. In our example, we create a correlation set named ReceivePortNameCorrelationSet and select ReceivePortNameCorrelationType. From the toolbox, drag the following onto the design surface in top-down order. The final orchestration is shown in Figure 1. Receive shape to receive the initial order message: Configure this shape to use the FinancialTransactionMessage, activate the orchestration, initialize ReceivePortNameCorrelationSet, and to use an orchestration receive port. Loop shape to allow the orchestration to receive multiple messages: Configure this shape with the expression Loop == true (allowing the orchestration to run in perpetuity). Send shape within the Loop shape: This delivers the financial transaction message the destination system. Configure this shape to use an orchestration send port. Receive shape within the Loop shape: This receives the next message (based on the order messages were received) in the convoy. Configure this shape to use the FinancialTransactionMessage, to follow the ReceivePortNameCorrelationSet and to use the same orchestration receive port as the first Receive shape. Figure 1. Configuring a sequential convoy Build and deploy the BizTalk project. Create a receive port and receive location to receive messages from the source system. Create a send port to deliver messages to the destination system. In our solution, we send messages to an MSMQ queue named TransactionOut. In the Transport Advanced Options section of the Send Port Properties dialog box, select the Ordered Delivery option, as shown in Figure 2. Figure 2. Configuring an ordered delivery send port Bind the orchestration to the receive and send ports, configure the host for the orchestration, and start the orchestration. 3. How It Works In this solution, we show how a convoy can be used to sequentially handle messages within an orchestration. The sequential convoy consists of the ReceivePortNameCorrelationSet and the ordered delivery flags specified on the receive location and send port. The first Receive shape initializes the correlation set, which is based on the receive port name by which the order was consumed. Initializing a correlation set instructs BizTalk Server to associate the correlation type data with the orchestration instance. This allows BizTalk to route all messages that have identical correlation type criteria (in our case, all messages consumed by the receive port bound to the orchestration) to the same instance. The Ordered Processing flag further instructs BizTalk Server to maintain order when determining which message should be delivered next to the orchestration. NOTE The adapter used to receive messages into sequential convoy orchestrations must implement ordered delivery. If an adapter supports ordered delivery, the check box will appear on the Transport Advanced Options tab. The Send shape in the orchestration delivers the financial transaction message to a destination system for further processing. The second Receive shape follows the correlation set, which allows the next message consumed by the receive port to be routed to the already running orchestration instance. Both the Send and second Receive shapes are contained within a loop, which runs in perpetuity. This results in a single orchestration instance that processes all messages for a given correlation set, in sequential order. This type of orchestration is sometimes referred to as a singleton orchestration. 3.1. Working with Sequential Convoys The term convoy set is used to describe the correlation sets used to enforce convoy message handling. While our example used only a single correlation set, you can use multiple correlation sets to implement a sequential convoy. Regardless of how many correlation sets are used, sequential convoy sets must be initialized by the same Receive shape and then followed by a subsequent Receive shape. Sequential convoys can also accept untyped messages (messages defined as being of type XmlDocument). You can see how this is important by extending the financial transaction scenario, and assuming that a deposit and withdrawal are actually different message types (defined by different schemas). In this case, a message type of XmlDocument would be used on the convoy Receive shapes. 3.2. Fine-Tuning Sequential Convoys While our example does implement a sequential convoy, you can fine-tune the solution to handle sequential processing in a more efficient and graceful manner. As it stands now, the SequentialConvoyOrchestration handles each message received from the source MSMQ queue in order. This essentially single-threads the integration point, significantly decreasing throughput. Single-threading does achieve FIFO processing, but it is a bit heavy-handed. In our example, all transactions do not have to be delivered in order—just those for a particular customer. By modifying the convoy set to be based on a customer ID field in the financial transaction schema (instead of the receive port name), you can allow transactions for different customers to be handled simultaneously. This change would take advantage of BizTalk Server's ability to process multiple messages simultaneously, increasing the performance of your solution. NOTE In this scenario, you must use a pipeline that promotes the customer ID property (such as the XmlReceive pipeline) on the receive location bound to the sequential convoy orchestration. The PassThru receive pipeline cannot be used in this scenario. Changing the convoy set to include a customer ID field would also impact the number of orchestrations running in perpetuity. Each new customer ID would end up creating a new orchestration, which could result in hundreds, if not thousands, of constantly running instances. This situation is not particularly desirable from either a performance or management perspective. To address this issue, you can implement a timeout feature allowing the orchestration to terminate if subsequent messages are not received within a specified period of time. Take the following steps to implement this enhancement. The updated orchestration is shown in Figure 3. Add a Listen shape in between the Send shape and second Receive shape. Move the second Receive shape to the left-hand branch of the Listen shape. Add a Delay shape to the right-hand branch of the Listen shape. Configure this shape to delay for the appropriate timeout duration. In our example, we set the timeout to be 10 seconds by using the following value for the Delay property: new System.TimeSpan(0,0,0,10) Add an Expression shape directly below the Delay shape. Configure this shape to exit the convoy by using the following expression: Loop = false; Figure 3. Configuring a terminating sequential convoy Finally, you can enhance the solution to ensure that messages are successfully delivered to the destination system before processing subsequent messages. In the current solution, messages are sent out of the orchestration to the MessageBox database via the orchestration port. Once this happens, the orchestration continues; there is no indication that the message was actually delivered to its end destination. For example, if the destination MSMQ queue was momentarily offline, a message may be suspended while subsequent messages may be delivered successfully. Take the following steps to implement this enhancement. The updated orchestration portion is shown in Figure 4. Change the orchestration send port's Delivery Notification property to Transmitted. Add a Scope shape directly above the Send shape. Move the Send shape inside the Scope shape. Add an exception handler by right-clicking the Scope shape. Configure this shape to have an Exception Object Type property of Microsoft.XLANGs.BaseTypes.DeliveryFailureException. Enter a descriptive name for the Exception Object Name property. Add an Expression shape inside the exception handler block added in the previous step. Configure this shape to appropriately handle delivery failure exceptions. In our solution, we simply write the event to the trace log via the following code: System.Diagnostics.Trace.Write("Delivery Failure Exception Occurred - " + deliveryFailureExc.Message); Figure 4. Capturing delivery failure exceptions