BizTalk 2010 WCF Custom Adapter Error : Unable to create binding configuration element for editing. Check the values of the bindyingType and BindingConfiguration properties.

Q: 

I have a project that was tested and is working fine in dev. I have exported it from dev and imported it to the production successfully (or at least the I had the impression that it was successful as BizTalk message said so). The project polled a record from Oracle db and does some transformation. We have one test and one production Oracle database. Since it is now in production, I want BizTalk to poll record from the production database and therefore went in the Receive port to configure the database. However, I can not do so as I got this message:

"Unable to create binding configuration element for editing. Check the values of the bindyingType and BindingConfiguration properties. (Microsoft.BizTalk.Adapter.Wcf.Converters.CreateBindingException) Unable to get binding type for binding extension "string". Verify the binding extersion is registered in the machine .config."

Please advise how to solve the issue? I used the import/export MSI to get and put from dev to the production.

Possible Sol 1 :

This error means that custom binding extension for a WCF-Custom or a WCF-CustomIsolated transport was not configured properly. To resolve this error do one or more of the following:

• Ensure the machine.config file in %WinDir%\Microsoft.NET\Framework\v4.0.30319\Config has the <bindingExtensions> element configured properly.
• In Windows Explorer, go to %WinDir%\Assembly, and make sure the assemblies implementing the custom binding extension are installed properly.
• For the WCF-Custom adapter, in the BizTalk Administration console, restart the host instance running the WCF transport.
• For the WCF-CustomIsolated adapter, in the IIS Management console, restart the application pool hosting the WCF transport.
• Open and close the BizTalk Administration console if it was opened.

If you do not have binding for WCF-Custom or WCF-CustomIsolated than first point (machine.config) still applies.

HTH

Read more ...

Cumulative Update Package for BizTalk Server 2009| BizTalk 2009 KB Update for visual studio bugs

Problems with BizTalk project that references another BizTalk project in Visual Studio on a computer that is running BizTalk Server 2009 (KB 977428)

The above fix is now included in the Cumulative update package 1 for BizTalk Server 2009 (KB 2429050)

Cumulative update package 1 for BizTalk Adapter Pack 2.0 (KB 2444952)

Read more ...

Saving suspended messages in BizTalk (filtering on orchestration or port name)

  Post By:  Thiago Almeida.

Last week, I needed to save 700+ suspended messages to file. I didn’t want to go in to each instance and click ‘save to file’ 700 times. A quick search in the web did not find what I want, however there were a number of articles about extracting messages from the BizTalk tracking database.

My scenario was for an existing BizTalk 2004 implementation in the company and is only a once-off thing I need to do. For the newer BizTalk 2006 applications, there is the ‘Failed Message Routing’ feature that can be enabled on send ports and receive ports. The failed messages could then be easily subscribed to a send port to output to file.

Then it occured to me that the WMI script Terminate.vbs has the option to save the messages before terminating the instances (http://go.microsoft.com/fwlink/?LinkID=107591 and slightly updated for 2009). Thus changing this script to do what I want required the least effort. I could just use the script as it is to save all the messages and terminate the instances. However it didn’t take much to modify it to take a parameter for filtering on an instance name and to only save messages (and not terminate them). Below is the usage description of the save_messages.vbs script and the actual script. It works for BizTalk 2004, 2006 and 2009.

There is also a replacement of the Terminate.vbs script: Biztalk Terminator.

Usage:
cscript save_messages.vbs < -Z | -A | -DIS | -SR | -SNR > [Port/Orchestration name]

-Z saves all “Zombie” instances (e.g. completed with discarded messages)
-A saves all suspended and zombie instances as well as all routing failure reports
-SR saves suspended resumable instances only
-SNR saves suspended non-resumable instances only
-DIS saves all dehydrated ‘isolated adapter’ instances
optionally supply the name of the orchestration or port name to filter on specific instances

Ensure that the C:\Temp folder exists before running as that is where it saves the instances

Example: cscript save_messages.vbs -z “E-Reporting Data Transform Port”

Visual Basic Script:

‘ save_messages.vbs
‘ Enter save_messages.vbs with no arguments from a command prompt for usage
‘ This script needs to be run under a user account that is a member of the BizTalk Administrators
‘ group. This script needs to be run on a machine that is configured with BizTalk administration
‘ tools.

dim objBtsWmiNS, objMsg, svcinsts, inst, msg, ndx, size

Dim aryHostNames()
Dim aryObjQueues()
Dim aryHostBatchSize()

Dim strKey2Instance
Dim strQuery2Msg
Dim strServiceName

On Error Resume Next
Dim objArgs: Set objArgs = WScript.Arguments
If ( objArgs.Count = 0 OR objArgs.Count > 2) Then
PrintUsage()
wscript.quit 0
End If

wmiQuery = “”

‘ServiceStatus = 16 – ‘Completed With Discarded Messages’ in BizTalk Server 2004
‘ServiceStatus = 32 – ‘Suspended (not resumable)’
‘ServiceStatus = 4 – ‘Suspended (resumable)’
‘ServiceClass = 64 – ‘Routing Failure Report’
‘ErrorId = “0xC0C01B4C” – is how ‘Completed With Discarded Messages’ are exposed in BizTalk Server 2006

If (objArgs(0) = “-Z” OR objArgs(0) = “-z”) Then
wmiQuery = “select * from MSBTS_serviceinstance where ServiceStatus=16 OR ErrorId=’0xC0C01B4C’”
End If

If (objArgs(0) = “-A” or objArgs(0) = “-a”) Then
wmiQuery = “select * from MSBTS_serviceinstance where ServiceStatus=4 OR ServiceStatus=32 OR ServiceStatus=16 OR ErrorId=’0xC0C01B4C’ OR ServiceClass=64″
End If

If (objArgs(0) = “-SR” or objArgs(0) = “-sr”) Then
wmiQuery = “select * from MSBTS_serviceinstance where ServiceStatus=4″
End If

If (objArgs(0) = “-SNR” or objArgs(0) = “-snr”) Then
wmiQuery = “select * from MSBTS_serviceinstance where ServiceStatus=32″
End If

If (objArgs(0) = “-DIS” or objArgs(0) = “-dis”) Then
wmiQuery = “select * from MSBTS_serviceinstance where ServiceClass=32 AND ServiceStatus=8″
‘ServiceClass = 32 ‘Isolated Adapter
‘ServiceStatus = 8 ‘Dehydrated
End If

saveMessagesBeforeTermination = True

If ( objArgs.Count > 1) Then
strServiceName = objArgs(1)
End If

If(wmiQuery = “”) Then
PrintUsage()
wscript.quit 0
End If

wscript.echo “–>Connecting to BizTalk WMI namespace”
Set objBtsWmiNS = GetObject(“WinMgmts:{impersonationLevel=impersonate, (security)}\\.\root\MicrosoftBizTalkServer”)
If Err <> 0 Then
CheckWMIError
wscript.quit 0
End If

wscript.echo “–>Getting BizTalk host collection”
Set hosts = objBtsWmiNS.ExecQuery(“select * from MSBTS_HostSetting”)
If Err <> 0 Then
CheckWMIError
wscript.quit 0
End If

hostCount = hosts.count

ReDim aryHostNames(hostCount – 1)
ReDim aryObjQueues(hostCount – 1)
ReDim aryHostBatchSize(hostCount – 1)

wscript.echo “–>Retrieve BizTalk host names and loading host queues”
ndx = 0
For Each host in hosts
wscript.echo “Found host ” & host.Properties_(“Name”)
aryHostNames(ndx) = host.Properties_(“Name”)
Set aryObjQueues(ndx) = objBtsWmiNS.Get(“MSBTS_HostQueue.HostName=”"” & aryHostNames(ndx) & “”"”)
If Err <> 0 Then
CheckWMIError
wscript.quit 0
End If
ndx = ndx + 1
Next

wscript.echo “–>Getting collection of service instances”
Set svcinsts = objBtsWmiNS.ExecQuery(wmiQuery)

‘Iterate through instances and save them in host-specific arrays.

wscript.echo “–>Start iterating service instances”
totalCount = 0
For Each inst in svcinsts
If (objArgs.Count = 1 Or (objArgs.Count > 1 And strServiceName = inst.Properties_(“ServiceName”) ) ) Then

wscript.echo “Found suspended instance “”" & inst.Properties_(“ServiceName”) & “”" on host ” & inst.Properties_(“HostName”)
‘Resolve host index
For hostIdx = 0 To hostCount-1
If aryHostNames(hostIdx) = inst.Properties_(“HostName”) Then
Exit For
End If
Next

’16 is an internal service class that cannot be terminated
If 16 = inst.Properties_(“ServiceClass”) Then
wscript.echo “Skipping BizTalk internal service instances (they cannot be terminated anyway)”
Else
’64 is a routing failure report and doesn’t have messages that can be saved
If 64 = inst.Properties_(“ServiceClass”) Or 16 = inst.Properties_(“ServiceClass”) Then
saveMessagesBeforeTermination = False
End If

errorCountSavingMessages = 0
If saveMessagesBeforeTermination Then
strQuery2Msg = “select * from MSBTS_MessageInstance where ServiceInstanceID=”"” & inst.Properties_(“InstanceId”) & “”"”
Set msgInsts = objBtsWmiNS.ExecQuery(strQuery2Msg)
For Each msg in msgInsts
msg.SaveToFile “C:\Temp”
If Err <> 0 Then
CheckWMIError
wscript.echo “Failed to save MSBTS_MessageInstance”
wscript.echo Err.Description & Err.Number
errorCountSavingMessages = errorCountSavingMessages + 1
Else
wscript.echo “Saved message ” & msg.Properties_(“MessageInstanceID”)
End If
Next
End If

totalCount = totalCount + 1
End If

End If
Next

‘ Delete whatever is left
For hostIdx = 0 To hostCount-1
If aryHostBatchSize(hostIdx) > 0 Then
TerminateAccumulatedInstacesForHost hostIdx
End If
Next

wscript.echo “SUCCESS> ” & totalCount & ” instances were found and attempted to be saved”

‘This subroutine deals with all errors using the WbemScripting object.
‘Error descriptions are returned to the user by printing to the console.
Sub CheckWMIError()

If Err <> 0 Then
On Error Resume Next
Dim strErrDesc: strErrDesc = Err.Description
Dim ErrNum: ErrNum = Err.Number
Dim WMIError : Set WMIError = CreateObject(“WbemScripting.SwbemLastError”)

If (TypeName(WMIError) = “Empty” ) Then
wscript.echo strErrDesc & ” (HRESULT: ” & Hex(ErrNum) & “).”
Else
wscript.echo WMIError.Description & “(HRESULT: ” & Hex(ErrNum) & “).”
Set WMIError = nothing
End If

‘wscript.quit 0
End If

End Sub

Sub PrintUsage()
wscript.echo “Usage:”
wscript.echo “cscript save_messages.vbs < -Z | -A | -DIS | -SR | -SNR > [Port/Orchestration name]“
wscript.echo
wscript.echo “  -Z saves all “”Zombie”" instances (e.g. completed with discarded messages)”
wscript.echo “  -A saves all suspended and zombie instances as well as all routing failure reports”
wscript.echo “  -SR saves suspended resumable instances only”
wscript.echo “  -SNR saves suspended non-resumable instances only”
wscript.echo “  -DIS saves all dehydrated ‘isolated adapter’ instances”
wscript.echo “  optionally supply the name of the orchestration or port name to filter on specific instances”
wscript.echo
wscript.echo “  Ensure that the C:\Temp folder exists before running as that is where it saves the instances”
wscript.echo
wscript.echo “  Example: cscript save_messages.vbs -z “”E-Reporting Data Transform Port”"”
wscript.echo
End Sub

Read more ...

Synchronous To Asynchronous Communication Without An Orchestration in BizTalk

Source : MSDN Blog  Paulo Salvatori

Scenario

Some months ago I had the chance to work with a customer who asked me a very interesting and tricky question: does BizTalk Server 2006 R2 support synchronous-to-asynchronous messaging only flows, without the need of an orchestration to couple a Two Way Request Response Receive Location with a One-Way Send Port to send out the request and a One-Way Receive Location to get response back from the invoked system?

The use case the customer wanted to reproduce was the following:

1. A Two-Way SOAP Receive Location receives a request message and publishes it to the BizTalkMsgBoxDb.
2. The message is consumed by a MQ Series One-Way Send Port which writes the document into an outgoing MQ queue.
3. The response message is received by a given MQ Series One-Way Receive Location which reads the document from an incoming MQ Queue and publishes the message to the BizTalkMsgBoxDb.
4. The response message is finally received by the Two-Way SOAP Receive Location which received the original request and then the document is returned to the caller.

I started to create a POC of this scenario, using the FILE adapter in place of the MQ Series Adapter and folders in place of MQ queues. In the majority of BizTalk applications that use a Request-Response receive port, the request message is used to start a given Orchestration or it is directly relayed to a solicit-response Send Port. When the Orchestration or Send Port provides the response message, this message is routed back to the Two-Way Receive Location that submitted the original request message. These two design patterns are fully supported by the product group and the picture below depicts the case where a Two-Way Request Response Receive Location is directly coupled with a Two-Way Solicit Response Send Port which is responsible for invoking an external web service.

But what about our case in which the request and response are respectively sent and received to/from an external target system not using a Two-Way Solicit-Response Send Port, but using a One-Way Send Port while and a One-Way Receive Location? The standard solution is using an Orchestration to correlate the request submitted to the external system with the response received from this latter. But as you can see from the picture below, this pattern requires 8 roundtrips to the BizTalkMsgBoxDb to complete, 4 message writes and 4 message reads.

In other words, this approach requires multiple roundtrips to the MessageBox to publish and consume messages to/from the BizTalkMsgBoxDb and to hydrate and dehydrate the internal state of the orchestration and all these operations can dramatically increase the contention on SQL and reduce the overall throughput and speed of the application.

So the following question spontaneously emerged: is there any technique to implement the same behavior getting rid of the orchestration? This pattern would allow to eliminate 2 messages publications and 2 message reads and this would allow to speed up the execution and to decrease the contention on the BizTalkMsgBoxDb.

The solution to this problem rests in understanding how BizTalk matches a response message to an initial request message. After you understood the internal subscription matching mechanism implemented by BizTalk, the second step is to individuate the context properties that need to be propagated along and promoted in the One-Way Receive Location in order to allow the Two-Way Receive Location to receive the response when this latter is published to the BizTalkMsgBoxDb.

When a message is received through a Request-Response Receive Location and published to the BizTalkMsgBoxDb, an instance subscription is created to allow the Receive Location to receive the response back when this latter is published to the MessageBox. This subscription expression is always composed of the EpmRRCorrelationTokenpromoted property that is in the request message's context, and a RouteDirectToTP promoted property with a value of True.

In particular the EpmRRCorrelationToken is a string containing the name of the BizTalk node which received the request, the process id of the host instance which received the message and a GUID which represents a correlation id. For more info on these and other context properties you can review the following article on MSDNhttp://msdn.microsoft.com/en-us/library/ms966048.aspx.

I made some tests and I quickly realized that those properties were not sufficient to correlate a response back to the original request. In other words, even if the subscription expression is composed of EpmRRCorrelationToken and the RouteDirectToTP properties, promoting them with the expected value in the message context of the response document before publishing this latter is not sufficient to pass message back to the Receive Location. It was clear that it was necessary to propagate some additional context properties, even if they were not used by the subscription expression. After some attempts, I was able to identify the set of context properties that were necessary to correlate a response message back and I was able to create a sample were a synchronous WCF Request Response is couple to 2 asynchronous One-Way FILE Ports.

The Solution

I created a sample BizTalk application called SyncToAsync containing a Two-Way Receive Port called SyncToAsync.Request.ReceivePort. Subsequently, I created a WCF-NetTcp Request Response Receive Location called SyncToAsync.Request.WCF-NetTcp.ReceiveLocation and a WinForm driver application to submit requests to it. Using the WCF Service Publishing Wizard I created a MetadataExchange Endpoint for the web service exposed by the WCF Receive Location and then I used the Add Service Reference command provided by Visual Studio 2005 to create a proxy class in order to submit messages to it. I selected the XmlReceive and PassThruTransmit pipelines on theSyncToAsync.Request.WCF-NetTcp.ReceiveLocation to process respectively the incoming request and the outgoing response messages. In particular, The XmlReceivepipeline contains the Xml Disassembler component which is responsible to probe and individuate the message type and to promote this information along with other context properties. The second step was creating a One-Way FILE Send Port that subscribed all the messages published by the former Receive Location. In order to that I just used theBTS.ReceivePortName = SyncToAsync.Request.ReceivePort predicate as Filter expression. At this point I enabled the SyncToAsync.Request.WCF-NetTcp.ReceiveLocation WCF Receive Location and maintained disabled the SyncToAsync.Request.FILE.SendPort Send Port. Then, using the .NET driver application, I submitted some messages to the BizTalk application. Since no subscribers were active, those messages were suspended. Therefore, using the Administration Console I could review the promoted properties of the incoming request message. After some attempts, I realized that the properties highlighted below (CorrelationToken, EpmReqRespCorrelationToken,ReqRespTransmitPipelineID, IsRequestResponse) plus the RouteDirectToTP that is not promoted on the original request message, were the properties that had to be propagated and subsequently promoted on the One-Way FILE Receive Location used to publish the response to MessageBox.

For the demo I created a very simple XML Schema to represent the basic attributes of a Book entity. I called the schema below InboundBook and this represents the format of the incoming message sent by the .NET Driver application and received by the WCF Receive Location.

Then I created an extended version of the above XML schema in order to demote some of the aforementioned context properties inside the outgoing message. I called this schema OutboundBook. As you can see in the picture below, this schema is composed of a Header and a Body:

• The Header contains the CorrelationToken, EpmRRCorrelationToken and ReqRespTransmitPipelineID elements. In particular, this latter will contain the GUID of the transmit pipeline used by the WCF Receive Location to process the outbound response message. I promoted these 3 elements respectively to the properties with the same name exposed by the BTS.bts_system_properties schema contained in the Microsoft.BizTalk.GlobalPropertySchemas assembly. In this way, if on the Send Port I use any send pipeline containing the XML Assembler component (e.g. the XmlTransmit standard pipeline), at run-time, when the pipeline is executed, the value of those message context properties are demoted to those elements.
• The Body contains the payload of the message, that is the ISBN and Title elements.

Then, I created a map to transform an InboundBook message into a OutboundBook document on the FILE Send Port.

For the response message, I created an XML Schema to represent the author of a certain book. Even in this case, the schema is composed of a Header and Body elements. The Header contains the following elements:

• CorrelationToken.
• EpmRRCorrelationToken.
• ReqRespTransmitPipelineID.
• IsRequestResponse.
• RouteDirectToTP.

Each of those elements are promoted to the context properties having the same name and contained in the Microsoft.BizTalk.GlobalPropertySchemas assembly. In particular, the IsRequestResponse and RouteDirectToTP elements are defined as boolean and they have a Fixed value equal to 1.

Finally, I created a One-Way Receive called SyncToAsync.Response.ReceivePort and a corresponding FILE Receive Location calledSyncToAsync.Response.FILE.ReceiveLocation. I set the standard XmlReceive pipeline on this Receive Location which contains the XmlDisassembler component that is responsible for promoting contained in the Header section of an Author message.

Inside the configuration file of the Driver application you can specify the location of the inbound and outbound folder used respectively by the FILE Receive and Send Ports. The client application used a FileSystemWatcher instance to trigger when a file is published by the Send Port on the outbound folder. When this event happens, a dialog opens where you can review the book information and enter the name and surname of the corresponding author. When you press the Submit button, an instance of the Author message is created for you by the Driver application, the value of the Header elements of the OutboundBook message are copied to the corresponding elements inside the Header of the Author message and the value of the IsRequestResponse and RouteDirectToTP elements is set to true. Then, the Author message is written in the inbound folder where it is received by the SyncToAsync.Response.FILE.ReceiveLocation which promotes the context properties and publishes the message to MessageBox. When the XML document is published, subscriptions are evaluated and the response message is passed to the WCF Receive Location which returns the message to the client application. The following picture depicts the overall architecture of the demo.

You can download the code here.

Conclusions

This technique is a good way to couple a synchronous Request Response Port with 2 Asynchronous One-Way Ports without using an Orchestration. This approach, compared with the standard approach which makes use of an Orchestration to implement the same behavior, allows saving 4 roundtrips to the MessageBox. Therefore, this pattern allows to decrease the contention on the SQL Server instance running the master BizTalkMsgBoxDb and speed up the overall message processing. By the way, I don’t think that the Product Group would support this solution, even if it doesn’t require any custom component. In fact, it uses context properties that were not probably meant to be used by developers, even if the BizTalk programmers’ community knows and uses them. Remember in fact that Microsoft could in any moment release a fix or a service pack that could change the behavior of the subscriptions evaluation and of the request-response correlation, invalidating this technique. On the other hand, this eventuality is not likely because the request-response correlation mechanism is quite consolidated and I don’t think they would have time and will to re-engineer this mechanism.

I have also some ideas on how to modify/customize the technique I described to obtain the same behavior in a slightly different way: for example, instead of including all the necessary context properties in the header of the response message and promote all of them, you could promote the EpmRRCorrelationToken, CorrelationToken andRouteDirectToTP, since the IsRequestReponse and ReqRespTransmitPipelineID are originally written properties. The solution I implemented is probably the fastest one, but it requires all the 5 properties to be propagated and promoted. Another approach could be to save the value of these properties in a custom DB or in a distributed cache likeAppFabric Caching with a custom pipeline component in the transmit pipeline run by the One-Way Send Port and retrieve them in the receive pipeline on the One-Way Receive Location. This mechanism would allow you to propagate just one unique identifier or correlation id that should be repeated in the response message. If you don’t want to use the EpmRRCorrelationToken or the CorrelationToken, you could even use an existing ID already present in the request and response documents as the OrderID orCustomerID.  This approach is less performant (since you need to access a custom DB on the send and receive pipelines) but more flexible and less intrusive, and it’s relatively easy to implement.

Read more ...

Biztalk Aggregation Pattern For Large Messages

Source : MSDN Blogs Richard Seroter

There are a few instances of Aggregator patterns for BizTalk Server, but I recently had a customer who was dealing with a very large batch, and therefore the existing patterns were insufficient. So, I threw together a sample that showed how to handle very large batches in a fairly speedy manner.

In this use case, we have an XML message come into BizTalk, want to split out the batch, process each message individually, then aggregate the whole batch back together again and send a single message out. Now using existing aggregation patterns didn't work since they mostly involve building up an XML file in memory. When a batch of 1000+ message got submitted it was taking over 10 minutes to process the batch, which was way too slow.

So, my solution uses a combination of envelope de-batching, sequential convoy pattern, and file streaming. The first part of the solution was to build the XML schemas necessary. The first schema, the envelope schema, looked like this ...

Three things to note here. First, I set the <Schema> node's Envelope property to Yes. This way I won't have to set anything in my pipeline for it to process the batch. Secondly, I promoted both BatchID andCount fields. We'll see the significance in a moment. Thirdly, I set the Body XPath property of the root node to the Record node of the schema. This way, the pipeline will see that it's an envelope, and, know where to split the batch at.

The second schema is the body schema. It looks like this ...

Notice that it too has promoted properties. By doing this, the values from the master envelope schema get copied down to EACH debatched message without me doing anything special. So, setting up my schemas this way, in combination with using the standard XML receive pipeline, will cause my message to be debatched, and, push values down from the envelope to each message. Cool!

Remember that I have to rebuild the message on the way out. So, I use the file system to build this message up, instead of doing it in memory and increasing my footprint. Therefore, when I first receive the message into my orchestration I need to create this temporary file on disk (and re-create the envelope header). Likewise, when all the messages have been received and processed, I also need to write the closing envelope text. This way I have a temporary file in disk that contains the envelope, and all the processed message bodies.

So, I wrote a quick component to write the envelope header and footer to disk. It also will load the completed file and return an XML document. It looks like this ...

public void WriteBatchHeader()
{
string docpath = @"C:\Data\AggTemp\Dropoff\output.xml";
//envelope header text
string header = "";

StreamWriter sw = new StreamWriter(docpath, true);
sw.Write(header);
sw.Flush();
sw.Close();
}

public void WriteBatchFooter()
{
string docpath = @"C:\Data\AggTemp\Dropoff\output.xml";
//closing envelope text
string footer = "";

StreamWriter sw = new StreamWriter(docpath, true);
sw.Write(footer);
sw.Flush();
sw.Close();
}

public XmlDocument LoadBatchFile()
{
string docpath = @"C:\Data\AggTemp\Dropoff\output.xml";

XmlDocument xmlDoc = new XmlDocument();
StreamReader sr = new StreamReader(docpath);
xmlDoc.Load(sr);
sr.Close();

return xmlDoc;
}

Alright, now I have my orchestration. As you can see below, in this process I start by:

• Receive the first debatched message.
• Call my helper component (code above), create the temporary file, and add the top XML structure.
• Then I send the message out (initializing the correlation set) via FILE adapter (see below for configuration).
• Then in the Expression shape I set the loop counter off of the promoted/distinguished Count field (which holds the number of records in the entire batch).
• I then process each subsequent debatched message via a sequential convoy, following the previously initialized correlation set. In this example, I do no further message processing and simply send the message back out via the FILE adapter.

Once the loop is done, that means I've processed all the debatched records. So, the last half of the orchestration deals with the final processing. Here I do the following:

• Add a 10 second delay. This was necessary because the messaging engine was still writing files to disk as the next component was trying to add the closing XML tags.
• Call the helper component (code above) to write the XML footer to the entire message.
• Next I construct the outbound message by setting the message variable equal to the response of the LoadBatchFile() defined above. I added trace points to see how long this took, and it was subsecond. This was to be expected even on the large file since I'm just streaming it all in.
• Finally, send the rebuilt message back out via any adapter.

The last step was to configure the FILE send adapter being used to write this file to disk. As you can see in the picture, I set the PassThruTransmit pipeline and configured the Copy mode to Append. This caused each message sent to to this port to be added to the same outbound file. 

So, when it was all said and done, I was able to process a batch with 1000 messages in around 2 minutes on a laptop. This was significantly better than the 10+ minutes the customer was experiencing. I'm still using a sequential convoy so I can't process messages as quickly as if I was spawning lots of unique orchestrations, but, our speed improvement came from streaming files around vs. continually accessing the XML DOM. Neat stuff.

Read more ...

Biztalk Asynchronous Aggregation Pattern

Source : MSDN , Windows Server AppFabric Customer Advisory Team, Technocrati Blog

Developing a BizTalk Server solution can be challenging, and especially complex for those who are unfamiliar with it. Developing with BizTalk Server, like any software development effort is like playing chess. There are some great opening moves in chess, and there are some great patterns out there to start a solution. Besides being an outstanding communications tool, design patterns help make the design process faster. This allows solution providers to take the time to concentrate on the business implementation. More importantly, patterns help formalize the design to make it reusable. Reusability not only applies to the components themselves, but also to the stages the design must go through to morph itself into the final solution. The ability to apply a patterned repeatable solution is worth the little time spent learning formal patterns, or to even formalize your own. This entry looks at how architectural design considerations associated to BizTalk Server regarding messaging and orchestration can be applied using patterns, across all industries. The aim is to provide a technical deep-dive using BizTalk Server anchored examples to demonstrate best practices and patterns regarding parallel processing, and correlation.

The blog entry http://blogs.msdn.com/b/quocbui/archive/2009/10/16/biztalk-patterns-the-parallel-shape.aspx examines a classic example of parallel execution. It is consists of de-batching a large message file into smaller files, apply child business process executions on each of the smaller files, and then aggregating all the results into a single large result. Each subset (small file) represents an independent unit of work that does not rely or relate to other subsets, except that it belongs to the same batch (large message). Eventually each child process (the independent unit of work) will return a result message that will be collected with other results, into a collated response message to be leveraged for further processing.

This pattern is a relatively common requirement by clients and systems that interact with BizTalk Server, where a batch is sent in and they require a batch response with the success/failure result for each message. With this pattern one can still leverage the multithreaded nature of BizTalk's batch disassembly while still conforming to requirement of the client.

Let’s closely examine how this example works. First, it uses several capabilities of BizTalk that are natively provided – some that are well documented, and some that are not. A relatively under-utilized feature - that BizTalk Server natively provides, is the ability to de-batch a message (file splitting) from its Receive pipeline. BizTalk Server can take a large message with any number of child records (which are not dependent with one another, except for the fact that they all belong to the same batch), and split out the child records to become independent messages for a child business process to act on. This child business process can be anything – such as a BizTalk Orchestration function, or a Windows Workflow.

	BizTalk Server uses the concept of envelopes, which are BizTalk Schemas to define the parent and child data structures. These schemas are then referenced in a custom receive pipeline’s disassembler properties. This capability works on both XML and Flat file type messages.
Two schema envelopes (a parent and a child) are normally required to instruct BizTalk Server on how to split the file. However, using one schema is also possible.

To learn how to split a large file (also known as as de-batching) into smaller files, please refer to the BizTalk Server 2006 samples – Split File Pipelineexample. Even though the sample is based on BizTalk Server 2006 (R1) – it remains relevant for BizTalk Server 2006 R2, BizTalk Server 2009, and BizTalk Server 2010. 
Note: There are several examples on how to de-batch messages, such as de-batching from SQL Server, as communicated on Rahul Garg’s blog on de-batching.

After the large file has been split into smaller child record messages, child business processes can simultaneously work on each of these smaller files, which will eventually create child result messages. These child result messages can then be picked up by BizTalk Server and published to the BizTalk MessageBox database.

	Once the messages are placed into the MessageBox database, they can be moved into a temporary repository, such as a SQL Server database table.
Messages collected into the database repository, can be collated, and even sorted by a particular identifier, such as the parent (the original large batched message) GUID.

To aggregate the results back to the same batch, correlation has to be used. BizTalk Orchestration provides the ability to correlate. Aggregation with Orchestration is relatively easy to develop. Using Orchestration, however, may require more hardware resources (memory, CPU) than necessary. One Orchestration instantiation will be required per batch (so if there’s 1000 large batch messages, then there will be 1000 Orchestration instances). Orchestrations linger until the last result message has arrived (or it can time out from a planned exception handling). There is an alternative way to aggregate, without Orchestration, but correlation is still necessary and required to collate all the results into the same batch. The trick is to correlate the result messages without Orchestration.

Let’s examine how correlation without Orchestration can be achieved.

Paolo Salvatori, in his first blog entry http://blogs.msdn.com/paolos/archive/2008/07/21/msdn-blogs.aspx describes how to correlate without Orchestration, by using a two-way receive port. This is important to note, because the two-way receive port provides key information that can be leveraged, by promoting them to the context property bag.

These key properties are EpmRRCorrelationID ReqRespTransmitPipelineID IsRequestResponse Correlation Token RouteDirectToTP

Correlation without Orchestration is as easy as just promoting the right information to the context property bag. Of course, this information needs to be persisted, maintained, and returned with the result messages. However, this method only works with a two-way receive port. What about from a one-way receive port, such as from a file directory or MSMQ?

That is still possible because the de-batching (file splitting) mechanism of BizTalk Server provides a different set of compound key properties that can be used.

These are Seq# (sequence number of child message) UID (Identifier of parent message) isLast? (Boolean whether if the child message is the last in the sequence)

The Sequence number of the child message, and the Boolean isLast can be used to determined the total number of records within the large message. For example, if there are 30 messages in a large message, the 30th message will have a sequence number of 30 (sequencing starts at 1), and isLast Boolean value of True.

The final step is to aggregate all the result messages and ensure that they’re correctly collated into the same batch that their original source messages from from. The UID that was used for correlation, is the parent identifier that can be used to group these result messages into. An external database table can be used to temporarily store the incoming messages, and a single SQL Stored Procedure can be used to extrapolate these messages into a single output file.

• A result message is received by a FILE Receive Port
• This result message is stored in a SQL Server table. This is achieved by being routed to a SQL Send Port where the original message is wrapped into a sort of envelope by a map which uses a custom XSLT. The outbound message is an Updategram which calls a stored procedure to insert the message into the table.
• A SQL receive port calls a stored procedure to retrieve the records of a certain type from the custom table. This operation generates a single document.
• This document is routed to a File Send Port where the message is eventually transformed into another format.

This method of using a SQL aggregator was communicated in http://geekswithblogs.net/asmith/archive/2005/06/06/42281.aspx

Note that the XSLT can be eventually replaced by a custom pipeline component or directly by an envelope (in this case the send pipeline needs to contain the Xml Assembler component).

Sample code is provided courtesy of Ji Young Lee, of Microsoft (South) Korea. Uncompress the ZIP file with the directory structure intact on the C:\ root directory (i.e.: c:\QuocProj). :http://code.msdn.microsoft.com/AsynchAggregation

Acknowledgements to: Thiago Almeida (Datacom), Werner Van Huffel (Microsoft), Ji Young Lee (Microsoft), Paolo Salvatori (Microsoft)

Read more ...