BizTalk: How To: FlatFile Batching and Promoted Properties

I recently had a bit of time to figure out answers to some questions about how BizTalk handles promoted values in batches. Specifically, do promoted values from the "envelope" transfer to the "body", and what if you want to interact with those values from an orchestration? I'll also show how to add the flat file header content to each debatched outbound message.

First step was to build working samples of both XML and flat-file debatching. Setting up an XML debatching scenario is fairly simple. First you build the "body" schema. Then you construct an Envelope schema. This is done by (a) marking the schema-level property called Envelope to true and (b) setting the Body XPath property of the root node to the XML element you wish to rip apart.

 

Now, I've demonstrated before that if you map fields in the header and body to the same Property Schema field, then the value in the header gets demoted down to each debatched message. However, what if you don't want to have a field set aside in the message body to hold header content values? So, I created a property schema and promoted two of the nodes from the header portion of the XML document. After deploying the solution, and feeding a batched up message into BizTalk, each message got debatched, AND, the promoted fields got copied down to each message. The documentation states this, but I had to see it.

 

Next, I wanted to see how it behaved with flat files. Given that we handle flat file debatching differently, I wanted to see if the promoted property behavior was the same. So, I used the Flat File Wizard to generate both a header and body schema. The debatching key in the body schema is to flip the repeating batch field element to Max Occurs equals 1.

 

Finally, I created the necessary receive pipeline to disassemble the flat file input. I pointed the Document Schema and Header Schema to the previously created flat file schemas. Also, I flipped the Preserve Header flag to true. This crams the header message into the context of each debatched message.

When I redeploy the solution (now with flat file schemas) and add new receive/send ports, drop the flat file in, I can see the header message sitting in the FlatFileHeader context value. You could use this data however you want. But, what if we want behavior just like the XML debatching example above? Specifically, distinct promoted values from the header? So, I went back and promoted the values in the flat file header schema. After redeployment, I can go back and see that indeed, I have not only the whole header in a single context field, BUT each promoted value is isolated as well.

Now that I've gotten this far, I can demonstrate adding the flat file header BACK to each outbound message. It's easy. Simply create a send pipeline with a flat file assembler component, and again choose the Document Schema AND add the Header Schema. What this does is take that header blob out of context and reapplies it to the outbound message automagically.

 

My last question was how do interject an orchestration into the mix. Clearly we are working with promoted fields that don't exist in the individual message. They've been snagged from the parent. So by default, you can't access these promoted values from within an orchestration. However, if you go to your property schema, and click on a given element, you'll see a field called Property Schema Base. This setting determines where BizTalk looks for the data in a promoted property. I flipped my value toMessageContextPropertyBase which means that the value will be looked for in context, not the message itself. After I change this flag, I can now see the property field in my orchestration.

 

What I wasn't sure of was how this solution would behave now. By flipping the value toMessageContextPropertyBase would I screw up the pipeline when it writes to context? I didn't think so, but wasn't 100% sure of what this would do. So, after dropping my XML batch message in, I checked the context of the message after it has passed through the orchestration (which changed the promoted property value to something else). Sure enough, the value in the outbound message was changed successfully.

Cool stuff. It would be easy then to change the promoted values inherited from the parent, and then send the message directly back to the MessageBox for additional content-based processing. So there you go, all sorts of fun with BizTalk batches.

post by Richard Seroter

Read more ...

Mapping Inbound XML Data in to Single Element / Node in BizTalk Mapper

post by : A Technical Perspective

Requirement:

We need to call an SQL Server stored procedure based on parameters specified in an incoming XML message. In fact, the XML document contained a set of unbounded records that each needed to be associated with a call to a stored procedure.I could have used the debatching capabilities of the XML Disassembler pipeline component and implemented an orchestration implementing a scatter-gather like pattern, but since this solution was part of an ESB Toolkit 2.0 itinerary, I wanted to keep it as simple as possible.
Microsoft SQL Server offers native capabilities to process XML data. For instance, it is possible to supply an appropriately formatted XML document directly to a stored procedure for processing, thanks the OPENXML and other various T-SQL constructs.
That’s why I opted to bypass any pre-processing on the supplied XML document and hand it over directly to the stored procedure. This stored procedure would, in turn, iterate over the different records and process them accordingly.

In order to do that, the contents of the XML document must be specified in a single argument to the stored procedure, like so.

Serializing the entire contents of an XML document with the BizTalk Mapper

In my solution, the stored procedure is invoked through the WCF-SQL adapter. Therefore, a special document that conforms to an adapter-specific generated schema must be sent to the corresponding solicit-response send port.

In order to map the entire contents of an XML subtree structure from the source schema to a single element in the target schema, one can use the Call Template functoid in a BizTalk map.

How does it Work?

In the Call Template functoid, notice the use of two different templates.

The first one, is the entry-point for the Call Template functoid when running the map. It declares a single parameter that corresponds to the node on the source document under which the entire structure must be serialized to the target node.

<!--Uncomment the following xslt for a sample Xslt Call Template
that creates a Field element whose value is the concatenatation of the
two inputs. Change the number of parameters of this template to be 
equal to the number of inputs connected to this functoid.-->

<xsl:template name="called-template">
  <xsl:param name="param1" />
  <xsl:element name="xmlDocument" namespace="http://schemas.microsoft.com/Sql/2008/05/TypedProcedures/dbo">
    <xsl:text disable-output-escaping="yes">&lt;![CDATA[</xsl:text>
    <xsl:call-template name="identity" />
    <xsl:text disable-output-escaping="yes">]]&gt;</xsl:text>
  </xsl:element>
</xsl:template>

First, the template produces an XML element, based upon its name and namespace as required in the target schema. In this case, the element corresponds to the name of the argument to the stored procedure as generated by the WCF-SQL adapter.

Notice that the serialized data is wrapped around a <!CDATA[]]> tag, so as to eliminate the need to escape XML-reserved characters.

The serialized data itself is produced by calling the second template. This one is none other than the identity template.

<xsl:template name="identity" match="@*|node()">
  <xsl:copy>
    <xsl:apply-templates select="@*|node()" />
  </xsl:copy>
</xsl:template>

This template iterates over all elements, texts, comments and processing instructions and recursively copies the contents to produce its output.

Read more ...

Advanced Users Biztalk WIKI : Useful How to Links Part - 3

Useful How to Links Part - 3

BizTalk Administrator's Checklist Compiled by Microsoft BizTalk Support

http://social.technet.microsoft.com/wiki/contents/articles/629.aspx?CommentPosted=true#commentmessage

Muenchian Grouping and Sorting in BizTalk Maps

http://blogs.msdn.com/b/chrisromp/archive/2008/07/31/muenchian-grouping-and-sorting-in-biztalk-maps.aspx

http://code.msdn.microsoft.com/windowsdesktop/Muenchian-Grouping-and-790347d2

Xslt Template StyleSheet For Biztalk Mapping : Query in Biztalk Map

http://social.msdn.microsoft.com/Forums/en-US/biztalkgeneral/thread/ecc69e42-f126-40b5-83c0-aefcfdf53a85

AppFabric-enabled WCF Data Service Walkthrough (C#)

http://blogs.msdn.com/b/rjacobs/archive/2010/06/30/appfabric-wcf-data-service-cs.aspx

Querying DataSets – Introduction to LINQ to DataSet

http://blogs.msdn.com/b/adonet/archive/2007/01/26/querying-datasets-introduction-to-linq-to-dataset.aspx

Accessing BRE RuleEngine Using .Net Framework

http://arnulfowing.blogspot.com/2007/09/how-to-access-bre-vocabularies-from-net.html

Few links that may help:

http://msdn.microsoft.com/en-us/library/aa995566.aspx                                                                

(MSDN: Walkthrough: Executing the Policy Programmatically)

http://technet.microsoft.com/en-us/library/microsoft.ruleengine.sqlrulestore_members.aspx (SQLRuleStore)

http://arnulfowing.blogspot.com/2008_07_01_archive.html

http://biztalkwithshashikant.blogspot.com/2007/08/implementing-business-rule-engine.html

http://msdn.microsoft.com/en-us/library/aa951249.aspx

iFactRetriever: http://msdn.microsoft.com/en-us/library/aa560118.aspx

http://talentedmonkeys.wordpress.com/2011/01/17/improved-method-for-calling-the-bre-from-net-applications/

http://talentedmonkeys.wordpress.com/2010/08/23/calling-the-bre-from-net-applications/

Read more ...

Advanced Users Biztalk WIKI : Useful How to Links Part -2

Useful How to Links Part -2

Implementing Biztalk Pipeline Trace

http://seroter.wordpress.com/2007/01/22/biztalk-application-tracing-part-i/

Biztalk Orchestration Tracing Using Biztalk Diagnostics Library

http://biztalkdiagnostics.codeplex.com/

http://www.codeplex.com/site/search?query=biztalk&ac=2

High Performance Message Transform Alternative to XslTransform vs Xsl 

CompileTransform

http://blogs.msdn.com/b/paolos/archive/2010/01/29/how-to-boost-message-transformations-using-the-xslcompiledtransform-class.aspx

Dynamic Mocking With N-Mock

http://msdn.microsoft.com/en-us/magazine/cc163904.aspx

Biztalk 2010 Documentation

http://www.microsoft.com/download/en/details.aspx?id=11503

Validating Incoming Xml using BRE

http://seroter.wordpress.com/2009/11/11/validating-incoming-data-using-the-biztalk-business-rules-engine/

BizTalk: Delivery Notification in Direct Send Ports

http://agilior.pt/blogs/bruno.camara/archive/2008/04/17/4344.aspx

BizTalk XSLT Reuse (xsl:include) by using MS Build

http://www.peterdol.nl/blog/biztalk-xslt-reuse-xslinclude-by-using-ms-build/

BizTalk Orchestrations XPath: Survival Guide http://social.technet.microsoft.com/wiki/contents/articles/6944.biztalk-orchestrations-xpath-survival-guide.aspx

BizTalk Server: Performance Tuning & Optimization

http://social.technet.microsoft.com/wiki/contents/articles/7801.biztalk-server-performance-tuning-optimization.aspx

http://social.technet.microsoft.com/wiki/contents/articles/5109.biztalk-databases-survival-guide.aspx

http://social.technet.microsoft.com/wiki/contents/articles/7253.biztalk-server-2010-database-biztalkdtadb.aspx

Read more ...

Using Table Looping Functoid in Biz talk

When dealing with existing systems, sometimes a challenge presents itself in the form of a flat file.   Trying to impose structure upon a flat file can be achieved, however, using the Table Looping and Table Extractor functoids.  Consider the following schemas:

Source schema:
 

Target schema:
 

One’s first attempt at a map to transform the source to destination might look something like this:
 

We’ll use the following input file to test the map.
 

The output isn’t quite what we were hoping for.
 

The Table Looping functoid is the key to what we’re trying to achieve.  Below is the map that uses the Table Looping and Table Extractor functoids to create the desired output.
 

The Borrower fields are used as inputs into the Table Looping functoid as well as some definitions about how many rows and columns there will be.
 

By opening up the Table Looping Grid, we’re able to define what fields will go into certain columns/rows:
 

The Table Extractor functoids are used to define which columns from the table map to use as inputs.  Each Table Extractor functoid corresponds to a column within the Table Looping Grid.
 

And finally the output from Table Looping functoid to the Borrower node dictates that a Borrower node be created for each row within the Table Looping Grid.  With that said, here’s the output from testing the second map:
 

 Source code for this example can be found here.

Read more ...

XSLT Custom Scripting Functoid - Dictionary Approach Repeating Record

Q:

I have a Repeating record Where it contains id and description i need to select  based on  the value of description and pass on value of id and description both of them
sample xml :
<applicablerecords>
  <id>10</id>
  <description>3months</description>
  <id>12</id>
  <description>6months</description>
   <id>20</id>
  <description>24months</description>
   <id>35</id>
  <description>18months</description>
   <id>45</id>
  <description>24months</description>
</applicablerecords>

SOl:

Use the following Custom Xslt on a scripting functoid or you can also use c# script to createa dictionry and copy all the elements in to it and then search into it


xslt approach:

<?xml version="1.0" encoding="ISO-8859-1"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:template match="/">
         <xsl:for-each select="applicablerecords[description='24months']">
             <xsl:if test= "position()=1">
               <xsl:value-of select="id"/></td>
               <xsl:value-of select="description"/></td>
             <xsl:if>
           </xsl:for-each>
    </xsl:template>
</xsl:stylesheet>





c# Approach:



public static string  populatevalues(string param1, string param2)
{
        system.collections.dictionary mydictionary = new system.collections.dictionary<String,string>;
        mydictionary.add(param1,param2);
}

use cumulative functoid to accumulate values thru records then

public static string  populatevalues(string param1, string param2)
{

        system.collections.dictionary mydictionary = new system.collections.dictionary<String,string>;
        if(mydictionary.containsvalue(param2))
               return mydictionary.id || mydictionary.description


}

Read more ...

Biztalk Maps Incrementing Variable Value in Multiple instances if Map| Orchestration Variable retriever Functoid

Q:
Hi All,

I have a following scenario.

I have a map which gets executed multiple times. I want to use one variable and its value which I should be able to increment for each execuation of map.

This can be done using orchestration paramaber or retaining value of static varible. (i am not sure).

This is urgent. Your help will be greatly appreciated.

Ans:

To pass a variable into a map you can either:

Send it in as part of the original XML input (naff if you don't want to start bolting fields onto your schema and injecting data)
Send it in as a second input message to your map (only possible if you use an Orchestration to call your map (or unless you're willing to write your own wrapper)).
Use a static variable in a .NET helper class to store your value (won't work if your map is running on different host instances)


Detail:

Source : Randal Van Spluttereen's Blog

You could try http://biztalkmessages.vansplunteren.net/2009/04/05/orchestration-variable-retriever-functoid-and-why-you-should-not-use-it

or

Add a .NET class library to your solution and create a class similar to this:

public class MapHelper {
	private static int _counter = 0;
	public static int GetNextCounter() {
		return ++_counter;
	}
}

Build and GAC your project, then use the Advanced Functoid shape to call an external .NET assembly, browse to your built library and select away.

The GetNextCounter() method will increment and return the next number.  You might want to expand this to provide some form of reset etc.

Remember, this is only available during the scope of the static object - so if you have any sort of complexity in your design or infrastructure you'll need to go to an external data store to ensure continuity.

source:  MSDN


This week I spend some time on writing a functoid that retrieves the value of a variable in an orchestration. Lets take a look on the functoid’s usage first.
Usage
This is the declaration of a string variable ‘lastName’ in a very simple test orchestration:

This is the expression shape where the value of that variable is set to my last name:

This is the map that is executed using a transform shape right after the expression shape above. The map contains the variable retriever functoid. It has one parameter that takes the name of the variable to fetch.
Please pay special attention to the icon because that bloody thing took me 50% of the development time. The result shows why I try to stay away from UI development as much as possible. 

Finally this is the Xml message returned from the orchestration via the file adapter.

Disadvantages
At first I was a little excited that I got this working. I did some testing with different orchestrations and it seems to work OK. After a while (and thinking this over) my excitement was tempered because I think the functoid has three big disadvantages:

1. Although questions related to this popup regularly in the BizTalk newsgroups I could not think of any real world examples. The sample above could also be implemented by using a message assignment shape after the map. In the message assignment shape the value of the variable can be assigned using xpath, properties or distinguished fields. The only way the functoid can be useful is when you need an orchestration variable value in a map to do some processing while the actual value is not mapped to the destination schema. But then again there are other ways to implement that. (Using a helper message and a multi message map). 
2.  The functoid code contains a considerable amount of reflection code. I didn’t do any performance tests but it is obvious that reflection comes with a cost. So in terms of performance it will probably be much better  to use alternative methods.
3.  This is probably not supported by MS. Mainly because it uses XLANG code which is normally hidden from the developers. 

These disadvantages make me conclude that this functoid is not very useful in real world scenarios. I really want to know what others think about this. So whether you agree or don’t agree please share your thoughts on this!
The other way around
Now that I figured out a way to access a variable it is a small step to take this a little further and build a functoid that WRITES the value of a variable in an orchestration. I didn’t implement such a functoid because of above mentioned points. I also think writing, as opposed to, reading is very tricky because you need to take things like serialization and locking into account.
If your still not convinced that you should not use this you can download the functoid “dll” from here.
Installation instructions:

• Orchestration variable retriever functoid (and why you should not use it)

• copy the .dll to the ‘Mapper Extensions’ folder which resides in the BizTalk installation folder.
• put the .dll in the gac.
• Open a map in Visual Studio, click right in the toolbox area and choose the functoids tab.
• Browse the the functoid dll in the ‘Mapper Extensions’ folder to add it to the toolbox.

The source is also available here. It is build using BizTalk 2006 R2.

Read more ...

Using the Value Mapping Functoid

The Value Mapping functoid requires two input parameters, and returns the value of the second input parameter if the value of the first parameter is "true". The following illustration shows a map with the Value Mapping functoid used in this way.

Click the illustration to enlarge or reduce.

To complete the map, you must set the input parameters for each functoid. The following illustration shows the property sheet for each of the three Value Mapping functoids.

Click the illustration to enlarge or reduce.

The following illustration shows the property sheet for the top Equal functoid.

Click the illustration to enlarge or reduce.

The middle Equal functoid property sheet is similar, but its second input parameter has a constant value of Y. The bottom Equal functoid property sheet is also similar, but its second input parameter has a constant value of Z.
You might have a source document instance that contains the following element.

<Field Name="X" Value="1"/>

A Name value of X makes the top Equal functoid of the map return a value of "true". The "true" value returned by the Equal functoid makes the Attribute Value functoid that it is linked to return a value of 1.
The following code is an example of document instance that corresponds to the source specification of the map.

<Root>
  <Record>
    <Field Name="X" Value="1"/>
    <Field Name="Y" Value="2"/>
    <Field Name="Z" Value="3"/>
  </Record>
  <Record>
    <Field Name="X" Value="4"/>
    <Field Name="Y" Value="5"/>
    <Field Name="Z" Value="6"/>
  </Record>
  <Record>
    <Field Name="X" Value="7"/>
    <Field Name="Y" Value="8"/>
    <Field Name="Z" Value="9"/>
  </Record>
</Root>

Using this map and this source document instance, BizTalk Server outputs the following document instance.

<Root>
<Record X="1"/>
<Record Y="2"/>
<Record Z="3"/>
<Record X="4"/>
<Record Y="5"/>
<Record Z="6"/>
<Record X="7"/>
<Record Y="8"/>
<Record Z="9"/>
</Root>

Important

• The Value Mapping functoid accepts Boolean input only in the form of the lowercase strings "true" and "false". For example, if a field in an incoming document instance has a value of "True" and is linked directly to the top input parameter of a Value Mapping functoid, the value of the second input parameter of the Value Mapping functoid is not passed to the output document

Read more ...

BizTalk Mapper Extensions UtilityPack updated for BizTalk Server 2010 on Code Plex

post by : sandro perira

“BizTalk Mapper Extensions UtilityPack” project (available on CodePlex) is now updated for BizTalk Server 2010 (sorry for the delay).

This is a simple migration of the project to 2010 and all functoids were tested in this environment. The project was originally published on 21 November 2010 and no new developments have been made ​​since then. This situation is about to change, I’m preparing a new version with a new set of functoids.

Project Description

BizTalk Mapper Extensions UtilityPack is a set of libraries with several useful functoids to include and use it in a map, which will provide an extension of BizTalk Mapper capabilities (more details of the project here).

Future developments (Quick overview):

• DateTime Format Convertions (actually this functoid is available in this release)
• Encryption Functoid
• Regular Expression Functoid
• Functoids to read configuration from config file or Registry Editor (regedit)
• 
  

Read more ...