BizTalk Q & A: Process / Debatch Multiple Flat Files in a Single Receive Pipeline

Hi all, 

I have several different flat files I have to consume.  I would like to be able to drop them all into one receive location and use one flat file pipeline to disassemble them.  I created a receive pipeline and put in several flat file disassembler components, configuring each for one of the appropriate document schema.  I assumed one could do this from the language in BTS 2006 R2 help for the SchemaResolverComponent.  It reads: 

The flat file disassembler component normally requires you to define a parsing schema at design time. So if you expect to receive different flat file documents on the same receive location, you typically include several flat file disassemblers in the receive pipeline, one for each schema. At run time, the correct disassembler component is selected using a pipeline probing mechanism. However, this approach is expensive if you have many flat file schemas because probing for every corresponding disassembler component degrades pipeline performance.

 After deploying my pipeline, however, I got errors like the one below. 

Event Type: Error
Event Source: BizTalk Server 2006
Event Category: BizTalk Server 2006 
Event ID: 5753
Date:  7/12/2008
Time:  22:32:27
User:  N/A
Computer: 
Description:
A message received by adapter "FILE" on receive location "Receive Location_EMDEON.RPT4" with URI "D:\HIPAA_HOME\EMDEON_RPT4\*.*" is suspended. 
 Error details: An output message of the component "Flat file disassembler" in receive pipeline "HMOS.BizTalk.Pipelines.FF_ReceivePipeline.FlatFileReceivePipeline, HMOS.BizTalk.Pipelines.FF_ReceivePipeline, Version=1.0.0.0, Culture=neutral, PublicKeyToken=27ef7ae893a91467" is suspended due to the following error: 
     Unexpected data found while looking for:
'\n'
The current definition being parsed is Root. The stream offset where the error occured is 268. The line number where the error occured is 7. The column where the error occured is 12. .
 The sequence number of the suspended message is 1.  
 MessageId:  {F7B3BE2F-5504-40AE-8973-A08F073D6131}
 InstanceID: {3BB9BC41-2394-4C1B-A9F2-9DD97D7A7641}

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Evidently, the "pipeline probing mechanism" mentioned in BTS help is not part of the pipeline already?  How do I enable this one pipeline with multiple flat file dis-assembler components to be able to resolve which dis-assembler component to use for a given message?  (The SchemaResolverComponent in the SDK is one obvious answer, but I was really hoping to avoid it, if possible).

Sol:

I just tried to do the mimic you did

I created 3 – flat file schema  (delimited) with one receive pipeline with 3-dis-assemblers

It works fine with no extra configuration , the BTS will successfully recognize all message types

So I do not think your problem related to enable probing “it is already their”

But I notice the following points in this scenario

Lets say I have 3-flat file schema's (f1 , f2 , f3)

Where f3 instances also valid on f1 schema 

Pipeline dis-assembler process the list of dis-assembler in the same order you apply it on the design time

I created 2-pipeline with the following orders

Pipeline 1 (f1 , f2 , f3) this pipeline consider the f3 files as f1 files 

Pipeline 2 (f3, f2 , f1) this pipeline process all the messages successfully

Read more ...

FIX Biztalk Error: 'Microsoft.XLANGs.BaseTypes.BPELExportable': inconsistent duplicate module attribute

FIX Biztalk Error: 'Microsoft.XLANGs.BaseTypes.BPELExportable': inconsistent duplicate module attribute

While working on a solution we suddenly encountered the following error message while trying to build the code.

'Microsoft.XLANGs.BaseTypes.BPELExportable': inconsistent duplicate module attribute

It can be a nuisance to find the problem if you don't work with Biz Talk BPEL capabilities that often. It is however quite easy to fix. The error message is due to one orchestration having a different Module Exportable setting than the other orchestrations in the project. Making sure that the setting match between all orchestrations will make the error message disappear.

Read more ...

ROLLBACK FOR Store Procedure called From BizTalk || Implement Optional Transaction in StoreProcedure

A simple trick that you could try would be to test for an exisitance of a
transaction before starting a new one in the proc. But this would require
minor modifications to your stored procedure. For example:

 

At the top of the proc,

 

Declare @MyTran bit

 

-- If Caller Started the Transaction, let the caller handle it
-- otherwise control it in the proc.
If @@Trancount = 0
Begin
    Begin Transaction
    Set @MyTran = 1
End

 

At the bottom of your proc, to rollback or commit:

 

If @MyTran = 1 and @@Trancount > 0
 Begin
          Rollback Transaction -- Or Commit Transaction
 End

 

Read more ...

Composite Operations on BizTalk WCF- SQL Adapter - Calling Multiple Store Procedures

The new SQL Adapter in the WCF Adapter Pack 2.0 supports composite operations, that is performing multiple SQL operations in response to a single input message. The purpose of this blog post is to provide a walkthrough of how this works. I am using the public beta of BizTalk Server 2009, the public beta of the WCF Adapter Pack 2.0 and released version of SQL Server 2008. See below for an important caveat, as well as a link at the end of the post to my test solution.

To start, as with most things in BizTalk’s contract-first world, we need a schema. In order to do this, choose “add generated items” from a BizTalk project in Visual Studio. Then, choose “Consume Adapter Service”. If you don’t see that Visual Studio template, then you haven’t installed the Adapter Pack, as that’s where it comes from.

Next, select the sqlBinding specify a server, and press “Configure”. Set the client credential type to Windows (assume appropriate SQL login rights), and then on the URI tab, specified the server and database to use:

Af6ter doing this, press “Connect”, and the metadata will be populated.

For the purpose of this walkthrough, I have created 2 stored procedures: the first one inserts a record into a table, the second returns all rows in that table. Those are shown in the UI below.

Note that in the category we have “Procedures” and “Strongly-Typed Procedures”. The distinction is that “Procedures” will create un-typed schemas, whereas “Strongly-Typed Procedures” will generate schemas that you can work with inside BizTalk for mapping, promoting properties, etc.

The “Filename Prefix” will be used as a prefix for all the generated schemas.

After that was configured, I clicked OK and all the schemas were generated for me.

Next step is that you need to create a composite schema that will define the message you send to the adapter. I’m not quite sure why this one wasn’t generated for me, it’d be nice (hint hint), but it’s trivial to do.

How I did this for the walkthrough:

• create a new schema
• rename the root to SQLMsg (or whatever you like, this is unimportant)
• add a sibling record called SQLMsgResponse (this name does matter, it is the name of the request, with “Response” appended)
• add two child records under SQLMsg, and another two under SQLMsgResponse (names don’t matter, they’ll get renamed below)
• right-click the topmost “<schema>” node, and in the “Imports” property, add the “CompositeTypedProcedure.dbo.xsd” schema
• in the first child under SQLMsg, set the “Data Structure Type” property to InsertIntoDestination (this is a reference that you just imported above)
• in the second child under SQLMsg, set the “Data Structure Type” property to SelectAllDestination
• in the first child under SQLMsgResponse, set the “Data Structure Type” property to InsertIntoDestination (this is a reference that you just imported above)
• in the second child under SQLMsgResponse, set the “Data Structure Type” property to SelectAllDestinationResponse

Your schema should now look like this:

 

The, create an instance of the new composite schema to use as a test message, and populate the request. Here’s mine:

I then created a simple orchestration that would receive a request, call the adapter, and persist the response from the adapter. The request and response messages are of the type we just created in the composite schema:

Build and deploy the solution. After deploying it, note that there was a binding file generated along with the schemas, which is awesome, as this means you don’t need to manually create the send port. So, import the binding file which exists in you Visual Studio project.

HOWEVER… pretty big caveat here…. after importing the binding, you need to change the action mapping. If you use the default value, it will fail. You need to replace what is generated with the magic keyword “CompositeOperation”. This tells the adapter that it needs to call multiple operations, which it will resolve based on the schemas and namespaces. I believe the reason this works the way it does is that it allows you to import multiple operations in a single pass, and then use some subset of those operations in a composite operation, thereby enabling re-use of the generated schemas to potentially cover multiple different combinations of composite operations. Either way, watch out for this one. The error message tells you exactly what the problem is, however it won’t tell you about the keyword.

As an aside, and for the benefit of those who have not worked with this adapter yet, here are the binding configuration properties you have access to:

Then:

• create an inbound file drop location
• create an outbound file drop folder
• bind everything
• start the application
• drop your instance doc into the file drop location, triggering the orchestration

Lastly, here’s the output file:

In closing, I think this is an awesome new capability, and I am really liking the new SQL adapter. In case you haven’t heard, the old SQL adapter is being deprecated, so you really should be working with this one going forward.

Read more ...

Biz Talk : Dynamically creating receive locations

post by Brian Loesgon

Scenario:

There is a need to retrieve current tournament standings from the PGA Web site, pulling scores and statistics from them by FTP every 2 minutes while the tournaments were in progress. Ultimately, the standings would trickle through to a data-driven Web site. The interesting part here was the pickup location, there was a predefined FTP folder structure, with a separate folder for each tournament (the files were always the same name, but the locations would vary). We could have potentially multiple receive locations that become active during a given week, but only for a finite period of time, yet all pointing at different FTP locations. Biz Talk has dynamic send ports of course, but no dynamic receive locations. How to handle this? Clearly, although manually managing all this was possible, it really wasn't a viable solution given the amount of on-going effort it would require.

There is a overall tournament schedule file that will be posted on a weekly basis, and that's how the solution is driven. The solution basically:

• retrieves a schedule file (on a weekly basis) and publishes the schedule to the message box
• a CreateReveiveLocations orchestration subscribes to schedule files, and picks it up
• the orchestration maps the schedule to a structure that is easier to work with (we map inside the orchestration as we thought we may have other interesting things we could do with the schedule file in a future release)
• the orchestration removes any previously-created dynamic receive locations (based on a standard naming pattern, and they all belong to a known receive port)
• the orchestration calls a helper class that creates all the FTP receive locations. We know when the tournament starts, so:
  • we only create receive locations for current and future tournaments
  • we set the start date for the tournament to be the day of the tournament, so no polling will happen before the tournament starts

How cool is that? It's very cool to drop an XML document into BizTalk, and watch it create a bunch (and there are a LOT) of receive locations that it will subsequently use to drive a process. I'm all in favor of self-configuring environments!

We were in POC mode, and this took me a couple of days to get it all done (using good naming conventions, etc of course), and get it all working end-to-end. I'm sure I could have come up with a pretty good way to do this just in code, but it would have taken a LOT longer that 2 days.

Here's what the orchestration looks like:

 

There are two calls to a helper class above. The first removes any receive locations that were previously created (I use a naming convention and a unique name suffix, so I know which ones are dynamic). The second call passes in all the information required to create the new batch of receive locations.

The schedule file contained a tournament date. Because of that, we can skip creating FTP receives for tournaments that have ended.

Drilling into the CreateFtpReceiveLocation code, this snippet :

• first gets a reference to the receive port (which has a well know name)
• creates a new receive location
• sets the transport type to FTP

Note that if this were not a POC, I would have cached the ReceiveHandler and done other optimizations, however, even without optimizations, this creates hundreds of receive locations in seconds.

This last snippet:

• assigns the pipeline to use for this receive location
• set receive locations
• saves the changes to the BizTalk management database

There you have it. With a trivial amount of code, I eliminated what would have been significant ongoing manual intervention in the process. I fully automated what would have been a very tedious, and error-prone, task. The client was thrilled as this was a very elegant solution to an ongoing problem.

For anyone after the code rather that pretty pictures of it :), please see below.

-----------------------------

public static class LocationManagement
{
    const string CONNECTION_STRING = "Integrated Security=SSPI;Persist Security Info=False;Initial Catalog=BizTalkMgmtDb;Server=(local)";

    public static void CreateFtpReceiveLocations(string receivePortName, XmlDocument receiveLocations, string prefix, string server, string addressTemplate, string uid, string pwd)
    {
        XmlNodeList locs = receiveLocations.SelectNodes("//ReceiveLocation");
        DateTime cutoffDate = System.DateTime.Now.Add(new TimeSpan(-7,0,0,0));
        foreach (XmlNode loc in locs)
        {
            // if start date is more than 7 days ago, we don't need this one any more
            if (DateTime.Parse(loc.Attributes["StartDate"].Value) > cutoffDate)
            {
                CreateFtpReceiveLocation(receivePortName, prefix + loc.Attributes["Address"].Value, string.Format(addressTemplate, loc.Attributes["Address"].Value), server, uid, pwd, DateTime.Parse(loc.Attributes["StartDate"].Value));
            }
        }

    }

    public static bool CreateFtpReceiveLocation(string receivePortName, string receiveLocationName, string address, string server, string uid, string pwd, DateTime startDate)
    {
        try
        {
            BtsCatalogExplorer root = new BtsCatalogExplorer();
            root.ConnectionString = CONNECTION_STRING;

            //GetReceive Port
            ReceivePort receivePort = root.ReceivePorts[receivePortName];

            //Create a new receive location and add it to the receive port
            ReceiveLocation myreceiveLocation = receivePort.AddNewReceiveLocation(); ;
            myreceiveLocation.Address = address;
            myreceiveLocation.Name = receiveLocationName;

            //Receive Handler
            foreach (ReceiveHandler handler in root.ReceiveHandlers)
            {
                if (handler.TransportType.Name == "FTP")
                {
                    myreceiveLocation.ReceiveHandler = handler;
                    break;
                }
            }

            //Associate a transport protocol and URI with the receive location.
            ProtocolType protocol = root.ProtocolTypes["FTP"];
            myreceiveLocation.TransportType = protocol;

            Pipeline pipeline = root.Pipelines["Microsoft.BizTalk.DefaultPipelines.PassThruReceive"];
            myreceiveLocation.ReceivePipeline = pipeline;
            myreceiveLocation.StartDate = startDate;
            myreceiveLocation.StartDateEnabled = true;

            string ReceiveLocationTransportTypeData = "<CustomProps><AdapterConfig vt=\"8\">&lt;Config xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\"&gt;&lt;uri&gt;ftp://BTS2006Dev:21/ClientName.PGA/*.xml&lt;/uri&gt;&lt;serverAddress&gt;BTS2006Dev&lt;/serverAddress&gt;&lt;serverPort&gt;21&lt;/serverPort&gt;&lt;userName&gt;administrator&lt;/userName&gt;&lt;password&gt;******&lt;/password&gt;&lt;accountName/&gt;&lt;fileMask&gt;*.xml&lt;/fileMask&gt;&lt;targetFolder&gt;ClientName.PGA&lt;/targetFolder&gt;&lt;representationType&gt;binary&lt;/representationType&gt;&lt;maximumBatchSize&gt;0&lt;/maximumBatchSize&gt;&lt;maximumNumberOfFiles&gt;0&lt;/maximumNumberOfFiles&gt;&lt;passiveMode&gt;False&lt;/passiveMode&gt;&lt;firewallType&gt;NoFirewall&lt;/firewallType&gt;&lt;firewallPort&gt;21&lt;/firewallPort&gt;&lt;pollingUnitOfMeasure&gt;Seconds&lt;/pollingUnitOfMeasure&gt;&lt;pollingInterval&gt;60&lt;/pollingInterval&gt;&lt;errorThreshold&gt;10&lt;/errorThreshold&gt;&lt;maxFileSize&gt;100&lt;/maxFileSize&gt;&lt;/Config&gt;</AdapterConfig></CustomProps>";

            XmlDocument docTransportTypeData = new XmlDocument();
            docTransportTypeData.LoadXml(System.Web.HttpUtility.HtmlDecode(ReceiveLocationTransportTypeData));

            docTransportTypeData.SelectSingleNode("//userName").InnerText = uid;
            docTransportTypeData.SelectSingleNode("//password").InnerText = pwd;
            docTransportTypeData.SelectSingleNode("//uri").InnerText = address;
            docTransportTypeData.SelectSingleNode("//serverAddress").InnerText = server;

            myreceiveLocation.TransportTypeData = ReceiveLocationTransportTypeData;

            //Enable the receive location.
            myreceiveLocation.Enable = true;

            //Save changes
            root.SaveChanges();

            return true;
        }
        catch (Exception ex)
        {
            throw new Exception("Error creating receive location", ex);
        }
    }

Read more ...