Pages

Showing posts with label Pipeline. Show all posts
Showing posts with label Pipeline. Show all posts

BizTalk : How To : Call a Web service Using Custom Pipeline in a Messaging Solution

Wednesday, April 10, 2013


In this article I'll explain how you can call a Web Service which requires multiple arguments using a Custom pipeline and a custom pipeline component in a messaging-only scenario without using any Orchestration.

Normally, when there is a requirement to call a web service from BizTalk, people tend to take the easy route of calling it via an Orchestration. When we do a web reference inside the orchestration, Orchestration does quite a lot of work for us. It creates all the required schemas, it creates all the required multipart messages, which will be passed to the web service as argument. It makes our life easier. But I guess like me, some of you out there might need to call the web service without using Orchestration. As shown in the above figure. I've one request-response HTTP receive port, and one Solicit response SOAP send port, through this I'm going to call a web service, which expects multiple argument (including one complex type) and return the result back to the caller (HTTP Response). Here are the steps: The attached sample file contains all the required file, I'm just going to explain the key factors in this article.
1. Web Service Definition:
[WebMethod]
public Person GetPersonInfo(Person person, string firstName, string secondName) {
//Some processing
return person;
}
2. Create a general custom pipeline component to construct the multipart message required for the Web Service call 
At run time SOAP Adapter in the send port will map the Biztalk multipart IBaseMessage to the Web Service argument based on the partName of IBaseMessage and argument names of the Webservice. The key factor is how we are going to construct the multipart message in the format required by the SOAP Adapter to make the WebService call. So, in this article we are going to create custom pipeline component which will construct the correct IBaseMessage required by the SOAP adapter based on the input message and some pipeline design time properties
The custom pipeline component we are going to use has 2 design time properties FirstName and SecondName, which will be passed as parameters to the web service (See webservice definition from Step 1). We'll pass the first webservice argument "Person" as the incoming message via HTTP receive port. The figure below show the custom design time properties configuration window within Biztalk Admin console. 
The code below is the snippet from the custom pipeline component (two important methods Execute and CreateMessage). The Execute method below without the first line of code will be equivalent to aPassThru pipeline component with default Biztalk IBaseMessage. 
#############################################################
public Microsoft.BizTalk.Message.Interop.IBaseMessage Execute(IPipelineContext pc, IBaseMessage inmsg)
{
IBaseMessage msg = CreateMessage(inmsg.BodyPart.GetOriginalDataStream(), pc.GetMessageFactory(),inmsg.Context);
return msg;
}
#############################################################
IBaseMessage CreateMessage(Stream s, IBaseMessageFactory msgFactory, IBaseMessageContext context)
{
IBaseMessage msg = msgFactory.CreateMessage();
IBaseMessagePart part = msgFactory.CreateMessagePart();
part.Data = s;
msg.AddPart("Person", part, true);
msg.Context = context;
//1st Part
IBaseMessagePart partFirstName = msgFactory.CreateMessagePart();
byte[] firstPart = System.Text.Encoding.UTF8.GetBytes(string.Format("<string>{0}</string>", _firstName));
partFirstName.Data = new MemoryStream(firstPart);
partFirstName.Charset = "utf-8"
partFirstName.ContentType = "text/xml"
msg.AddPart("firstName", partFirstName, false);
//2nd Part
IBaseMessagePart partSecondName = msgFactory.CreateMessagePart();
byte[] secondPart = System.Text.Encoding.UTF8.GetBytes(string.Format("<string>{0}</string>", _secondName));
partSecondName.Data = new MemoryStream(secondPart);
partSecondName.Charset = "utf-8"
partSecondName.ContentType = "text/xml"
msg.AddPart("secondName", partSecondName, false);
return msg;
}
#############################################################
Our user defined function CreateMessage will create the required BizTalk IBaseMessage as shown in the below figure
In the above code snippet, the important things to note are highlighted in RED. The incoming message ("Person") will go as the first part (BodyPart) of the IBaseMessage with the name "Person", and then we added two more addional parts "firstName" and "secondName" to the IBaseMessage with correct partNames inline with the web service arguments. The other important thing to note is how the basic data types gets serialized. In our example we got "<string>{0}</string>" as value for firstName and secondName, because they are of type string. If for example you got int as your argument then you need to create the part in the format <int>5</int>.
NOTE: See the web service signature defined in Step 1 for comparison
3. Create a Custom Receive Pipeline using the custom pipeline component
Create a new Biztalk Receive Pipeline and place the custom pipeline component we created in the "Decode" stage of the pipeline.
4. Configure the ports
As shown in our design diagram at the beginning we need 2 ports to send and receive the message, the attached sample file got a binding file, this section is just for explanation, doesn't explain in detail how to configure the ports. Make sure the URL are correct, both on Receive and Send side after importing the binding. You need to configure IIS as well to receive messages via HTTP, follow the link to configure IIS for HTTP receive  http://msdn2.microsoft.com/en-us/library/aa559072.aspx.
Two-Way HTTP Receive Port:
Solicit-Response SOAP Send Port:
We used the .NET Proxy class on our SOAP port to make the call.
Filter Condition on the Send Port
5. Post a Message.
I used WFetch to post the message to BizTalk. You can see on the result pane the request message is posted and you got the response back from the web service synchronously on a two way connection.
Troubleshooting:
Some of the common exceptions you'll see while calling a webservice via SOAP adapter is shown below (from HAT and eventviewer)
1. "Failed to retrieve the message part for parameter "firstName". "
2. "Failed to serialize the message part "firstName" into the type "String" using namespace "". Please ensure that the message part stream is created properly."
The reason for the first error message is due to wrongly named IBaseMessage partName. ReadSection 2 carefully to overcome this error.
The reason for the second error message is mainly due to some problem with serializing the IBaseMessage parts to the correct web service arguments. Best approach to overcome this error will be to build a .net console/windows application, add a web reference to the webservice and try to serialize each argument to the corresponding type. For example for this example you can try the following
FileStream fs = new FileStream(@"C:\Documents and Settings\SaravanaK\Desktop\FailedMessages\_Person.out",FileMode.Open,FileAccess.Read);
XmlSerializer serialise = new XmlSerializer(typeof(LH.WebReference.Person));
LH.WebReference.Person per = (LH.WebReference.Person)serialise.Deserialize(fs);
fs.Close();
fs = new FileStream(@"C:\Documents and Settings\SaravanaK\Desktop\FailedMessages\_secondName.out",FileMode.Open,FileAccess.Read);
serialise = new XmlSerializer(typeof(string));
string s2 = (string)serialise.Deserialize(fs);
fs.Close();
The files "_Person.out" and "_secondName.out" are saved from HAT tool. See the exception detail and fix the issue, it will be some namespace issue or data issue.
Read the readme.txt file inside to configure it. Will take approximately 5-20 minutes based on your BizTalk knowledge level.
Read more ...

BizTalk - Testing Pipeline Components Approaches

Friday, March 15, 2013

We will begin this post by discussing some of the traditional ways I have seen pipeline components tested, then continue to 2 more recent techniques which I believe to offer significant advantages.  To begin with the traditional techniques are: 
Traditional Approach 1 - Testing as part of a larger process
In this technique the Pipeline component is developed and then deployed along with a BizTalk solution.  Tests are then conducted against the overall process and it is assumed that if the end to end test is successful then the pipeline component has been adequately tested.
The key points about this approach are:
  • Often problems with the pipeline component are not detected during development because the end to end test does not cover all cases in the component
  • It is difficult to obtain code coverage information for the pipeline component
  • It is time consuming as it requires a deployment to BizTalk to be able to test
  • It is error prone because often you forget to restart host processes and think you havent fixed something that you really have
  • The component has limited reusability as it is only tested within the context of this process
Traditional Approach 2 - Using abstraction to make the component more testable
In this technique you find the developer has abstracted the logic using the facade pattern which the pipeline component then uses.  This means the code in the pipeline component is as simple as possible.  The more complex code is in other classes which do not depend on the BizTalk classes and interfaces such as IBaseMessage.  This in turn makes these classes easier to test outside BizTalk.
I think this pattern in general isnt a bad thing.  but I often see this technique used in conjunction with technique 1.  So we end up with a situation where the underlying classes are tested with unit tests and the pipeline component itself is assumed as tested as part of the larger process.  The key points of this technique are:
  • It is better than technique 1 as we are performing some unit tests which validates most of the functionality of the component before BizTalk becomes involved
  • We still cant test the pipeline component interface without having to deploy to BizTalk
  • Most of the other points from technique 1 still apply
Traditional Approach 3 - Using Pipeline.exe
I sometimes have seen the technique where a developer will create the pipeline component and then some pipelines.  The developer will then use Pipeline.exe to execute the test cases. 
The key points of this approach are as follows:
  • This does not require the artifacts to be deployed to BizTalk
  • It requires additional BizTalk pipelines to be defined to test the pipeline component
  • This tool needs to be used from the command line (although you could use the process object to call it from a C# test)
  • When using this approach you would probably want to validate the output document from the pipeline.exe call to ensure the message is as expected
  • You cant really interact with the message context before or after the test so this might limit your testing capability or require additional components needing to be added to the pipeline.
The challenge of the traditional approaches
The main challenge which limited the traditional approach to how you would test pipeline components was the ability for the developer to create and setup the IBaseMessage and IPipelineContext objects which you would then use for testing.  This resulted in the above 3 approaches being (in my opinion) the most popular way of testing pipeline components.
As result developers were often making their best effort at being able to test pipeline components but always knowing that they could only effectively test so much and there was always a reasonable chance that the component is going to have problems when used.
Newer Approachs
As with previous posts in this series I'm trying to encourage the following desired practices when testing:
  • We want to make testing the component relatively simple
  • We want to test the component as much as possible before we start using it in BizTalk
  • We want the tests to be automated and part of a continuous integration process
In order to implement the approach to testing pipeline components I would recommend either of the following 2 techniques (outlined below) when testing pipeline components.  Before I discuss the two techniques some background on the sample (available for download at the bottom of the article):
The sample contains a simple pipeline component which will read the input message using the XPathMutatorStream.  When it finds an element matching the desired XPath query the value from this element will be promoted to the File.ReceivedFileName promoted property.
The sample pipeline component is intended to be a fairly straightforward component which can be used to demonstrate how to test a component.  The following picture shows the main part of the pipeline component:
In the tests project there are 2 test classes each one demonstrating each technique.
Approach 1 - Testing with the Pipeline Component Test Library
The Pipeline Component Test Library has been around for a little while, but I dont think its used as much as it should be.  The library basically provides a simpler API to the PipelineObjects.dll which comes with the Pipeline.exe tool in the SDK.
This means you can interact with the Pipeline.exe type facilities in a simpler way directly from your C# test.  You can also access the message and its context much easier than you would be able to by using Pipeline.exe.  The following picture shows the code snippet which forms the test of the pipeline component using the pipeline component test library:
 
 In the test you can see you use the Pipeline Library to help tackle the key challenges of the IBaseMessage and the IPipelineContext.  In terms of the message you use the libraries helpers to create this message from an input document.  For the IPipelineContext this is handled by the library internally because you are creating a pipeline in code to execute the component in.
The advantages of this technique are:
  • You have full access to the proper IBaseMessage before the test.  This lets you remove the dependancy on things like components before yours in a pipeline or adapters because you can do things like set properties yourself.
  • The technique uses objects that a BizTalk person will be familiar with so the learning curve is not that steep
  • The tests can be developed very quickly
  • You control the pipeline so you can add additional components as required
With this technique it allows you to treat the pipeline component like a black box.  You put a message in and check the message and context that comes out. 
Useful Resources:
Some useful resources on this technique are:
Tomas Restrepo  - Creator of the Pipeline Component Test Library
Nick Heppleson - Has an article on how he tests his Message Archive component using this technique
Approach 2 - Testing with Rhino Mocks
In approach 2 i'm going to demonstrate how you can use a mocking framework to help you test the pipeline component.  In this example I am using Rhino Mocks.  In this technique you are basically defining a dynamic mock for the objects which will be used by the pipeline component.  On the mock objects you set expectations for what should happen each time a method is called on the mock object.  You then execute the pipeline component and then verify that all of the expectations happened as you planned.
The below code sample shows the equivelent test implemented using Rhino Mocks.
 

The advantage of this technique is:
  • It is a very powerful technique which gives you full control over pretty much all of the objects
  • It is a technique which is common to C# developers
  • Encourages the developer to think more about the component
  • Again this does not require the code to be deployed to BizTalk
This technique is much more white box, requiring the developer to have a much more intermate knowledge of what the component is doing when creating the test or as we are all test driven developers this makes you think a little harder about what the component does internally.   
Useful resources:
For more info on BizTalk and Rhino Mocks check out the following (click here)
Summary
 I think the key differences between the pipeline component test library and Rhino Mock techniques are as follows (i will refer to the pipeline component test library as PCTL):
  • The PCTL offers a technique which has a shallower learning curve and will be familiar to most BizTalk developers
  • Rhino mocks offers probably more control over things for very complicated tests
  • The PCTL is a much quicker way of developing tests, i find that using Rhino Mocks is quite time consuming in working out all of the expectations (especially when you are new to the technique)
  • It would be easier to refactor PCTL tests when there are changes to your component
  • In my opinion the PCTL just gives me a little more confidence than Rhino Mocks.  This is mainly because the test technique gives me the gut feeling that it is performing like how it will in BizTalk.  Where as with Rhino Mocks it sometimes feels that there is a bit of a gap between the mocking and what will happen when it is in BizTalk.  I dont really have any hard evidence to back this up but I think the fact that the tests themselves are that bit more complicated to write that they almost need testing in their own right.
So based on this article I would make the following recommendations for your approach to testing pipeline components:
  1. Use the traditional abstraction technique anyway as this is a pattern that can make your component simpler to understand and test
  2. As a default technique use the Pipeline Component Test Library
  3. When you have a special case or unusual component that has advanced testing requirements, compliment the Pipeline Component Test Library tests with ones which use Rhino Mocks to help you do those more advanced things
  4. Use a code coverage tool to ensure you dont miss any tests
  5. Remember to test more than just the core interface such as IComponent as the rest of the code needs testing too!
Read more ...

BizTalk Q & A: Convert pipeline message to XDocument

Thursday, December 27, 2012
Q:
How can I transform an  Microsoft.BizTalk.Message.Interop.IBaseMessaget to XDocument to make changes in the message. And how I canrecreate my message after changes ?publicMicrosoft.BizTalk.Message.Interop.IBaseMessageExecute(Microsoft.BizTalk.Component.Interop.IPipelineContextpc,                                                                      Microsoft.BizTalk.Message.Interop.
IBaseMessageinmsg)        {
returninmsg;        }
thank you
Sol:
public IBaseMessage Execute(IPipelineContext pContext, IBaseMessage pInMsg)
        {
            //To get Incoming message
            System.IO.Stream originalStream = pInMsg.BodyPart.GetOriginalDataStream();
            //Working with XDocument
            XDocument xDoc;
            using (XmlReader reader = XmlReader.Create(originalStream))
            {
                reader.MoveToContent();
                xDoc = XDocument.Load(reader);
            }
            
            // Returning stream
            byte[] output = System.Text.Encoding.ASCII.GetBytes(xDoc.ToString());
            MemoryStream memoryStream = new MemoryStream();
            memoryStream.Write(output, 0, output.Length);
            memoryStream.Position = 0;
            pInMsg.BodyPart.Data = memoryStream;
            return pInMsg;
        }

or
Converting it to XmlDocument:
// Load original message to XmlDocument
Stream originalMessage = inMsg.BodyPart.GetOriginalDataStream();
XmlDocument xmlDoc = new XmlDocument();
xmlDoc.Load(originalMessage);
//
//Set output message
IBaseMessage outMsg = pipelineContext.GetMessageFactory().CreateMessage();
outMsg.AddPart("Body", pipelineContext.GetMessageFactory().CreateMessagePart(), true);
Stream outputStream = new MemoryStream();
xmlDoc.Save(outputStream);
outputStream.Position = 0;
outMsg.BodyPart.Data = outputStream;
return outMsg;

For additional information, check the following links
Read more ...

BizTalk EDI Error SOLVED : There was a failure executing the receive pipeline: Microsoft.BizTalk.Edi.DefaultPipelines.EdiReceive Reason: No Disassemble stage components can recognize the data.

Wednesday, December 26, 2012

In my last two posts I showed two possible errors, and their respective solutions, that can happen when we are validating dummy EDI message, provided by our partners, against the schema using Visual Studio, but these errors could also happen in runtime.
In this post I will show you one common error when we are validating an EDI solution.
After I deploy and correctly configured the solution, I was trying to receive an EDI document from a Receive Port, in order to convert it to XML format using the generic EDI pipeline: “Microsoft.BizTalk.Edi.DefaultPipelines.EdiReceive”, but I was always getting this error:
“There was a failure executing the receive pipeline:
“Microsoft.BizTalk.Edi.DefaultPipelines.EdiReceive,
Microsoft.BizTalk.Edi.EdiPipelines, Version=3.0.1.0, Culture=neutral,
PublicKeyToken1bf3856ad364e35″ Source: “EDI disassembler” Receive Port: “IN_ORDER_PORT” URI: “E:\PORTS\EDI\IN_ORDER\*.*” Reason: No Disassemble stage components can recognize the data.“.
The message provided by our partner was this:
01UNH+1000100+ORDERS:D:93A:UN:EAN007'
02BGM+220+01521710'
03DTM+137:120530:101'
04DTM+64:120604:101'
05DTM+63:120004:101'
06FTX+AAI+++SOME TEXT'
07FTX+AAI+++SOME TEXT'
08FTX+AAI+++SOME TEXT'
09NAD+BY+8000000001164::9'
10NAD+DP+8000000009463::9'
11NAD+IV+8000000013002::9'
12NAD+SU+8000001459008::9'
13NAD+PR+8000000016003::9'
14LIN+1++4001518722937:EN'
15PIA+1+00:PV+14001518722937:EN'
16IMD+F+M+:::SOME DESCRIPTION'
17QTY+21:52'
18LIN+2++5000014010034:EN'
19PIA+1+00:PV+15701014010031:EN'
20IMD+F+M+:::SOME DESCRIPTION'
21QTY+21:152'
22LIN+3++5000014016142:EN'
23PIA+1+00:PV+15701014016149:EN'
24IMD+F+M+:::SOME DESCRIPTION'
25QTY+21:304'
26LIN+4++5006879009752:EN'
27PIA+1+00:PV+15776879009759:EN'
28IMD+F+M+:::SOME DESCRIPTION'
29QTY+21:720'
30UNS+S'
31UNT+31+1000100'
CAUSE
After some time trying to understand the reason of this problem, I realize that’s nothing wrong with my project, the problem is actually in the message that I’m using.
This error occurs because the message does not contain the EDIFACT interchange envelope segments (UNB and UNZ). An EDIFACT interchange begins with a UNB. It contains version release information, syntax information, and partner information. An interchange ends with a UNZ.
UNA Segment: The UNA segment is optional in an EDIFACT interchange. The specifications in the UNA segment define the characters used as separators and indicators for the interchange. Use this segment only if the interchange contains non-standard separator characters. (More information here)
UNB Segment: The UNB segment is compulsory to an EDIFACT interchange. This segment acts as the interchange header for a set of EDIFACT documents. The UNB segment elements identify the sender and recipient of the interchange, together with the date and time that the interchange was prepared and the agency controlling the syntax of the interchange. (More information here)
UNZ Segment: This segment is the Interchange Trailer segment of an EDIFACT document. This segment indicates the end of an interchange and checks the interchange reference and number of documents in the interchange. (More information here)
SOLUTION
Add this segment to your message: UNA (optional), UNB and UNZ and the message will be processed correctly.
Correct message:
01UNA:+,?*'
02UNB+UNOB:1+UNB2.1+UNB3.1+012301:0123+UNB5'
03UNH+1000100+ORDERS:D:93A:UN:EAN007'
04BGM+220+01521710'
05DTM+137:120530:101'
06DTM+64:120604:101'
07DTM+63:120004:101'
08FTX+AAI+++SOME TEXT'
09FTX+AAI+++SOME TEXT'
10FTX+AAI+++SOME TEXT'
11NAD+BY+8000000001164::9'
12NAD+DP+8000000009463::9'
13NAD+IV+8000000013002::9'
14NAD+SU+8000001459008::9'
15NAD+PR+8000000016003::9'
16LIN+1++4001518722937:EN'
17PIA+1+00:PV+14001518722937:EN'
18IMD+F+M+:::SOME DESCRIPTION'
19QTY+21:52'
20LIN+2++5000014010034:EN'
21PIA+1+00:PV+15701014010031:EN'
22IMD+F+M+:::SOME DESCRIPTION'
23QTY+21:152'
24LIN+3++5000014016142:EN'
25PIA+1+00:PV+15701014016149:EN'
26IMD+F+M+:::SOME DESCRIPTION'
27QTY+21:304'
28LIN+4++5006879009752:EN'
29PIA+1+00:PV+15776879009759:EN'
30IMD+F+M+:::SOME DESCRIPTION'
31QTY+21:720'
32UNS+S'
33UNT+31+1000100'
34UNZ+1+UNB5'
Read more ...