Biz Talk processing Large messages

Tuesday, July 5, 2011
What is a large message?

Unfortunately, the answer to this question isn't tied directly to a specific message size, but rather, depends on specific bottlenecks in your Microsoft BizTalk Server 2006 system. The problems associated with large messages can be divided into the following categories:

Out of memory errors :
Certain types of message processing - such as mapping, validation, and property promotion - load the entire message into memory. If the size of the message in memory exceeds available resources, then an out of memory error occurs. The size threshold for messages that fall into this category is much lower than the size threshold for messages that are not loaded into memory. For example, a 10 MB flat file that is parsed into XML and then mapped may grow by a factor of 10 or more to consume over 100 MB of memory, whereas a 100 MB XML document that is not parsed or mapped may actually only consume 1 MB of memory as it is streamed to the MessageBox database.

Performance problems for messages that are not loaded into memory :

Messages that are not required to be loaded into memory are streamed to the MessageBox database using the .NET XmlReader interface. While they are not subject to the size limitations on messages that must be loaded into memory, there are some important factors that impact how BizTalk Server processes messages that are streamed to the MessageBox database

Factors that affect the processing of large messages

The original message size, message format, and type of message processing all affect how BizTalk Server processes large messages.

Original message size :
The size of the message received by BizTalk Server is the most visible indication of how large the message will be when it is processed by BizTalk Server. The original size of a message has a much greater impact on performance if the entire message is loaded into memory than if the message is streamed to the MessageBox database.

Message format :
Messages are received into BizTalk Server in one of two primary formats: XML files or flat files.

XML files :

In order for BizTalk Server to perform any processing on a message other than pass through routing, the message must be in the XML file format. If the files to be processed are received in XML format then they will retain their original size for the most part.

Flat files :

Flat files must be parsed into an XML format before BizTalk Server can perform any processing other than pass through routing. Parsing a flat file into an XML file can greatly increase the size of the file. Flat files do not contain XML tags with descriptive information about their data. XML files, on the other hand, wrap all of their data in descriptive XML tags. In some scenarios parsing can increase the size of a flat file by a factor of 10 or more, depending on how much descriptive data is contained in the XML tags for the file.
Flat file documents wrapped in a single CDATA section node in an XML document :
This type of document is a combination of both XML and flat file and can be problematic because BizTalk Server must load the entire wrapped flat file document into memory before processing it.

Type of message processing :

In BizTalk Server, there are two types of message processing: Routing only and Mapping. The performance variables tied to the type of message processing that is performed are message size and whether the message is loaded into memory.

Routing only :

If BizTalk Server is only used only for routing messages based upon promoted message properties, then the message is streamed into the Messagebox database using the .NET XmlReader interface, and message parts are not individually loaded into memory. In this scenario, out of memory errors are not an issue and the primary consideration is the amount of time that is required to write very large messages (over 100 MB) into the Messagebox database. The BizTalk Server development team has successfully tested the processing of messages up to 1 GB in size when performing routing only.The primary factor that determines performance in this scenario is the Large message fragment size used to fragment the data into the database. The Large message fragment size is a configurable option on the BizTalk Group Properties configuration page and has a default value of 102400 bytes (100 KB). Increasing this value reduces the number of round trips required to stream a message into the MessageBox database.

Mapping :

Transforming a document with a map is a memory-intensive operation. When a document is transformed by a map, BizTalk Server passes the message stream to the .Net XslTransform class, which then loads the document into a .NET XPathDocument object for processing. Loading the document into the .NET XPathDocument can potentially expand the original file size in memory by a factor of 10 or more. This expansion may be more pronounced when mapping flat files because flat files must be parsed into XML before they can be transformed.BizTalk Server 2006 significantly improves memory management for large documents by implementing a configurable message size threshold for loading documents into memory during transforms. Any message with a size below this threshold is handled in memory; any message with a size above this threshold is buffered to the file system to reduce memory requirements. The default message size threshold is 1 MB.

Guidelines for processing large messages:

Follow these guidelines to improve performance when processing large messages in BizTalk Server:
Adjust the message size threshold above which documents are buffered to the file system during mapping. To modify the size threshold, create a DWORD value named TransformThreshold at the following location in the BizTalk Server registry:

HKLM\Software\Microsoft\BizTalk Server\3.0\Administration\TransformThreshold

After you have created this value, enter a decimal value with the number of bytes to set the new threshold to. For example, enter a decimal value of 2097152 to increase the message size threshold to 2 MB (from the default of 1 MB). Increase this value on systems with a large amount of available memory to improve throughput. Buffering documents to disk conserves memory at a cost to overall throughput.

By default, documents that are buffered to the file system during mapping are written to the %temp% directory of the BizTalk Server computer. Change the setting for the %temp% environment variable to a non-system disk to improve performance when buffering large messages to the file system during mapping

No comments:

Post a Comment

Post Your Comment...