Determining the optimum number of BizTalk Hosts

post by Marcel

This post is written to get an overview of the different aspects one should take into consideration when creating multiple BizTalk hosts. There is no one-size-fits-all optimum for each BizTalk project. Generally it is a good practice to have a separate host for Receiving, Sending, Processing and Tracking and then add additional hosts as machine resources allow and tuning and managing require.

Hosts
The BizTalk Host object represents a logical set of zero or more runtime processes in which you can deploy services, pipelines, and other artifacts. The Host object also represents a collection of runtime instances (zero or more) where the deployed items physically run.

After you create a host (a logical container), you can add physical BizTalk servers (host instances) to the host.

Items—such as adapter handlers, receive locations (including pipelines), and orchestrations—contained in BizTalk hosts can perform the following functions:

Receiving. These items do the initial processing of messages after they are picked up in a receive location. When a host contains a receiving item, such as a receive location or pipeline, it acts as a security boundary, and the message decoding and decrypting occurs in a pipeline within the host.
Sending. These items do the final processing of messages before they are sent out to the send port. When a host contains a sending item, such as a send port or pipeline, the host acts as a security boundary, and the message signing and encryption occurs in a pipeline within the host.
Processing. These items process messages based on the instructions in an orchestration.
One BizTalk Host can contain items that receive, send, and process messages. It is recommended that you create different hosts for each function to create security boundaries and facilitate management. In particular, it is recommended that you use different hosts for processing and for receive/send, and that you separate trusted and non-trusted items.



Separating Host Instances by Functionality

In addition to the high availability aspects of the host instance configuration, you should separate sending, receiving, processing, and tracking functionality into multiple hosts. This provides flexibility when configuring the workload in your BizTalk group and is the primary means of distributing processing across a BizTalk group. This also allows you to stop one host without affecting other hosts. For example, you may want to stop sending messages to let them queue up in the MessageBox database, while still allowing the inbound receiving of messages to occur. Also, orchestration and adapter functionality should be separated into different BizTalk Server hosts to minimize resource contention.

Separating host instances by functionality also provides the following benefits:

· Each host instance has its own set of resources such as memory, handles, and threads in the .NET thread pool.
· Multiple BizTalk Hosts will also reduce contention on the MessageBox database host queue tables since each host is assigned its own work queue tables in the MessageBox database.
· Throttling is implemented in BizTalk Server at the host level. This allows you to set different throttling characteristics for each host.
· Security is implemented at the host level; each host runs under a discrete Windows identity. This would allow you, for example, to give Host_A access to FileShare_B, while not allowing any of the other hosts to access the file share.

Redundancy
This host should be run on at least two computers running BizTalk Server (for redundancy in case one fails). Depending on the type of host either fail-over or clustering should be used.



Disadvantages of additional hosts
While there are benefits to creating additional host instances, there are also potential drawbacks if too many host instances are created. Each host instance is a Windows service (BTSNTSvc.exe), which generates additional load against the MessageBox database and consumes computer resources (such as CPU, memory, threads).

Several performance counters are reported per host.

Host consumes memory (at least 200MB (?) per host)

Polling receive adapters will poll for each host that they are enlisted in

SQL connections: Each host instance will poll the db at intervals of 500ms.

Too many hosts will:

· Cause a heavy load on the SQL Server and possibly will prevent processing of messages
· Consume too much memory on the BizTalk servers leading to a throttling situation (reduces performance)
· Too many receive hosts that poll the receive location will degrade the performance
· Too much granularity in performance counters. This has negative impact on the overall view an administrator has.

Tune the Host polling interval
BizTalk Server uses a polling mechanism to receive messages from its host queues in the MessageBox. The MaxReceiveInterval value in the adm_ServiceClass table of the BizTalk Management (BizTalkMgmtDb) database is the maximum value in milliseconds that each BizTalk host instance will wait until it polls the MessageBox. The adm_ServiceClass table contains a record for the following service types:

XLANG/S – for BizTalk orchestration host instances
Messaging InProcess – for in-process host instances
MSMQT – for MSMQT adapter host instances
Messaging Isolated – for out of process host instances, used by the HTTP, SOAP, and certain WCF receive adapter handler

By default, this value is set to 500 milliseconds, which is optimized for throughput rather than low-latency. In certain scenarios, latency can be improved by reducing this value.

Note
Changes to this value impact all instances of the associated service type, therefore, take care to evaluate the impact on all host instances before changing this value.
Note
This value is only used if the MessageBox has no remaining unprocessed messages. If there is a constant backlog of unprocessed messages in the MessageBox, BizTalk Server will attempt to process the messages without waiting on the polling delay. After all messages are processed, BizTalk Server will begin polling using the value specified for MaxReceiveInterval.
Note
In a BizTalk Server environment with a high ratio of host instances to MessageBox database instances, decreasing the value for MaxReceiveInterval may cause excessive CPU utilization on the SQL Server computer that houses the MessageBox database instance. For example, if the MaxReceiveInterval is decreased to a low value (< 100) in a BizTalk Server environment with a single MessageBox and > 50 host instances, CPU utilization on the SQL Server may climb above 50%. This phenomenon can occur because the overhead associated with continually polling host queues is significant. If you reduce MaxReceiveInterval to a value less than 100, you should also evaluate the impact that this has on your SQL Server computer’s CPU utilization.

Quote: “(having many BizTalk hosts) seems like a bad solution. They will most likely see serious performance issues in the SQL layer as each host instance is “polling” for work. I have seen situations where many (maybe around 40) host instances were creating so much chatter with the MsgBox database that even when NO messages were being processed the CPU on the SQL box was at about 60%. Now, the customer did have the MaxReceiveInterval turned way down as well.”



Configuring a Dedicated Tracking Host
BizTalk Server is optimized for throughput, so the main orchestration and messaging engines do not actually move messages directly to the BizTalk tracking or BAM databases, as this would divert these engines from their primary job of executing business processes. Instead, BizTalk Server leaves the messages in the MessageBox database and marks them as requiring a move to the BizTalk Tracking database. A background process (the tracking host) then moves the messages to the BizTalk Tracking and BAM databases.

Advantages of Using a Dedicated Tracking Host
A BizTalk Host that hosts tracking is responsible for moving the DTA and BAM tracking data from the MessageBox database to the BizTalk Tracking (DTA) and BAM Primary Import databases. This movement of tracking data has an impact on the performance of other BizTalk artifacts running in the same host that is hosting tracking. Thus, you should use a dedicated host that does nothing but host tracking.

Using a dedicated tracking host also allows you to stop other BizTalk hosts without interfering with BizTalk Server tracking.



Best practices
Make your plan on how to separate hosts beforehand and how to name them. It is not possible to rename hosts.

Create separate hosts for different functions.

· Receiving
· Sending
· Processing
· Tracking



Add hosts for protocol or application specific requirements.

· Prioritization
· Batching vs. messaging
· Low latency
· Protocols that require clustering
· Protocols that require scale-out
· Separate hosts for applications with known or expects problems
When problematic applications are in a separate host, crashes will not affect applications in other hosts.
· Licensing of adapters
Certain 3rd party adapters are licensed separate from BizTalk Server. Separating per host can be a cost-effective deployment scheme.



Resources
MSDN: Hosts
http://msdn.microsoft.com/en-us/library/aa578695.aspx

MSDN: Host instances
http://msdn.microsoft.com/en-us/library/aa560673.aspx

Microsoft BizTalk Server Operations Guide
http://msdn.microsoft.com/en-us/library/cc296643.aspxMicrosoft BizTalk Server Performance Optimization Guide
http://msdn.microsoft.com/en-us/library/cc558617.aspx