.net - WCF is slow when reliable session is ON and with burst async request

Question

For experiments, I created a simple "Hello World" WCF service and client using .NET 4.5 on VS2012. The server is hosted on a console application and use net.tcp binding. I wrote the client code to send a burst of async requests (a total of 700 requests) to the server. Everything went fine until I turned ON the Reliable session feature of the service. After it is ON, the service suddenly ran very slow and it took almost a minute on my machine to complete the 700 requests. I tried to fine tune the Concurrency and Throttling parameters (see below) but it didn't help.

Does anyone know why this happen ? Is there anyway to avoid this ?

The slowness didn't happen if I turned OFF the Reliable session feature, or if I made the service call synchronous. So I think it may relate to the way WCF handles pending requests in WS-ReliableMessaging mode.

EDIT: Also this didn't happen when I chaned netTcpBinding to wsHttpBinding. This is very weird because in this case wsHttpBinding is much faster than netTcpBinding.

EDIT: Running Perfmon.exe on the server side shows that the "Thread Count" gradually increase from 8 to beyond 100 in the above case.

EDIT: Some measured throughput on my PC (local network). See that the performance of case 1 is very sluggish and practically useless.

1. Async + NetTcpBinding/Reliable throughput -> approx. 14 call/s (70 ms/call)
2. Async + WsHttp/Reliable throughput -> 7957 call/s (0.12 ms/call)
3. Sync + NetTcpBinding/Reliable throughtput -> 3986 call/s (0.25 ms/call)

Below are the codes and configuration for the server and client I used in the experiments.

Server:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

using System.ServiceModel;
using System.ServiceModel.Description;

[ServiceContract]
public interface IHelloService
{
    [OperationContract(IsOneWay=false)]
    string SayHello(string name);
}

[ServiceBehavior(ConcurrencyMode=ConcurrencyMode.Multiple, InstanceContextMode=InstanceContextMode.PerSession)]
public class HelloService : IHelloService
{
    public string SayHello(string name) {
        String s = string.Format("Hello {0}", name); 
        return s; 
    }
}

namespace WcfServer
{
    class Program
    {
        static void Main(string[] args)
        {
            Uri baseAddress = new Uri("net.tcp://localhost:8080/hello");
            using (ServiceHost host = new ServiceHost(typeof(HelloService), baseAddress)){
                // Open and listen
                host.Open();
                Console.WriteLine("The service is ready at {0}", baseAddress);
                Console.WriteLine("Press <Enter> to stop the service.");
                Console.ReadLine();
                // Close the ServiceHost.
                host.Close();
            }
        }
    }
}

Client:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

using System.ServiceModel;
using WcfClient.WcfServer;

namespace WcfClient
{
    class Program
    {
        static async Task PrintNameAsync(HelloServiceClient client, int cnt) {
            string s = await client.SayHelloAsync(string.Format("-- {0} --", cnt));
            Console.WriteLine(s);
        }

        static void Main(string[] args)
        {
            HelloServiceClient client = new HelloServiceClient("HelloService", "net.tcp://10.20.61.13:8080/hello");
            List<Task> tasks = new List<Task>();
            for(int i=0; i < 700; i++){
                Task t = PrintNameAsync(client, i);
                tasks.Add(t);
            }
            Task.WhenAll(tasks).Wait();
            client.Close();
        }
    }
}

Server's App.config:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
    <startup> 
        <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5" />
    </startup>
    <system.serviceModel>
        <bindings>
            <netTcpBinding>
                <binding name="HelloServiceBinding">
                    <reliableSession ordered="true" enabled="true" />
                    <security mode="None" />
                </binding>
            </netTcpBinding>
        </bindings>
        <behaviors>
            <serviceBehaviors>
                <behavior name="HelloServiceBehavior">
                    <serviceMetadata policyVersion="Policy15" />
                    <serviceDebug includeExceptionDetailInFaults="true" />
                    <serviceThrottling maxConcurrentCalls="1000" maxConcurrentSessions="1000"
                        maxConcurrentInstances="1000" />
                </behavior>
            </serviceBehaviors>
        </behaviors>
        <services>
            <service behaviorConfiguration="HelloServiceBehavior" name="HelloService">
                <endpoint address="net.tcp://localhost:8080/hello" binding="netTcpBinding"
                    bindingConfiguration="HelloServiceBinding" name="HelloService" contract="IHelloService" />
                <endpoint address="mex" binding="mexTcpBinding" contract="IMetadataExchange" />
            </service>
        </services>
    </system.serviceModel>
</configuration>

Client's App.config:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
    <startup> 
        <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5" />
    </startup>
    <system.serviceModel>
        <bindings>
            <netTcpBinding>
                <binding name="HelloServiceBinding" sendTimeout="00:01:00">
                    <reliableSession enabled="true" />
                    <security mode="None" />
                </binding>
            </netTcpBinding>
        </bindings>
        <client>
            <endpoint address="net.tcp://localhost:8080/hello" binding="netTcpBinding"
                bindingConfiguration="HelloServiceBinding" contract="WcfServer.IHelloService"
                name="HelloService">
            </endpoint>
        </client>
    </system.serviceModel>
</configuration>

Answer 1

Found a partial workaround for the problem from the below links:

• http://blogs.msdn.com/b/endpoint/archive/2011/05/04/wcf-scales-up-slowly-with-bursts-of-work.aspx
• http://support.microsoft.com/kb/2538826

With the workaround (using the WorkerThreadPoolBehavior), the measured throughputs are as follows:

1. Async + NetTcpBinding/Reliable throughput -> 474 call/s (2.1 ms/call) ... improved but not satisfactorily
2. Async + WsHttp/Reliable throughput -> 7856 call/s (0.13 ms/call) ... no change
3. Sync + NetTcpBinding/Reliable throughtput -> 2110 call/s 0.47 ms/call) ... degraded

Note that the case 1 above is improved significantly from 70 ms/call. However, it still lags from case 2. And for case 3, introducing WorkerThreadPool behavior cause performance degradation from 0.25 ms/call to 0.47 ms/call.