At first it seemed easy, but it’s been a long distance hurdle race. At the end, I have solved the problems I’ve found and my painful experience can be useful for other people, so this is it.

First of all, my topology is the following: a client, in this case for testing purposes, my machine, which is running an ASP.NET web application and a NServiceBus host that processes all the messages. This has been configured following the official documentation and looking at the samples (check out the full NServiceBus source code, you’ll need it). It’s a bit messy but with some trial and error you can make sense of it. There are some obscure properties that have to be googled to know what they mean.

Processing the messages is not trivial for the bus host. Some of them attack the database and need to use MSDTC to manage the transactions. This means that MSDTC must be up and running without problems.

One trick I learned after a lot of researching is the Runner.exe tool from NServiceBus. This comes with NServiceBus source code (and I think that also with paid version) and ends up in build\tools\MsmqUtils if you build it with build.bat or build-net4.bat. Just open a command line as administrator and run:

Runner.exe /i

This will install MSDTC and configure it properly. I don’t think you’ll need anything else but check the MSDTC configuration in Component Services -> Computers -> My Computer -> Distributed Transaction Coordinator -> Local DTC. Right click to go to properties and ensure everything is configured like in this Troubleshooting DTC post.

At this point, I recommend you to disable the Windows Firewall on both machines, at least for domain network, just to be sure it’s not interfering and breaking the tests, we will solve this later.

Then, DTCPing is your friend. This small tool will check the DTC connectivity between two machines. It’s a bit awkward. You have to start the tool in both machines, then put the name of the other and click ping on each one. This will make a test and if everything goes well, you’ll see a detailed log without errors:

++++++++++++Validating Remote Computer Name++++++++++++
Please refer to following log file for details:
Invoking RPC method on REMOTEMACHINE
RPC test is successful
++++++++++++RPC test completed+++++++++++++++
++++++++++++Start DTC Binding Test +++++++++++++
Received reverse bind call from REMOTEMACHINE
++++++++++++DTC Binding Test END+++++++++++++
Please send following LOG to Microsoft for analysis:
	Partner LOG: REMOTEMACHINE5928.log
++++++++++++Start Reverse Bind Test+++++++++++++
Received Bind call from REMOTEMACHINE
Trying Reverse Bind to REMOTEMACHINE
++++++++++++Reverse Bind Test ENDED++++++++++

Here started my nightmare, I got the damned: RPC error 5 (Access denied). After hours googling for the problem I found this post about MSDTC problems (I was looking for RPC problems, which actually was the root cause). The cause is that remote RPC is disabled in Windows workstations by default.

I opened for the N-th time the registry and added the RPC key to HKLM\Software\Policies\Microsoft\Windows NT and the RestrictRemoteClients DWORD value to 0. Rebooted without much hope. Magically, DTCPing worked and life seemed better and brighter.

Then, the Windows Firewall must be configured. In Windows Server 2008 (or Windows 7, it’s the same interface), go to Control Panel -> System and Security -> Windows Firewall (or just execute firewall.cpl at the command prompt). Go to Advanced Configuration to open a new window. Go to inbound rules and look for the Distributed Transaction Coordinator. If they are disabled, enable them. On the workstation the firewall can be enabled. Beware that DTCPing may not work, because the Firewall is only allowing connections to/from svchost.exe and msdtc.exe (if you want it to work, add a rule for that process).