Troubleshooting MSDTC, RPC and NServiceBus issues

At first it seemed easy, but it’s been a long distance hurdle race. At the end, I have solved the problems I’ve found and my painful experience can be useful for other people, so this is it.

NServiceBus old logo

First of all, my topology is the following: a client, in this case for testing purposes, my machine, which is running an ASP.NET web application and a NServiceBus host that processes all the messages. This has been configured following the official documentation and looking at the samples (check out the full NServiceBus source code, you’ll need it). It’s a bit messy but with some trial and error you can make sense of it. There are some obscure properties that have to be googled to know what they mean.

Processing the messages is not trivial for the bus host. Some of them attack the database and need to use MSDTC to manage the transactions. This means that MSDTC must be up and running without problems.

One trick I learned after a lot of researching is the Runner.exe tool from NServiceBus. This comes with NServiceBus source code (and I think that also with paid version) and ends up in build\tools\MsmqUtils if you build it with build.bat or build-net4.bat. Just open a command line as administrator and run:

Runner.exe /i

This will install MSDTC and configure it properly. I don’t think you’ll need anything else but check the MSDTC configuration in Component Services -> Computers -> My Computer -> Distributed Transaction Coordinator -> Local DTC. Right click to go to properties and ensure everything is configured like in this Troubleshooting DTC post.

At this point, I recommend you to disable the Windows Firewall on both machines, at least for domain network, just to be sure it’s not interfering and breaking the tests, we will solve this later.

Then, DTCPing is your friend. This small tool will check the DTC connectivity between two machines. It’s a bit awkward. You have to start the tool in both machines, then put the name of the other and click ping on each one. This will make a test and if everything goes well, you’ll see a detailed log without errors:

++++++++++++Validating Remote Computer Name++++++++++++
Please refer to following log file for details:
Invoking RPC method on REMOTEMACHINE
RPC test is successful
++++++++++++RPC test completed+++++++++++++++
++++++++++++Start DTC Binding Test +++++++++++++
Received reverse bind call from REMOTEMACHINE
++++++++++++DTC Binding Test END+++++++++++++
Please send following LOG to Microsoft for analysis:
	Partner LOG: REMOTEMACHINE5928.log
++++++++++++Start Reverse Bind Test+++++++++++++
Received Bind call from REMOTEMACHINE
Trying Reverse Bind to REMOTEMACHINE
++++++++++++Reverse Bind Test ENDED++++++++++

Here started my nightmare, I got the damned: RPC error 5 (Access denied). After hours googling for the problem I found this post about MSDTC problems (I was looking for RPC problems, which actually was the root cause). The cause is that remote RPC is disabled in Windows workstations by default.

I opened for the N-th time the registry and added the RPC key to HKLM\Software\Policies\Microsoft\Windows NT and the RestrictRemoteClients DWORD value to 0. Rebooted without much hope. Magically, DTCPing worked and life seemed better and brighter.

Then, the Windows Firewall must be configured. In Windows Server 2008 (or Windows 7, it’s the same interface), go to Control Panel -> System and Security -> Windows Firewall (or just execute firewall.cpl at the command prompt). Go to Advanced Configuration to open a new window. Go to inbound rules and look for the Distributed Transaction Coordinator. If they are disabled, enable them. On the workstation the firewall can be enabled. Beware that DTCPing may not work, because the Firewall is only allowing connections to/from svchost.exe and msdtc.exe (if you want it to work, add a rule for that process).

Related Posts

Bye bye Kurobox

A deserved farewell to a device that has worked flawlessly for so many years

DIY Arduino Christmas tree lights

A detailed explanation of my home made light switcher made with Arduino and a relay shield

Back to blogging

My statement on getting back to blogging. Will this go anywhere?

Lessons learned optimizing MySQL

A summary of some things I discovered while trying to optimize the performance of a production MySQL server.

Simple Mini Profiler Glimpse plugin

I just created my first Glimpse plugin, integration between Mininprofiler and Glimpse.

Carbon Copy Cloner saved my day

Just a gratitude post because this piece of software was extremely useful when others don't. And it was my birthday.

Using fail2ban with nginx in Debian

fail2ban helps you fight spam and bots but comes with an Apache sample. I converted it to handle Nginx information.

How msiinv saved my day

mssinv is a tool to manage MSI installed packages and I used it to track a problem with a package partially installed, that had to be removed with this tool

Install ASP.NET MVC 3 Manually

Due to some strange problems in my PC, I had to install this package manually and this is applicable to many other installers

Deployment web site Zip with MsBuild and TeamCity

I needed to make a Zip file of a web site and automate it with TeamCity. I used MsBuild and some obscure tricks