Troubleshooting MSDTC, RPC and NServiceBus issues

At first it seemed easy, but it’s been a long distance hurdle race. At the end, I have solved the problems I’ve found and my painful experience can be useful for other people, so this is it.

First of all, my topology is the following: a client, in this case for testing purposes, my machine, which is running an ASP.NET web application and a NServiceBus host that processes all the messages. This has been configured following the official documentation and looking at the samples (check out the full NServiceBus source code, you’ll need it). It’s a bit messy but with some trial and error you can make sense of it. There are some obscure properties that have to be googled to know what they mean.

Processing the messages is not trivial for the bus host. Some of them attack the database and need to use MSDTC to manage the transactions. This means that MSDTC must be up and running without problems.

One trick I learned after a lot of researching is the Runner.exe tool from NServiceBus. This comes with NServiceBus source code (and I think that also with paid version) and ends up in build\tools\MsmqUtils if you build it with build.bat or build-net4.bat. Just open a command line as administrator and run:

Runner.exe /i

This will install MSDTC and configure it properly. I don’t think you’ll need anything else but check the MSDTC configuration in Component Services -> Computers -> My Computer -> Distributed Transaction Coordinator -> Local DTC. Right click to go to properties and ensure everything is configured like in this Troubleshooting DTC post.

At this point, I recommend you to disable the Windows Firewall on both machines, at least for domain network, just to be sure it’s not interfering and breaking the tests, we will solve this later.

Then, DTCPing is your friend. This small tool will check the DTC connectivity between two machines. It’s a bit awkward. You have to start the tool in both machines, then put the name of the other and click ping on each one. This will make a test and if everything goes well, you’ll see a detailed log without errors:

++++++++++++Validating Remote Computer Name++++++++++++
Please refer to following log file for details:
	C:\tools\DTCPing\LOCALMACHINE2928.log
Invoking RPC method on REMOTEMACHINE
RPC test is successful
++++++++++++RPC test completed+++++++++++++++
++++++++++++Start DTC Binding Test +++++++++++++
Trying Bind to REMOTEMACHINE
Received reverse bind call from REMOTEMACHINE
Binding success: LOCALMACHINE-->REMOTEMACHINE
++++++++++++DTC Binding Test END+++++++++++++
Please send following LOG to Microsoft for analysis:
	Partner LOG: REMOTEMACHINE5928.log
	My LOG: LOCALMACHINE2928.log
++++++++++++Start Reverse Bind Test+++++++++++++
Received Bind call from REMOTEMACHINE
Trying Reverse Bind to REMOTEMACHINE
Reverse Binding success: LOCALMACHINE-->REMOTEMACHINE
++++++++++++Reverse Bind Test ENDED++++++++++

Here started my nightmare, I got the damned: RPC error 5 (Access denied). After hours googling for the problem I found this post about MSDTC problems (I was looking for RPC problems, which actually was the root cause). The cause is that remote RPC is disabled in Windows workstations by default.

I opened for the N-th time the registry and added the RPC key to HKLM\Software\Policies\Microsoft\Windows NT and the RestrictRemoteClients DWORD value to 0. Rebooted without much hope. Magically, DTCPing worked and life seemed better and brighter.

Then, the Windows Firewall must be configured. In Windows Server 2008 (or Windows 7, it’s the same interface), go to Control Panel -> System and Security -> Windows Firewall (or just execute firewall.cpl at the command prompt). Go to Advanced Configuration to open a new window. Go to inbound rules and look for the Distributed Transaction Coordinator. If they are disabled, enable them. On the workstation the firewall can be enabled. Beware that DTCPing may not work, because the Firewall is only allowing connections to/from svchost.exe and msdtc.exe (if you want it to work, add a rule for that process).

Using fail2ban with nginx in Debian

Taking a look at the logwatch mails I see a common pattern of attacks, coming from China and trying to find details of my server configuration, which is something I dont like.

Looking around I found fail2ban which is a tool that does som regex matches on the server logs (sshd, httpd, authd, …) and takes proper actions, like banning the offending IP.

I then installed fail2ban in my Debian box:

> apt-get install fail2ban

Then, I took a look at /etc/fail2ban/jail.conf and found some entries for Apache but none for nginx, my server of choice, so I decided to create a jail.local to add some nginx stuff (this is recommended in Debian to allow hassle-free upgrades).

I started copying the Apache sections of the default fail2ban as the log files in my case use the same format that allows me to use Awstats easily. Then, I modified my log routes to point to the nginx ones and using Apache rules, if they don’t work I’ll tune them later.

[nginx]

enabled = true
port    = http,https
filter  = apache-auth
logpath = /var/log/nginx*/*error.log
maxretry = 6

[nginx-noscript]

enabled = false
port    = http,https
filter  = apache-noscript
logpath = /var/log/nginx*/*error.log
maxretry = 6

[nginx-overflows]

enabled = false
port    = http,https
filter  = apache-overflows
logpath = /var/log/nginx*/*error.log
maxretry = 2

Although this is ok, the bots I see don’t leave a trace in error.log but in access.log so I took a look at the filter.d folder where an interesting apache-badbots.conf was present. Then, I found the default fail2ban documentation in /usr/share/doc/fail2ban where there’s an usage example of the badbots script. I added I to my jail.local:

[apache-badbots]

enabled  = true
port    = http,http
filter   = apache-badbots
logpath  = /var/log/nginx*/*access.log
bantime  = 172800
maxretry = 1

Finally, I added this to the top of the file, to send mails to myself when a rule matches and a host is banned.

[DEFAULT]

action = %(action_mwl)s

Finally, restart the service and start receiving mails:

> sudo /etc/init.d/fail2ban restart

I’m sure this will need further adjustments, but it’s a beginning in my bot fighting war. I’ll make some updates as I find interesting results.

How msiinv saved my day

In my previous post I reported my problems installing MVC3 in my machine. That led me to further investigation as that worked perfectly in my co-workers’ machines.

I tried to repair Visual Studio 2010 and told me that it had troubles with vs_setup.msi… strange. Looking around I found a post that led me to an explanation of the msiinv.exe tool.

It took me a while to get the tool as the author’s site is broken, but Stackoverflow came on rescue.

Then, I ran the tool:

msiinv -p > msiinv.txt

Then opened the file and… surprise!

Microsoft Visual Studio 2010 Professional - ESN
    Product code:    {725041D1-9F45-30D3-A78E-DF08C4E1A297}
    Product state:    (1) The product is advertised, but not installed.
    Package code:    {9E58363D-E4A9-49D3-82B7-42584AFCA9E7}
    AssignmentType:    1
    Language:    3082
        Package:    vs_setup.msi
    0 patch packages.

Strange. I tried to fix it with MSI:

msiexec /f {725041D1-9F45-30D3-A78E-DF08C4E1A297}

It asked me for the vs_setup.msi, it’s in the root of the Visual Studio 2010 DVD in case it’s not found automatically. Magically it worked and now it’s installed:

Microsoft Visual Studio 2010 Professional - ESN
    Product code:    {725041D1-9F45-30D3-A78E-DF08C4E1A297}
    Product state:    (5) Installed.
    Package code:    {9E58363D-E4A9-49D3-82B7-42584AFCA9E7}
    Version:    10.0.30319
    AssignmentType:    1
    Publisher:    Microsoft Corporation
    Language:    3082
       Installed from: D:\VS2010\
    Package:    vs_setup.msi
    Help link:    http://go.microsoft.com/fwlink/?LinkId=133405
    Local package:    C:\Windows\Installer\3f3bc84.msi
    Install date:    2011\01\14
    0 patch packages.

Great! Now the ASP.NET MVC3 RTM setup package worked like a charm.