This week I’ve been quite entertained at work optimizing a big MySQL database, 18M rows and about 10Gb of table data for a high traffic web site (11M unique views per month). Not optimizing for perfomance is overkill in this case, so I have been learning and enjoying a lot during these days.

The first thing to note is that database servers are complex beasts (don’t underestimate them) and each and every database provider has its own strengths but also its own caveats.

Sooner or later you’ll face a behavior that may seem counter-intuitive and there’s when you must read a lot of documentation. In this case, MySQL Performance Blog and MySQL documentation have been of great help, as well as the now unmantained-but-still-useful Hack MySQL site, which provides great tips very well explained.

Second. As always, doing an 80% optimization consumes 20% of the time and most of it is quite straightforward. EXPLAIN, review results, create index, EXPLAIN again. This way you can do a good job just knowing some indexing basics. Your indexes may be a bit fat, but will do the job.

Then you’ll need to be patient, creating indexes is not always fast, in fact the number, size and heterogeneusy of the columns involved will increase the time it takes to create them. While you wait, the best is to read good books about MySQL or relational databases like: Relational Database Index Design and the Optimizers.

You’ll need to be organized. Creating indexes randomly just looking a couple of slow SELECTs is something, but you must look at the big picture: make an inventory of all the application queries, no matter how unimportant look like because if they attack a big table with a WHERE or an ORDER BY they will perform bad. Fact. This list will help you organize the indexes and see which ones are used and which not and where.

Learn as well what the configuration parameters mean, MySQL has a bunch of them. Just increasing numbers will not always solve your bottlenecks. If you see something strange, there might be a parameter misconfigured and causing bad results.

Sometimes you’ll find yourself frustrated by something that you can’t understand. I found that almost anything in this field has an explanation, even a reasonable one.

Finally, don’t overlook other thinks. Database performance is important but there may be a lot of room to improve in your code, don’t blame MySQL as the root of all problems!

I recently published on Github a simple small project (miniprofiler-glimpse-plugin) to integrate the results collected by the excellent MVC Mini Profiler created by the people at Stackoverflow with Glimpse, the Firebug for ASP.NET and MVC applications.

Creating Glimpse plugins is really easy (just implement IGlimpsePlugin and decorate it with GlimpsePluginAttribute), implement the methods and reference the plug-in assembly in the project where you are using Glimpse. MEF will do the gluing and the JSON.NET will do its best to deserialize the results in a human readable way.

To use this plug-in, just compile it and reference the assembly in your Glimpse-enabled project. I you have questions, suggestions or bug reports, you can use Github’s own issue reporting system, drop me a line o fork the project and make your improvements.

So, I hope you like it.

Link: miniprofiler-glimpse-plugin [github.org]

At first it seemed easy, but it’s been a long distance hurdle race. At the end, I have solved the problems I’ve found and my painful experience can be useful for other people, so this is it.

First of all, my topology is the following: a client, in this case for testing purposes, my machine, which is running an ASP.NET web application and a NServiceBus host that processes all the messages. This has been configured following the official documentation and looking at the samples (check out the full NServiceBus source code, you’ll need it). It’s a bit messy but with some trial and error you can make sense of it. There are some obscure properties that have to be googled to know what they mean.

Processing the messages is not trivial for the bus host. Some of them attack the database and need to use MSDTC to manage the transactions. This means that MSDTC must be up and running without problems.

One trick I learned after a lot of researching is the Runner.exe tool from NServiceBus. This comes with NServiceBus source code (and I think that also with paid version) and ends up in build\tools\MsmqUtils if you build it with build.bat or build-net4.bat. Just open a command line as administrator and run:

Runner.exe /i

This will install MSDTC and configure it properly. I don’t think you’ll need anything else but check the MSDTC configuration in Component Services -> Computers -> My Computer -> Distributed Transaction Coordinator -> Local DTC. Right click to go to properties and ensure everything is configured like in this Troubleshooting DTC post.

At this point, I recommend you to disable the Windows Firewall on both machines, at least for domain network, just to be sure it’s not interfering and breaking the tests, we will solve this later.

Then, DTCPing is your friend. This small tool will check the DTC connectivity between two machines. It’s a bit awkward. You have to start the tool in both machines, then put the name of the other and click ping on each one. This will make a test and if everything goes well, you’ll see a detailed log without errors:

++++++++++++Validating Remote Computer Name++++++++++++
Please refer to following log file for details:
	C:\tools\DTCPing\LOCALMACHINE2928.log
Invoking RPC method on REMOTEMACHINE
RPC test is successful
++++++++++++RPC test completed+++++++++++++++
++++++++++++Start DTC Binding Test +++++++++++++
Trying Bind to REMOTEMACHINE
Received reverse bind call from REMOTEMACHINE
Binding success: LOCALMACHINE-->REMOTEMACHINE
++++++++++++DTC Binding Test END+++++++++++++
Please send following LOG to Microsoft for analysis:
	Partner LOG: REMOTEMACHINE5928.log
	My LOG: LOCALMACHINE2928.log
++++++++++++Start Reverse Bind Test+++++++++++++
Received Bind call from REMOTEMACHINE
Trying Reverse Bind to REMOTEMACHINE
Reverse Binding success: LOCALMACHINE-->REMOTEMACHINE
++++++++++++Reverse Bind Test ENDED++++++++++

Here started my nightmare, I got the damned: RPC error 5 (Access denied). After hours googling for the problem I found this post about MSDTC problems (I was looking for RPC problems, which actually was the root cause). The cause is that remote RPC is disabled in Windows workstations by default.

I opened for the N-th time the registry and added the RPC key to HKLM\Software\Policies\Microsoft\Windows NT and the RestrictRemoteClients DWORD value to 0. Rebooted without much hope. Magically, DTCPing worked and life seemed better and brighter.

Then, the Windows Firewall must be configured. In Windows Server 2008 (or Windows 7, it’s the same interface), go to Control Panel -> System and Security -> Windows Firewall (or just execute firewall.cpl at the command prompt). Go to Advanced Configuration to open a new window. Go to inbound rules and look for the Distributed Transaction Coordinator. If they are disabled, enable them. On the workstation the firewall can be enabled. Beware that DTCPing may not work, because the Firewall is only allowing connections to/from svchost.exe and msdtc.exe (if you want it to work, add a rule for that process).

In my previous post I reported my problems installing MVC3 in my machine. That led me to further investigation as that worked perfectly in my co-workers’ machines.

I tried to repair Visual Studio 2010 and told me that it had troubles with vs_setup.msi… strange. Looking around I found a post that led me to an explanation of the msiinv.exe tool.

It took me a while to get the tool as the author’s site is broken, but Stackoverflow came on rescue.

Then, I ran the tool:

msiinv -p > msiinv.txt

Then opened the file and… surprise!

Microsoft Visual Studio 2010 Professional - ESN
    Product code:    {725041D1-9F45-30D3-A78E-DF08C4E1A297}
    Product state:    (1) The product is advertised, but not installed.
    Package code:    {9E58363D-E4A9-49D3-82B7-42584AFCA9E7}
    AssignmentType:    1
    Language:    3082
        Package:    vs_setup.msi
    0 patch packages.

Strange. I tried to fix it with MSI:

msiexec /f {725041D1-9F45-30D3-A78E-DF08C4E1A297}

It asked me for the vs_setup.msi, it’s in the root of the Visual Studio 2010 DVD in case it’s not found automatically. Magically it worked and now it’s installed:

Microsoft Visual Studio 2010 Professional - ESN
    Product code:    {725041D1-9F45-30D3-A78E-DF08C4E1A297}
    Product state:    (5) Installed.
    Package code:    {9E58363D-E4A9-49D3-82B7-42584AFCA9E7}
    Version:    10.0.30319
    AssignmentType:    1
    Publisher:    Microsoft Corporation
    Language:    3082
       Installed from: D:\VS2010\
    Package:    vs_setup.msi
    Help link:    http://go.microsoft.com/fwlink/?LinkId=133405
    Local package:    C:\Windows\Installer\3f3bc84.msi
    Install date:    2011\01\14
    0 patch packages.

Great! Now the ASP.NET MVC3 RTM setup package worked like a charm.

Update 14-jan-2011: I found the root solution to all my problems with msiinv, check my post on the subject.

I don’t know exactly why but I can’t install the MVC 3 package provided by Web Platform Installer nor the standalone one, so I uncompressed it and tried to figure out how to install it manually.

The problem I have is with vs10-kb2465236-x86.exe that gives me a “failed with 0x8007066a” error. It seems to be for enabling Razor syntax but the prerequisites are Visual Studio 2010 Ultimate ENU. I have Visual Studio 2010 Professional ESN and that may be the reason or may not.

Anyway, I found the way to install the package manually, although the Razor Intellisense stuff is not available in my setup.

First you have to uncompress the AspNetMVC3Setup.exe into a folder and then install the following files (I got the sequence from the parameterinfo.xml file, which describes the installation process):

• VS10-KB2465236-x86.exe (it fails in my case, I don’t install it)
• AspNetWebPages.msi – ASP.NET Web Pages
• AspNetWebPagesVS2010Tools.msi – ASP.NET Web Pages Visual Studio 2010 Tools
• AspNetMVC3.msi – ASP.NET MVC 3 Runtime
• AspNetMVC3VS2010Tools.msi – ASP.NET MVC 3 Visual Studio 2010 Tools
• NuGet.msi – NuGet

And that’s all. In case you have Web Developer Express instead of full Visual Studio, you have ton install the tools specifically for that edition, AspNetWebPagesVWD2010Tools.msi instead of AspNetWebPagesVS2010Tools.msi and AspNetMVC3VWD2010Tools.msi instead of AspNetMVC3VS2010Tools.msi.

Good luck!

One of the most useful practices that I’ve been using during the last years is continuous integration and for that purpose, TeamCity is the tool of choice, which offers a free professional version with up to 20 users and 20 build configurations, enough for a small team.

Recently I tried (again, first time I found some trouble) to create an MsBuild script to make a web publication to a folder an then zip that folder in order to make a simple distributable with all the things needed to set up our web project that anyone in the organization could download and install, instead of having to create that zip myself or publishing through Visual Studio.

Finally, I got the script and seems to work using MsBuildCommunityTasks and some community knowledge.

The first part deals with publishing the site:

  <PropertyGroup>
    <OutputFolder>$(MSBuildProjectDirectory)\publish</OutputFolder>
    <TempFolder>$(MSBuildProjectDirectory)\temp\</TempFolder>
  </PropertyGroup>
  <Target Name="PublishSite">
    <Message Text="Publishing in folder: $(OutputFolder)" />
    <RemoveDir Directories="$(OutputFolder);$(TempFolder)" ContinueOnError="true" />
    <MSBuild
      Projects="src\Adapting.Web\Adapting.Web.csproj"
      Targets="ResolveReferences;_WPPCopyWebApplication"
      Properties="Configuration=Release;WebProjectOutputDir=$(OutputFolder);OutDir=$(TempFolder)"
      StopOnFirstFailure="true" />
    <RemoveDir Directories="$(TempFolder)" ContinueOnError="true" />
  </Target>

I define some variables and then I call the _WPPCopyWebApplication which is the important target (in .NET 2.0 was _CopyWebApplication). This should be the same as publishing a web site to a folder using the wizard.

Then, I get the files in the OutputFolder and make a zip with them:

  <Target Name="PackageSite" DependsOnTargets="PublishSite">
    <ItemGroup>
      <PublishFiles Include="$(OutputFolder)\**" />
    </ItemGroup>
    <Message Text="Packaging Site" />
    <Zip
      Files="@(PublishFiles)"
      ZipFileName="release-web.zip"
      WorkingDirectory="$(OutputFolder)"
      />
  </Target>

I simplified this a little bit because I have as well a target that generates the build number from SVN and changes the AssemblyInfo.cs and uses this numbers to name the zip.

In TeamCity, you can create a new configuration with *.zip as artifacts, that points to your SCM (Subversion in my case) and the most important thing is that uses the MsBuild runner targeting PackageSite.

The only thing that was that if I put the <ItemGroup> section out of the <Target>, works on the command line but not inside TeamCity.

That’s all, I hope this helps.

Sources:

• How to publish a web site with MsBuild
• Using MsBuild to create a deployment zip

Creating ConfigurationSections can be tricky but it’s quite straightforward to test them, provided you take care and don’t try to do too much like this subtle bug in MvcContrib.IncludeHandling that lead me crazy for a couple of hours.

The thing is that the code does like this:

public IncludeHandlingSectionHandler()
{
    _types = new Dictionary<IncludeType, IIncludeTypeSettings>
    {
        { IncludeType.Css, Css },
        { IncludeType.Js, Js }
    };
}

Where this is the constructor and .Css and .Js properties are marked with [ConfigurationProperty], so in theory seems correct.

In the tests, there’s an XML file with the section and it’s loaded using the DeserializeSection method of the ConfigurationSection base class. And this seems correct, everything goes ok and the dictionary is populated correctly with the data from the XML file.

But it’s wrong. If you add the section to the app.config (or web.config) file and instead of using DeserializeSection you use ConfigurationManager.GetSection(sectionName), the constructor is called before populating the Css and Js properties so the dictionary is not created properly but as those properties return a default implementation in case there’s no configuration, it seems to work, but the configured values are never loaded.

This way, tests pass but it’s impossible to configure the library via web.config. Anyway, in my opinion the problem is in .NET Framework because internally, GetSection should use DeserializeSection to be consistent.

After dealing with several options, most of them involving a call to a command line tool and parsing the output, which is not really elegant. Other solutions were based on obsolete VBScript client code and ActiveX, even worse.

Finally I found in the excellent pinvoke.net site a very nice and simple example using a simple call to iphlpapi.dll’s SendARP method. The trick was to use ARP to get the MAC from the IP.

Note that this has it’s limitations, as I will only work if the remote machine is in the same network and the packet does not need to go through a router. I was enough for our needs.

Here’s the code I adapted from the example:

public static string GetMAC(IPAddress ipAddr)
{
var mac = new byte[6];
var len = (uint)mac.Length;

var result = SendARP((int)ipAddr.Address, 0, mac, ref len);

if (result != 0)
{
return "MAC not found!";
}

var str = new string[(int)len];

for (var i = 0; i < len; i++)
{
str[i] = mac[i].ToString("x2");
}

return string.Join(":", str);
}

[DllImport("iphlpapi.dll", ExactSpelling = true)]
private static extern int SendARP(int destIP, int srcIP, byte[] pMacAddr, ref uint phyAddrLen);

Enjoy!

Since I acquired a NHibernateProfiler license, an initialization warning message draw my attention, the text seemed quite easy to solve:

WARN: custom type is not Serializable: MyLibrary.MyCustomDataType

TypeFactory.InjectParameters(userType, parameters);
if (!userType.ReturnedClass.IsSerializable)
{
LogManager.GetLogger(typeof(CustomType)).Warn("custom type is not Serializable: " + userTypeClass);
}

After some research, I posted a question at stackoverflow.com without much success, but the problem was that the answer was already there.
I you are working on any application that needs to serve files on it’s own, for example for encrypted storage or a rich ACL that needs to be enforced, you will need to provide the most correct Content-Type possible or MIME type from a given file extension to the reponse in order to achieve the best user experience.
For example, if you serve a PDF and use a generic MIME like application/octet-stream, the user may not se a PDF reader to open it, depending on the browser and platform.
Then, in .NET there are 4 alternatives to do this, with their pros and their cons:

1. Use the Windows registry – This may be a good solution for desktop applications but depends on the software installed on the machine and typically web servers don’t have Acrobat or Office.
2. Use urlmon.dll’s FindMimeFromData – I don’t know exactly the efectiveness of this method, but it’s used internally by Windows. It should give good and reliable information but you have to read the file and give the method up to 256 bytes of the file header.
3. Use IIS information – Maybe the most obscure mechanism, based on Directory Services and COM stuff. It’s based on the same mechanism IIS uses, so it should be quite reliable, but complex and may not be the fastest.
4. Use your own dictionary – Maybe not the most elegant but even .NET uses this internally (see System.Web.MimeMapping… you know how). You get a bunch of types from somewhere and add them to a dictionary with the associated extension. Easy.

So, the optiones are clear. Finally in my case, I used a simple dictionary to ensure compatibility between platforms and homogeneity. Using other mechanisms could lead to strange errors on some platforms that I wanted to avoid over other benefits. But there are much more elaborate mechanisms that can be done.