Marc's Codelog

Bye bye Kurobox

Sat, 24 Jan 2015 23:15:00 +0000

Broadcast message from root@kurobox (pts/0) (Sat Jan 24 23:55:26 2015):
The system is going down for system halt NOW!

With this message my Kurobox turned off -actually it didn’t because something is broken in the ACPI kernel stuff- for the last time.

I’ve just replaced it by a brand new Synology DS215j.

Some history

I acquired this machine from an online store called Revogear.com -now extinct- in November 2006, more than eight years ago. I can’t think how many things happened since then, in my professional and personal life and obviously in this unstoppable tech world.

But there it stood, running almost 24/7 since I installed it at my parents’ home where I lived then. It was very appealing: a small sized low power Linux server with 128Mb of RAM, a MPC8241 RISC processor @266MHz, a 3.5” HD IDE bay, Gigabit Ethernet and a single USB 2.0 port.

It really was a rebranded Buffalo Linkstation HG with some modifications to make it more customizable and a nice Kanji logo on the side. More customizable didn’t mean easily customizable, because it was pretty low level stuff using netcat and sending some esoteric commands over the wire. But that was pretty fun, at least for me 8 years ago.

After the Kurobox HG, Buffalo released the PRO, which changed the processor architecture from PowerPC to the now popular ARM and over the years the project was discontinued and forgotten. At the beginning it looked promising and there was a vibrant community and many pages and forums to find and ask stuff but soon started to languish with the poor effort from Buffalo to continue with this series.

As of today, many Linux distros still provide PowerPC packages but the main support is for Intel x86 platform. On the other hand, ARM is gaining traction, specially in the NAS field, as Synology, QNAP and others are using architecture for their low-end products. The Debian Popularity Contest has some figures about this fact, although it still offers good support for PowerPC. Fedora Project dropped 32bit PowerPC support four releases ago.

Obsolescence

There are several problems that have arisen with my Kurobox during its lifetime. I don’t know which one of them is the most important but the sum of them made me replace it.

Having just an IDE bay does not help. It’s not difficult at all to find IDE disks -and they are cheap- but they are not larger than 160-250Gb. Mine is 250Gb and it’s always full and slow. Even if I don’t put all my backups there -mainly photos- there’s not enough room. I know I can add an external USB drive but that idea is not a good idea because it won’t perform well given the limited hardware resources.

The processor was modest at the time of purchase but now is completely outdated. This is a bottleneck for file transfers, no matter if I use SCP, SMB or NFS, the transfer rate is never better than 1MB/s and generally is about 600-700KB/s. Compiling a kernel took about 20-24h…

As it is the HG version it came with 128Mb RAM soldered on the motherboard and it’s a tight number to run a lot of processes. I have used it as a test web server with a LAMP stack -it was lighttpd instead of Apache-, SSH server, mail server with Courier MTA, DNS server with djbdns, MLdonkey server and rtorrent/Transmission headless server. This was really too much: in the last years I have only used it as a NAS with a Transmission server. It worked, just required some patience.

Finally, upgrading the kernel or installing a new Linux distro are difficult tasks. I wanted to update the original kernel a couple of years ago because it was too outdated and the thing is facing Internet so, from a security standpoint was a good idea. The documentation was too hard to find -mostly at Buffalo@NAS-Central-, many 404, many pages in Japanese and too many visits to (https://archive.org/) to find just some instructions on how to do it. I succeeded but was not that fun and some stuff is broken, specially the ACPI part as I can’t reboot and shutdown anymore -unless I unplug the cord-.

The newcomer

Now I’m pretty happy :happy: with the Synology ds215j. I don’t like that is not as open as the Kurobox but the default software is really cool and works out of the box. Nowadays I value more my free time and I don’t enjoy that much hacking with old rare hardware.

So, farewell ol’ good Kurobox!

DIY Arduino Christmas tree lights

Mon, 15 Dec 2014 21:06:00 +0000

Last year I built a very simple light switch with and Arduino board and a relay shield, I put a video on Youtube buy here is the detailed explanation of the code and the parts used.

The only drawback is that the Relays have mechanical switching parts and are a bit noisy, specially in the silence of the night.

I bought the board and the shield on BricoGeek, which is a fine shop from Spain but they can be found anywhere. The other smaller parts were bought at my local electronics shop.

So, these are the required parts:

Arduino Uno board
Relay shield - I’ve got v1.3 but for this particular project it’s irrelevant.
Non blinking Christmas tree lights. Mine are LED and connected directly to the mains power.
Universal power supply. USB 5V supply is not enough to power the Relays, so you need an external supply of 9V for them to work properly.
Electrical tape (to cover the USB port of the Arduino and avoid accidental contact with the AC power damaging the board)
Two wire electrical cable
Male and female AC plugs
Small regular screwdrivers for the relays and maybe other types to mount the plugs

The job is easy, just be careful because mains current levels are dangerous and some parts of the board leave it exposed.

The electrical cable has two wires, just cut one of them and connect one lead to the COM1 and the other to the NO1. If you have more lights just connect them to COM2, COM3, COM4 and NO2, NO3 and NO4 respectively.

You can also use NCn instead of NOn: NC stands for Normally Closed which means it’s connected -closed- when pin is low (0) and disconnected -open- when pin is high. NCn works the opposite.

A detailed explanation of the relay can be found on Seeedstudio wiki page

Then, mount the plugs on both sides of the cable, male on one side an female on the other.

Finally put some electrical tape over the Arduino USB connector and mount the Relay Shield on top of the Arduino board.

Finally it’s time to program the board. On my Github arduino-christmas-lights project you can find the example code.

Basically on the code you can see two state arrays states1[] and states2[], one for each connected light. They should be the same size to maintain synchronization between both light strips. On each state array you can define the HIGH/LOW sequences, as long as you want.

Finally there’s a loop that turns the relay on and off with a 500ms delay between changes. If you put two equal consecutive states, the relay will maintain it’s state.

As you can see it’s very simple, but works and opens a world of possibilities. For example syncing the lights with a song adjusting the patterns and the delay to the rhythm.

If you have any doubt just add a comment, drop me a line or add an issue on the Github project.

Merry Christmas and have fun!

Back to blogging

Fri, 28 Nov 2014 21:59:00 +0000

It’s been roughly 3 years and a half since my last blog post. Why I abandoned my blog? Don’t I have anything to say?

Reasons for not blogging

I can identify several.

But first of all is laziness and procrastination. Both are tied but the effect is the same: willing to post but never finding time to do so, there’s always something better to do. As I grow, I tend to be more rigid with this kind of stuff: do it or don’t but don’t keep telling yourself: “I’ll do it later”.

On the other hand, I’ve not been vegetating on my couch watching TV cheap soap operas the whole time but, definitely I could’ve found some time to post many things if I wanted.

But on my discharge I can say that this last year has been quite enduring with the kid, having children changes your life.

Why coming back?

Writing. I have written a lot of things in the past, specially when I was younger and I find it a good exercise.

Learning. Explaining things is a great way to learn, and explaining those things in another language -if you’re proficient in English, you may have noticed it’s not mine- makes me improve my fluency, add new vocabulary and it’s fun.

Showing. Internet is for showing off. Period. Anyone in their houses can show the world the stuff they do, what they think, who they are. And they can reach the whole connected world, just as I do.

Will this last?

Who knows. Maybe I become lazy again, or feel I don’t have anything to share or feel I don’t have time to. Time will tell.

But for the moment, I welcome myself back, and you too.

Lessons learned optimizing MySQL

Fri, 05 Aug 2011 11:41:18 +0000

This week I’ve been quite entertained at work optimizing a big MySQL database, 18M rows and about 10Gb of table data for a high traffic web site (11M unique views per month). Not optimizing for performance is overkill in this case, so I have been learning and enjoying a lot during these days.

The first thing to note is that database servers are complex beasts (don’t underestimate them) and each and every database provider has its own strengths but also its own caveats.

Sooner or later you’ll face a behavior that may seem counter-intuitive and there’s when you must read a lot of documentation. In this case, MySQL Performance Blog and MySQL documentation have been of great help, as well as the now unmantained-but-still-useful Hack MySQL site, which provides great tips very well explained.

Second. As always, doing an 80% optimization consumes 20% of the time and most of it is quite straightforward. EXPLAIN, review results, create index, EXPLAIN again. This way you can do a good job just knowing some indexing basics. Your indexes may be a bit fat, but will do the job.

Then you’ll need to be patient, creating indexes is not always fast, in fact the number, size and heterogeneity of the columns involved will increase the time it takes to create them. While you wait, the best is to read good books about MySQL or relational databases like: Relational Database Index Design and the Optimizers.

You’ll need to be organized. Creating indexes randomly just looking a couple of slow SELECTs is something, but you must look at the big picture: make an inventory of all the application queries, no matter how unimportant look like because if they attack a big table with a WHERE or an ORDER BY they will perform bad. Fact. This list will help you organize the indexes and see which ones are used and which not and where.

Learn as well what the configuration parameters mean, MySQL has a bunch of them. Just increasing numbers will not always solve your bottlenecks. If you see something strange, there might be a parameter misconfigured and causing bad results.

Sometimes you’ll find yourself frustrated by something that you can’t understand. I found that almost anything in this field has an explanation, even a reasonable one.

Finally, don’t overlook other thinks. Database performance is important but there may be a lot of room to improve in your code, don’t blame MySQL as the root of all problems!

Simple Mini Profiler Glimpse plugin

Sun, 19 Jun 2011 16:01:35 +0000

I recently published on Github a simple small project (miniprofiler-glimpse-plugin) to integrate the results collected by the excellent MiniProfiler created by the people at Stackoverflow with Glimpse, the Firebug for ASP.NET and MVC applications.

Creating Glimpse plugins is really easy (just implement IGlimpsePlugin and decorate it with GlimpsePluginAttribute), implement the methods and reference the plug-in assembly in the project where you are using Glimpse. MEF will do the gluing and the JSON.NET will do its best to deserialize the results in a human readable way.

To use this plug-in, just compile it and reference the assembly in your Glimpse-enabled project. I you have questions, suggestions or bug reports, you can use Github’s own issue reporting system, drop me a line o fork the project and make your improvements.

So, I hope you like it.

Link: miniprofiler-glimpse-plugin [github.org]

Carbon Copy Cloner saved my day

Sun, 12 Jun 2011 21:13:43 +0000

This post is just an agreement to the great people behind Carbon Copy Cloner, a great piece of backup software for Mac.

I trusted Time Machine, and actually it seemed a good program and a very convenient way of making and managing backups but… it does not matter how pretty is if when you need to restore a backup it gives you strange errors.

The whole story is that I wanted to upgrade my Mac hard drive to a bigger one and reading Time Machine documentation, seems that I can use the OSX Installation disk to restore a backup to a new fresh drive. Well, that’s just theory but when doing as instructed I got an “Error -22” after about an hour restoring… epic fail. Good that I did keep the original drive untouched.

Then I tried donationware Carbon Copy Cloner to create a bootable clone backup of my working hard drive. After a couple of hours I had an external USB that mimicked my original drive. I replaced the hard drive, created a new partition and cloned again the external drive backup to the new one. Now I’m writing this on my upgraded Mac.

So, thanks to Carbon Copy Cloner from Bombich I was able to upgrade without problems. Great job!

Troubleshooting MSDTC, RPC and NServiceBus issues

Mon, 06 Jun 2011 16:19:47 +0000

At first it seemed easy, but it’s been a long distance hurdle race. At the end, I have solved the problems I’ve found and my painful experience can be useful for other people, so this is it.

First of all, my topology is the following: a client, in this case for testing purposes, my machine, which is running an ASP.NET web application and a NServiceBus host that processes all the messages. This has been configured following the official documentation and looking at the samples (check out the full NServiceBus source code, you’ll need it). It’s a bit messy but with some trial and error you can make sense of it. There are some obscure properties that have to be googled to know what they mean.

Processing the messages is not trivial for the bus host. Some of them attack the database and need to use MSDTC to manage the transactions. This means that MSDTC must be up and running without problems.

One trick I learned after a lot of researching is the Runner.exe tool from NServiceBus. This comes with NServiceBus source code (and I think that also with paid version) and ends up in build\tools\MsmqUtils if you build it with build.bat or build-net4.bat. Just open a command line as administrator and run:

Runner.exe /i

This will install MSDTC and configure it properly. I don’t think you’ll need anything else but check the MSDTC configuration in Component Services -> Computers -> My Computer -> Distributed Transaction Coordinator -> Local DTC. Right click to go to properties and ensure everything is configured like in this Troubleshooting DTC post.

At this point, I recommend you to disable the Windows Firewall on both machines, at least for domain network, just to be sure it’s not interfering and breaking the tests, we will solve this later.

Then, DTCPing is your friend. This small tool will check the DTC connectivity between two machines. It’s a bit awkward. You have to start the tool in both machines, then put the name of the other and click ping on each one. This will make a test and if everything goes well, you’ll see a detailed log without errors:

++++++++++++Validating Remote Computer Name++++++++++++
Please refer to following log file for details:
	C:\tools\DTCPing\LOCALMACHINE2928.log
Invoking RPC method on REMOTEMACHINE
RPC test is successful
++++++++++++RPC test completed+++++++++++++++
++++++++++++Start DTC Binding Test +++++++++++++
Trying Bind to REMOTEMACHINE
Received reverse bind call from REMOTEMACHINE
Binding success: LOCALMACHINE-->REMOTEMACHINE
++++++++++++DTC Binding Test END+++++++++++++
Please send following LOG to Microsoft for analysis:
	Partner LOG: REMOTEMACHINE5928.log
	My LOG: LOCALMACHINE2928.log
++++++++++++Start Reverse Bind Test+++++++++++++
Received Bind call from REMOTEMACHINE
Trying Reverse Bind to REMOTEMACHINE
Reverse Binding success: LOCALMACHINE-->REMOTEMACHINE
++++++++++++Reverse Bind Test ENDED++++++++++

Here started my nightmare, I got the damned: RPC error 5 (Access denied). After hours googling for the problem I found this post about MSDTC problems (I was looking for RPC problems, which actually was the root cause). The cause is that remote RPC is disabled in Windows workstations by default.

I opened for the N-th time the registry and added the RPC key to HKLM\Software\Policies\Microsoft\Windows NT and the RestrictRemoteClients DWORD value to 0. Rebooted without much hope. Magically, DTCPing worked and life seemed better and brighter.

Then, the Windows Firewall must be configured. In Windows Server 2008 (or Windows 7, it’s the same interface), go to Control Panel -> System and Security -> Windows Firewall (or just execute firewall.cpl at the command prompt). Go to Advanced Configuration to open a new window. Go to inbound rules and look for the Distributed Transaction Coordinator. If they are disabled, enable them. On the workstation the firewall can be enabled. Beware that DTCPing may not work, because the Firewall is only allowing connections to/from svchost.exe and msdtc.exe (if you want it to work, add a rule for that process).

Using fail2ban with nginx in Debian

Sun, 13 Feb 2011 14:03:16 +0000

Taking a look at the logwatch mails I see a common pattern of attacks, coming from China and trying to find details of my server configuration, which is something I don’t like.

Looking around I found fail2ban which is a tool that does some regex matches on the server logs (sshd, httpd, authd, …) and takes proper actions, like banning the offending IP.

I then installed fail2ban in my Debian box:

apt-get install fail2ban

Then, I took a look at /etc/fail2ban/jail.conf and found some entries for Apache but none for nginx, my server of choice, so I decided to create a jail.local to add some nginx stuff (this is recommended in Debian to allow hassle-free upgrades).

I started copying the Apache sections of the default fail2ban as the log files in my case use the same format that allows me to use Awstats easily. Then, I modified my log routes to point to the nginx ones and using Apache rules, if they don’t work I’ll tune them later.

[nginx]
enabled = true
port    = http,https
filter  = apache-auth
logpath = /var/log/nginx*/*error.log
maxretry = 6

[nginx-noscript]
enabled = false
port    = http,https
filter  = apache-noscript
logpath = /var/log/nginx*/*error.log
maxretry = 6

[nginx-overflows]
enabled = false
port    = http,https
filter  = apache-overflows
logpath = /var/log/nginx*/*error.log
maxretry = 2

Although this is ok, the bots I see don’t leave a trace in error.log but in access.log so I took a look at the filter.d folder where an interesting apache-badbots.conf was present. Then, I found the default fail2ban documentation in /usr/share/doc/fail2ban where there’s an usage example of the badbots script. I added I to my jail.local:

[apache-badbots]
enabled  = true
port    = http,http
filter   = apache-badbots
logpath  = /var/log/nginx*/*access.log
bantime  = 172800
maxretry = 1

Finally, I added this to the top of the file, to send mails to myself when a rule matches and a host is banned.

[DEFAULT]
action = %(action_mwl)s

Finally, restart the service and start receiving mails:

sudo /etc/init.d/fail2ban restart

I’m sure this will need further adjustments, but it’s a beginning in my bot fighting war. I’ll make some updates as I find interesting results.

How msiinv saved my day

Fri, 14 Jan 2011 11:22:31 +0000

In my previous post I reported my problems installing MVC3 in my machine. That led me to further investigation as that worked perfectly in my co-workers’ machines.

I tried to repair Visual Studio 2010 and told me that it had troubles with vs_setup.msi… strange. Looking around I found a post that led me to an explanation of the msiinv.exe tool.

It took me a while to get the tool as the author’s site is broken, but Stackoverflow came on rescue.

Then, I ran the tool:

msiinv -p &gt; msiinv.txt

Then opened the file and… surprise!

Microsoft Visual Studio 2010 Professional - ESN
    Product code:    {725041D1-9F45-30D3-A78E-DF08C4E1A297}
    Product state:    (1) The product is advertised, but not installed.
    Package code:    {9E58363D-E4A9-49D3-82B7-42584AFCA9E7}
    AssignmentType:    1
    Language:    3082
        Package:    vs_setup.msi
    0 patch packages.

Strange. I tried to fix it with MSI:

msiexec /f {725041D1-9F45-30D3-A78E-DF08C4E1A297}

It asked me for the vs_setup.msi, it’s in the root of the Visual Studio 2010 DVD in case it’s not found automatically. Magically it worked and now it’s installed:

Microsoft Visual Studio 2010 Professional - ESN
    Product code:    {725041D1-9F45-30D3-A78E-DF08C4E1A297}
    Product state:    (5) Installed.
    Package code:    {9E58363D-E4A9-49D3-82B7-42584AFCA9E7}
    Version:    10.0.30319
    AssignmentType:    1
    Publisher:    Microsoft Corporation
    Language:    3082
       Installed from: D:\VS2010\
    Package:    vs_setup.msi
    Help link:    http://go.microsoft.com/fwlink/?LinkId=133405
    Local package:    C:\Windows\Installer\3f3bc84.msi
    Install date:    2011\01\14
    0 patch packages.

Great! Now the ASP.NET MVC3 RTM setup package worked like a charm.

Install ASP.NET MVC 3 Manually

Wed, 12 Jan 2011 15:49:08 +0000

Update 14-jan-2011: I found the root solution to all my problems with msiinv, check my post on the subject.

I don’t know exactly why but I can’t install the MVC 3 package provided by Web Platform Installer nor the standalone one, so I uncompressed it and tried to figure out how to install it manually.

The problem I have is with vs10-kb2465236-x86.exe that gives me a failed with 0x8007066a error. It seems to be for enabling Razor syntax but the prerequisites are Visual Studio 2010 Ultimate ENU. I have Visual Studio 2010 Professional ESN and that may be the reason or may not.

Anyway, I found the way to install the package manually, although the Razor Intellisense stuff is not available in my setup.

First you have to uncompress the AspNetMVC3Setup.exe into a folder and then install the following files (I got the sequence from the parameterinfo.xml file, which describes the installation process):

~~VS10-KB2465236-x86.exe~~ (it fails in my case, I don’t install it)
AspNetWebPages.msi - ASP.NET Web Pages
AspNetWebPagesVS2010Tools.msi - ASP.NET Web Pages Visual Studio 2010 Tools
AspNetMVC3.msi - ASP.NET MVC 3 Runtime
AspNetMVC3VS2010Tools.msi - ASP.NET MVC 3 Visual Studio 2010 Tools
NuGet.msi - NuGet

And that’s all. In case you have Web Developer Express instead of full Visual Studio, you have ton install the tools specifically for that edition, AspNetWebPagesVWD2010Tools.msi instead of AspNetWebPagesVS2010Tools.msi and AspNetMVC3VWD2010Tools.msi instead of AspNetMVC3VS2010Tools.msi.

Good luck!