Netuality

One of the ways of scaling a heavy-traffic LAMP web application is to transform the server into a cluster of servers. Some may opt to walk on the easy path by using an overpriced appliance load balancer, but the most daring [and budget-restrained] will go for free software solutions such as pound or haproxy.

Although excellent performers, these free balancers have lots of missing features when compared with counterpart commercial solutions. One of the most embarrassing misses is the lack of flexibility in producing decent access logs. Both pound (LogLevel 4) and haproxy (option httplog) may generate Apache-like logs in their logfiles or the syslog, however none offers the level of customization encountered in Apache. Basically, you're left with using the logs from the cluster nodes. These logs present a couple of problems:

• the originating IP is always the internal IP of the balancer
• there is one log/node, while log analysis tools can usually cope with a single log file/report

First problem is relatively easy to solve. Start by activating the X-Forwarded-For header in the balancing software : for instance configuring haproxy with option forwardfor. A relatively unknown Apache module called mod_rpaf will solve the tedious task of extracting the remote IP from X-Forwarded-For header and copying it in the remote address field of Apache logs. For Debian Linux fans, it's nice to note that libapache-mod-rpaf is available via apt.

Now that you have N realistic Apache weblogs, 1 per cluster node, you just have to concatenate and put them in a form understandable by your log analysis tools. Just simply cat-ing them in a big file, won't cut it [arf] because new records will appear in different regions of the file instead of appending chronologically to its tail. The easiest solution in that case is to perform a sort on these logs. Although I am aware of the vague possibility of sorting on the Apache datetime field, even taking the locale into account, I confess my profound inability of finding the right combination of parameters. Instead, I choose to add a custom field in the Apache log; using the following log format:

LogFormat "%h %V %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" "%{Cookie}i" %c %T "%{%Y%m%d%H%M%S}t"" combined

where %{%Y%m%d%H%M%S}t is a standard projection of current datetime in an easily sortable integer, like for instance 20050925120000 – equivalent of 25 Sep 2005 12:00:00. Now, considering the quote as a separator in the Apache log format, is easy to sort upon this custom field [the 10th]:

sort -b -t """ -T /opt -k 10 /logpath/access?.log > /logpath/full.log

And there you are, having this nice huge log file to munch on. On a standard P4 with 1GB of RAM it takes less than a minute to obtain a 2GB log file…

In case the web traffic is really big and log analysis process impacts the existing web activity, use a separate machine instead of overloading one of the cluster nodes. For automated transfer of log files, generate ssh keys on all the cluster nodes for paswordless login from the web analytics server in the web logfiles owner account. Minimization of traffic between these machines is done by installing rsync on them and them using rsync via ssh:

rsync -e ssh -Cavz www-data@node1:/var/log/apache/access.log /logpath/access1.log

Now, you know all the steps required to fully automate the log aggregation and its processing. One may ask why all the fuss when in fact a simple subscription to a ASP style web analytics provider should suffice. Yes, it's true however… The cluster that I've recently configured with this procedure has a few million hits per week. Yes, we're talking about page hits. At this level of traffic, the cost for a web analytics service starts from 10.000$/year. It's certainly a nice amount of money, which will allow you to afford your own analytics tool [such as for instance Urchin v5] and keep some cash from the first year. Some might say that this kind of commercial tools have their own load balancer analysis techniques. Sure, but it all comes with a cost. In the case of Urchin, you just saved 695$/node and some bragging rights with your mates. Relax and enjoy.

PS: Yes we're talking millions of page hits LAMP solution not J2EE… Maybe I'll get into details on another occasion, assuming that somebody is interested. Leave a comment, send a mail or something.

Recently, HTTPUnit project reached version 1.6. While this nifty API is mainly targeted at unit testing webapps, I have also succesfully used it for other purposes such as :

HTTPUnit as a benchmarking tool

There is a plethora of web benchmarking tools out there, both freeware and commercial. However, my customer requested some features for testing, that I've had troubles satisfying simultaneously with the existing tools:

• the tests must run on headless systems (console mode, non GUI)
• load testing should simulate complex and realistic user interactions with the site

AFAIK, all the testing tools that allow recording and replaying of intricate web interaction scenarios are GUI-based. And then, command-line tools are also unfit for the job, take for instance Apache Jmeter which is basically a command-line tool with a Swing GUI slapped on it. While Jmeter is great when it comes to bombing a server (or a cluster, for that matter) with requests, it seriously lacks features when it comes to scripting complex interaction (you'd better know your URL's and cookies by heart … well, almost).

Another problem I see with existing automated testing solutions is with their error detection mechanisms. While the vast majority of tools are scanning for standard HTTP error codes such as 404 or 500 in order to find out if the response is erroneous or not, errors in complex Java apps might come as plain web pages containing strack traces and environment information (a good example is the error page in Apache Tapestry).

So eventually I had to come up with an ad-hoc solution – basic idea was to leverage the existing HTTP unit tests for benchmarking purposes. I had to get out of the toolbox another rather under-rated open-source gem: JUnitPerf, in fact a couple of decorators for Junit tests. LoadTest is the decorator I'm interested in : it allows running a test in multithreaded mode, simulating any number of concurrent users. Thus, I am able to reproduce heavy loads of complex user interaction and precisely count the number of errors. The snippet of code is something like:

SimpleWebTestPerf swtp = new SimpleWebTestPerf("testOfMyWebapp");
Test loadTest = new LoadTest(swtp, nbConcurrentThreads);
TestResult tr = TestRunner.run(loadTest);
int nbErr = tr.errorCount();

Now, we'll call this code with increasing values of nbConcurrentThreads and see where the errors start to appear. Might as well write the results in a log file and even create a nice PNG via JFreeChart. Alas, things become a little trickier when we want to measure the bandwidth; in our case we'll have to write something very lame in the TestCase; and it goes like that:

private long bytes = 0;

private synchronized void addBytes(long b)
{
	bytes += b;
}

/**
 * After a test was run, returns the volume of HTTP data.
 * @return
 */
public long getBytes()
{
	return bytes;
}

public void testProdsPerformance() throws MalformedURLException,
  IOException, SAXException
{
	[...]
	WebConversation wc = new WebConversation();
	WebResponse resp = wc.getResponse(something);
	addBytes(resp.getContentLength());
	[...]
}

Then, in the benchmarking code, we'll do swtp.getBytes() in order to find out how many bytes passed between the server and the test client. It is still unclear for me if this value is correct if mod_gzip is activated on the server (we might actually measure the bandwidth of the 'deflated' pages !?).

In order to measure the elapsed time, we'll do a similar (lame) trick with a private long time member and a private synchronized void addTime(long millis). Unfortunately, we do not [yet?] have a getElapsedTime() for the WebResponse, so we'll have to use the good old System.currentTimeMillis() before and after extraction of each WebResponse. Of course, this is also measuring the parsing time of WebResponse, but this isn't usually a problem when you are testing a large number of concurrent users, as the parsing time is much smaller when compared with the response time of a stressed server. But, you'll need a strong system for the client-side tests.

Another tip I've found: use Random in order to test different slices of data on different test runs. This way, when you run, let's say, the 20 threads test, you'll kick different data compared to the previous test, on 10 threads. In this manner, the results will be less influenced by the tested application cache(s). It's perfectly possible to launch LoadTest threads with delays between thread activation, which means that the Random seed could be different within each simulated client – if you're looking for even more realistic behavior.

HTTPUnit as a Web integration API

Besides being a great testing tool, Httpunit is also a cool API for Web manipulation, you can use it to perform data integration with all sorts of websites. For instance, let's log on the public demo instance of MantisBT bug tracking system, as user 'developer', and extract the descriptions of the first three bugs in the list.

package webtests;

import java.io.IOException;
import java.net.MalformedURLException;

import org.xml.sax.SAXException;

import com.meterware.httpunit.WebConversation;
import com.meterware.httpunit.WebForm;
import com.meterware.httpunit.WebResponse;
import com.meterware.httpunit.WebTable;

/**
 * Simple demo class using httpunit to extract the description of
 * three most recent bugs from the MantisBT public demo,
 * logged as 'developer'.
 * @author Adrian Spinei [email protected]
 * @version $Id: $
 */
public class MantisTest
{

	public static void main(String[] args) throws MalformedURLException, IOException, SAXException
	{
		WebConversation wc = new WebConversation();
		WebResponse resp = wc.getResponse("http://mantisbt.sourceforge.net/mantis/login_page.php");
		WebForm wForm = resp.getFormWithName("login_form");
		wForm.setParameter("username", "developer");
		wForm.setParameter("password", "developer");
		//submit login, conect to front page
		resp = wForm.submit();
		//'click' on the 'View Bugs' link
		resp = resp.getLinkWith("View Bugs").click();
		//retrieve the table containing the bug list
		//you'll have to believe me on this one, I've counted the tables !
		WebTable webTable = resp.getTables()[3];
		//first three rows are : navigation and header, then a blank formatting row

		//interesting data starts from the 4th column

		System.out.println(webTable.getCellAsText(3, webTable.getColumnCount() - 1));
		System.out.println(webTable.getCellAsText(4, webTable.getColumnCount() - 1));
		System.out.println(webTable.getCellAsText(5, webTable.getColumnCount() - 1));
	}
}

The code speaks for itself: HTTPUnit is beautiful, intuitive and easy to use.

Other interesting HTTPUnit-related articles:

• HttpUnit – how to write tests on CodeCzar.
• Getting Started with HttpUnit and JDeveloper 10g
• Test Web applications with HttpUnit, published on JavaWorld
• HttpUnit tutorial on C2 wiki, extracting a Dilbert cartoon (that's a cool integration example)
• JWebUnit (suggested in the comments) is a wrapper around httpunit, offering an even higher level API and simpler usage.

Until now, I've always had a certain respect for Sybase database.
Based on their history, I thought that the product is a sort of MS SQL
without the glitzy features – think Las Vegas without the lights, the cowboy boots and Eiffel Tower. Which gives: huge crowds of fat tourists in tall, dull buildings. But you always have Celine Dion, right ?*

Wrong ! Sybase ASE is a large, enormous, huge piece of steaming …
ummm … code. Ok, ok, I'm a bit over-reacting. In fact, Sybase is a
very good database – if you are still living in the 90s and the only
Linux flavour that you are able to manage is RedHat ASE. Otherwise,
it's a huge … you know.

Let's not talk about the extreme fragility of this … this product.
You never know when it crashes on you without any specific reason.
Sometimes, it's the bad weather. Crash. Restart. Sometimes, you
flushed twice. Crash. Restart. And maybe, yes maybe, you spent more
than 12 minutes in your lunch break. Crash.

Let's just talk about the JDBC drivers – latest version, downloaded
from the site in a temporary moment of insanity. Man, this is cutting
edge. The sharpest cutting edge you'll ever find – the more advanced
JDBC drivers bar none ! Excepting of course the fact that these
classes were compiled in that memorable day of 6 January 2002 [the day
when the last Sybase JDBC drivers were compiled]. But, don't let such
obscure details ruining your enthusiasm. Just download and use them
and you'll be amazed at their unique features – it's the only JDBC
driver which manages to put down DbVisualizer in different and
innovative ways. I'm restarting the poor thing (dbvis) at least 2 or 3
times per day, when working with Sybase.

Also, as a little quasi-undocumented tip, the letter d from
jconn2d.jar does not mean development [drivers] as some of you would be
inclined to think. In fact, it means debug which is the
abbreviation for put me in production poor lousy bastard and I'll
start spewing reams of useless messages through all the logging APIs
known by man and a few yet to be discovered, therefore instantly
slowing down to a crawl your puny little software.

Ermmm … well, all this nice introduction just to humbly confess that
we do have a couple of them Sybase licences floating around here and
some of our production databases are on Sybase ASE. While I can assure
you that this is going to change at least on the web backend, it's
also true that the beast must be alive in order to keep our company up
and running. And, that's part of my job.

Which is of course very strange for an IT management job. But then again:
when your sole DBA is overwhelmed by a horde of Sybase-specific tasks
(like for instance the log configured to truncate at a specific
threshold which naturally [for Sybase] does not truncate, suddenly
throwing all the users into log suspend in the middle of the peak
production time) – you have to enter into the damn kitchen and do some
cooking !

My endeavour was to perform simple data mining in order to migrate
some reports from a legacy system [no, you don't really want to know
what system]. Due to my limited time, starting a mildly complex Java
reporting project was out of the question, so a scripting language was
the natural choice. Python being the first option – unfortunately,
finding a working Sybase driver for Python is a challenge in itself.
But, thank God for Jython and zxJDBC ! In just a few minutes I was
wiring the tables for reporting. Here's a fake code snippet which lists the 'orders' with amounts < 1000, and you can't go any simpler:

from com.ziclix.python.sql import zxJDBC 

conn = zxJDBC.connect("jdbc:sybase:Tds:myserver.mydomain.com:4110/mydb",
	"user","pass","com.sybase.jdbc2.jdbc.SybDriver")
cursor = conn.cursor() 

cursor.execute("select count(orderid) from orders")
nb_orders=cursor.fetchone()[0]
print '%.0f total orders ...'%(nb_orders,)

cursor.execute("select orderid, amount from orders where amount <1000")
oids=cursor.fetchall()
for o in oids:
	print 'Order %.0f for amount %.2f'%(o[0],o[1])

cursor.close()
conn.close()

Easy as pie. And once you got one down, you got'em all. For instance,
yesterday evening I wrote in about an hour a small Jython script which
exports some data. The same export process (running about an order of
magnitude slower) needed previously a couple days of development in
FoxPro. Ah, FoxPro - very juicy subject, but I'll keep it for my next
horror story. Until then, don't forget, when you have a monster to
tame and no time at all: try Jython !

*Let's suppose for a very brief instant that having Celine Dion at a certain location is a positive thing.

… then a software team is like an aquarium.

“Programming is Gardening, not Engineering” says Andy Hunt (of Pragmatic Programmer fame) in one of his well-known Artima conversations.

Inspired by such an interesting ‘organical’ comparison, it’s my metaphor of a software team which behaves quite like an aquarium. I assume not all my blog readers are aquaria hobbists, so let me explain:

• Permanent monitoring and adjusting. Left alone and unsupervised, an aquarium apparently manages to ‘survive’ by itself. However, subtle changes in water chemistry will slowly start to build up. Interesting fact is that fishes seem to cope well with these changes – until a certain balance is reached and they get sick and eventually die. In my experience, the threshold is rather thin, one day everything seems ok and the next day it’s a major disaster. The effort necessary to clean up the situation is significantly bigger than the effort spared by not taking care of the aquarium. The parallel here is quite obvious : you can’t manage what you can’t measure, you can’t control what you can’t manage. Software metrics, code reviews, frequent releases, testing and feedback, these practices are vital if you want a ‘healthy’ project and a ‘living’ team. Otherwise, beware, the inflexion point might be just a few days away*.
• However, changes must be done gradually. Supposing that a major shift in water parameters was detected, taking immediate and radical measures will generally worsen the situation (unless the catastrophy is already there). It is highly recommended to distribute the change over a reasonable period of time and generally never try to influence two major water parameters at the same time (Ph and Gh for instance). Explanation: all these parameters are interconnected in intricate ways, by changing one you’ll automatically influence the others. By changing two or more, the outcome is hard to predict and might open the path to a disaster. There’s a nice parallel here. A major change in methodology with sudden introduction of multiple new/modified development practices, will only make the team unstable. Even if, globally speaking, the change is a highly beneficial one. ‘Good things come to those who wait’ … and measure … and change … and wait … and measure … and change …
• A beautiful aquarium is a visible one. Transparent glass, lights and everything. You wouldn’t feed and keep your fishes if they were living in a black box and you are afraid to look inside it ?

*Of course [and fortunately], the developers do not get sick because of a reeking team/project, they simply leave.

Nice excerpt from an Inc. article :

Future Beef's real albatross, says Darrell Wilkes, was an expensive computer system that ran enormously complex software from J.D. Edwards. “We were pouring all this data in, but we could never get the data out,” says Wilkes, whose job, as the company's cattle and supply expert, was to monitor the progress of about 300,000 head of cattle. And these weren't just any cattle. These were Future Beef cattle, raised by ranchers who met its standards for feeding and monitoring their herds. Wilkes had to keep track of which steers had been given certain region-specific mineral supplements, which had been fed their necessary doses of vitamin E, which had been measured for yields of particular tissues and fats, and what the optimum dates were for shipping each steer off to the slaughterhouse.

When Wilkes asked his staff for the numbers, they didn't have a clue. They had not been able to retrieve the data from the computer system.

“I swore a lot, and jumped up and down a lot, but it didn't do a lot of good,” he says. “We would have been better off going in there with a very simple system. At least the simple systems give you your damn yield report.”

Click here to read the whole article.

Short sample from the rants of a software engineer temporarily converted into a lazy network administrator…

Should you ever need to install a forwarding DNS proxy on Debian, which also acts as DNS for the local network, don't even think about using Bind. This is a very powerful tool, but can be difficult to configure, definitely not for the faint at heart. The frequence of Bind vulnerabilities is sometimes worrying, and you'll have to dedicate more time to Bind administration than you dedicate to walking your dog at dawn. A dubious pleasure, especially when you don't have a dog.

Next contender is a stable, rock solid, simple to use tool : djbdns. Which might prove problematic on Debian, because djbdns and daemontools are not (yet ?) in the main distribution. A kind soul on the net offer some 'homemade' Debian packages, that you'll easily install. Then, you'll quickly remove it, since these packages do not have the slightest intention to work on the current Debian Sarge. My last choice was to compile from scratch and maintain djbdns by myself. With the fresh memories of administrating a bunch of Gentoo servers (perfectly comparable with walking a pack of 23 to 426 dogs, depending on how often you update your systems) – I decided to skip this option. Note to self : never ever maintain software packages built from source needing frequent manual updates.

Hopefully, there's dnsmasq to save the day. In my case, the night. An apt-get install dnsmasq later, I have a working DNS server, resolving internal names from /etc/hosts and forwarding all the other queries to my ISP DNS servers. As a bonus, it also makes for a nice DHCP server, which I don't need for the moment but might come handy later, who knows ?

Recently, an Indian poster on a Slashdot thread ('India Outsourcers Find Back Door in Canada') mentioned that:

Software Developers in India (including me) are paid 350 times the prevailing minimum wage in India.

I thought it would be interesting to make a comparison with the same data in Europe:

• in a country such as France, this makes 402.640EUR/month.
• in one of the countries wich has recently join EU, such as Hungary, this means almost 67.000EUR/month (Hungary does not have the biggest wages in the recent wave of new EU members).
• finally in a country such as Romania which is scheduled to join EU in 2007, 350 times the minimum wage is however close to 25.000EUR/month and rising with two figures percentage in the last couple of years, due to economic growth. Romania and Bulgaria are considered the poorest countries to join EU in the next years.

Outsourcing in India is more than just a temporary industry trend … and will touch all EU as strong as it does with US. Dear US developers, would you please let us join the club ? (we'd rather not, but you know …)

While the REST API is mentioned in the Top 10 Reasons to use Jira, I can hardly see any reference to such an external API in the documentation (I mean, besides XML-RPC and upcoming SOAP support) and even Google can't clear the issue. But I can confirm you, it is possible to fully(?) automate Jira using HTTP requests.

Recently, I was asked to programatically login a user into Jira and open a browser with the results of an advanced search. This is part of a really slick integration between one of my employer's products and a JIRA server – mainly helping our testers to avoid bug duplication and also providing insightful statistics for QA dept.

I've started doing it the hard way, trying to obtain and pass JSESSIONID and other useless stuff between my application and the browser, until I realized that Jira can be fully controlled through HTTP in a REST-like manner. Let me explain. Normally, if you are not logged into Jira and launch a browser with a carefully crafted search request (well, you know how to reverse enginner POSTs into GETs don't you ?) – then a graceful invitation to log in is everything you'll ever obtain. But, if you add at the end of your request “&os_username=” + USER + “&os_password=”+ PASS bang ! not only you obtain the desired search results, but you are automagically logged into Jira for the rest of the browsing session. Yes, yes yes : here I come, integration ! A couple of hours later, I am able to programatically insert bug reports, extract bug details, compute statistics and open customized advanced searches.

To quote a classic : Jira ! Jira ! Jira !. Docs would be nice, though.

PS I'm testing this on a Jira Professional 2.6.1-#65. YMMV.

This is the summary of a nice thread spotted on eclipse.platform.rcp. The initial question was I am constantly having to deal with resistance from my developers, which mostly boils down to resistance to SWT, and hesitance from peers in the company. There are compelling arguments to counter nearly every criticism that I've encounted. However, I'm having difficulty with one question, has anyone else based a non-IDE product on Eclipse?. A few answers later, boy do we have some nice eye candy to show !

	The XMF Mosaic Development Environment from Xantium provides “platform for engineering semantically rich languages and tools” (errr, whatever this means, just looks nice).
	Eclipse Trader is not quite a fully-fledged app, but a set of plugins for the Eclipse platform, dedicated to the building of an online stock trading system. This is an open-sourced, very recent project, with interesting features such as charts built with data extracted from Yahoo! Finance.
	EPIC from Incremental stands for a nice looking management software for the ship repair industry. You'll spot some gorgeous Eclipse Forms in the screenshots !

Being extremely busy lately, I arrive a bit late at the Junit destruction feast. While it is probably true that some guys with a certain gift for writing blog articles may “come up with something far more useful in a couple of days”, I think the discussion is missing an important point: there's a whole ecosystem living around Junit. We have Ant integration, we have the choice between code coverage tools (both commercial and open-source), plugins for mainstream IDEs and a certain number of useful or less-useful extensions. We have extensive documentation and a plethora of examples to feed the small fishes. Throwing Junit down the drain means throwing all these down the drain. Or, at least: write your own Ant integration, adapt a code coverage tool and rewrite the IDE integration, rewrite documentation and examples – this is not going to be done in “a couple of days”.

Another Junit advantage is that this little simplistic API is ubicuous. I mean, every developer heard about it and knows how to use it, unless of course he/she was living under a rock for the last few years. And I don't mean every Java developer, but just about every developer for a language under the xunit umbrella. Meaning : all the programming languages (unless you consider “languages” such as Whitespace, Brainfuck and INTERCAL).

Beck and Gamma have not only written some “crappy” classes and put the few “laughable” chunks of code on Sourceforge, they have done it first. Now, there is some well-founded criticism about the lack of evolution in Junit, but one thing is undeniable : it really did fill a niche, back then in 2000. The code may not be beautiful (and this is not good coming from XPers) but it serves its purpose : to provide a simple framework for unit testing.

Competition is the key here and smart newcomers on this “market” are good news for us programmers. But, it's gonna take some time and a lot of work to build a similar ecosystem, a similar mindshare and usurp Junit's kingdom. That would be of course more interesting to see than denial of four years of Junit influence in a few well-rounded, but futile phrases.